A method for splicing Dunhuang manuscript fragments based on multi-feature mining and collaborative constraints

Abstract

The present invention discloses a method for splicing Dunhuang manuscript fragments based on multi-feature mining and collaborative constraints, comprising the following steps: A: preprocessing images of Dunhuang manuscript fragments to be spliced; B: constructing a multi-feature extraction network, comprising a heterogeneous feature mining module, a feature aggregation module, and a feature encoding module; C: optimizing the multi-feature extraction network using a contrastive learning module; and D: inputting the preprocessed images of the Dunhuang manuscript fragments to be spliced ​​into the optimized multi-feature extraction network to extract key features, and using the encoded aggregated features to determine the splicing matching degrees of multiple Dunhuang manuscript fragments to be spliced. The present invention utilizes the multi-feature extraction network to fully mine the writing style features, text layout features, residual edge features, and sentence coherence features of the Dunhuang manuscript fragments, and aggregates them to obtain aggregated features, ultimately obtaining the splicing matching degrees of the Dunhuang manuscript fragments, thereby achieving accurate splicing of the Dunhuang manuscript fragments.

Description

Technical Field

[0001] The present invention relates to a method for splicing Dunhuang manuscript fragments, and in particular to a method for splicing Dunhuang manuscript fragments based on multi-feature mining and collaborative constraints. Background Art

[0002] The Dunhuang manuscripts are important research materials for the study of history, archaeology, religion, anthropology, sociology, linguistics, literary history, art history, science and technology history, and ethnic history of medieval China, Central Asia, East Asia, and South Asia. They have extremely high cultural relic value and documentary research value. The images of the Dunhuang manuscript fragments are the main materials for the study of the Dunhuang manuscripts.

[0003] In existing Dunhuang manuscript research, experts typically use their domain expertise to manually piece together fragments to determine whether they belonged to the same site before they were damaged. This manual method suffers from low accuracy and efficiency, and is labor-intensive.

[0004] The invention patent with application number CN202110440552.4 and titled “A method for automatically splicing images of Dunhuang posthumous book fragments” discloses a method for automatically splicing images of Dunhuang posthumous book fragments, which can take into account the tightness of the broken edge slag and the accuracy of the grid unit width formed after the broken edges are spliced, thereby improving the efficiency and accuracy of splicing images of Dunhuang posthumous book fragments. However, the above patent only uses the physical information of the Dunhuang posthumous book fragment images as a splicing reference factor, relying on the vertical grid lines of the book scroll. The reference factor is relatively single, and the splicing accuracy needs to be further improved. Moreover, when the slag of a certain Dunhuang posthumous book fragment image to be spliced ​​is flat, because its edge features are extremely insignificant, it is very easy to cause other Dunhuang posthumous book fragment images with flat edges to be returned as candidate results, thereby bringing a large number of interfering candidate results, and ultimately resulting in poor splicing effect.

[0005] The invention patent application number is CN202211276002.4, and the title is "A Method for Splicing Dunhuang Manuscript Fragment Images Based on Sentence Fluency." It discloses a method for splicing Dunhuang manuscript fragment images based on sentence fluency. This method can comprehensively consider the sentence fluency and edge similarity of the text content of the Dunhuang manuscript fragment images to be spliced, thereby improving the efficiency and accuracy of splicing Dunhuang manuscript fragment images. Although the above patent uses both sentence fluency and edge similarity as splicing reference factors, compared with the authorized invention patent "A Method for Automatic Splicing Dunhuang Manuscript Fragment Images," the reference factors have been increased, and the splicing accuracy has also been improved to a certain extent, but there is still room for further improvement. Summary of the Invention

[0006] The purpose of the present invention is to provide a method for splicing Dunhuang manuscript fragments based on multi-feature mining and collaborative constraints, construct a multi-feature extraction network based on Transformer encoder and perform network training through a contrastive learning module, utilize the network model to fully mine the writing style features, text layout features, residual edge features and sentence fluency features of the Dunhuang manuscript fragments and perform feature aggregation, obtain the aggregated features of each Dunhuang manuscript fragment image, and finally obtain the splicing matching degree of the Dunhuang manuscript fragments, thereby achieving accurate splicing of the Dunhuang manuscript fragments.

[0007] The present invention adopts the following technical solutions:

[0008] A method for splicing Dunhuang manuscript fragments based on multi-feature mining and collaborative constraints includes the following steps:

[0009] A: Preprocessing of the images of the Dunhuang manuscript fragments to be spliced, including image resizing, text string selection, binarization of text and non-text areas, image edge delineation, image text extraction, and the construction of a text string set.

[0010] B: Construct a multi-feature extraction network based on a Transformer encoder. The multi-feature extraction network is used to mine multiple key features of the Dunhuang manuscript fragment images and obtain the aggregated features of each image after encoding through feature aggregation and feature encoding based on the Transformer encoder. The multi-feature extraction network includes a heterogeneous feature mining module, a feature aggregation module, and a feature encoding module.

[0011] The heterogeneous feature mining module includes a writing style feature embedding learning network, a text layout feature embedding learning network, a residual edge feature embedding learning network, and a sentence coherence feature embedding learning network, which are used to mine and extract writing style features, text layout features, residual edge features, and sentence coherence features from images, respectively.

[0012] The feature aggregation module is used to aggregate the obtained writing style features, text layout features, residual edge features and sentence coherence features, and obtain the aggregated features based on the four key features for each image by stacking them side by side;

[0013] The feature encoding module is used to fuse the aggregated features and obtain the encoded aggregated features of each image through the Transformer encoder. The encoded aggregated features include global features and local features.

[0014] C: Use the contrastive learning module to learn the encoded aggregate features obtained by the multi-feature extraction network to further optimize the multi-feature extraction network;

[0015] D: The pre-processed images of the Dunhuang manuscript fragments to be spliced ​​are input into the optimized multi-feature extraction network to extract key features, obtain the encoded aggregated features of each image, and use the encoded aggregated features to determine the splicing matching degree of multiple Dunhuang manuscript fragments to be spliced.

[0016] The writing style feature embedding learning network includes a text-independent writing style feature embedding learning network and a text-dependent writing style feature embedding learning network; the text-independent writing style feature embedding learning network is used to obtain the writing style features of the image of the Dunhuang manuscript fragment; the text-dependent writing style feature embedding learning network is used to obtain the writing style features of the text image of a single text;

[0017] The text-independent writing style feature embedding learning network is trained as follows:

[0018] B111: Construct a training set containing multiple fragments of Dunhuang manuscripts written by the same person and multiple fragments of Dunhuang manuscripts written by different people. The fragments of Dunhuang manuscripts written by the same person are used as positive samples, while the fragments of Dunhuang manuscripts written by different people are used as negative samples.

[0019] B112: Segment the images corresponding to all text columns separately. Use a sliding window of the same size to continuously move according to a set step size to segment the image corresponding to each text column into n non-overlapping image blocks.

[0020] B113: The obtained n image blocks are fed into a text-independent writing style feature embedding learning network to extract writing style features. Finally, n image features are obtained, with each image block corresponding to one image feature.

[0021] B114: Randomly divide n image features into two groups of image feature groups, and randomly select The image features are fused to generate a fused writing style feature, and then the two writing style features are sent to the fully connected layer for dimensionality reduction to obtain two length-transformed writing style features.

[0022] B115: Send the two handwriting style features after dimensionality reduction operation to the contrastive learning module for learning;

[0023] B116: Using the positive and negative samples in the training set, the character-independent writing style feature embedding learning network is trained until the error of the character-independent writing style feature embedding learning network reaches a set threshold, or the number of iterative training reaches a set threshold number of iterations; ultimately, a trained character-independent writing style feature embedding learning network is obtained;

[0024] In the process of training with positive samples, contrastive learning is used to shorten the Euclidean distance of the writing style features of two images of Dunhuang manuscript fragments that have been determined to be written by the same person; in the process of training with negative samples, contrastive learning is used to shorten the Euclidean distance of the edge features of two images of Dunhuang manuscript fragments that have been determined to be written by different people.

[0025] The text-related writing style feature embedding learning network is trained as follows:

[0026] B121: Construct a training set containing multiple images of the same character from one or more Dunhuang manuscript fragments. Two images of the same character written by the same person are used as positive samples, while two images of the same character written by different people are used as negative samples.

[0027] B122: Input each character image in the training set into the character-related writing style feature embedding learning network;

[0028] B123: The text-related writing style feature embedding learning network extracts image features from the input text image. After processing through the fully connected layer, the content features and writing style features of the input text image are obtained through decoupled learning.

[0029] B124: Send the writing style features of two identical characters to the comparative learning module for learning;

[0030] B125: Using the positive and negative samples in the training set, the character-related writing style feature embedding learning network is trained until the error of the character-related writing style feature embedding learning network reaches a set threshold, or the number of iterative training reaches a set threshold number of iterations; finally, a trained character-related writing style feature embedding learning network is obtained;

[0031] In the process of training with positive samples, contrastive learning is used to shorten the Euclidean distance between the writing style features of two text images of the same character written by the same person; in the process of training with negative samples, contrastive learning is used to shorten the Euclidean distance between the writing style features of two text images of the same character written by two different people.

[0032] The text layout feature embedding learning network is trained according to the following method:

[0033] B21: Construct a training set, which contains multiple groups of images of Dunhuang manuscript fragments that have been determined to be able to be pieced together, and multiple groups of images of Dunhuang manuscript fragments that have been determined to be unable to be pieced together. Each image group contains two binary images of Dunhuang manuscript fragments. The image groups that can be pieced together are used as positive samples, and the image groups that cannot be pieced together are used as negative samples.

[0034] B22: The binarized image of each Dunhuang manuscript fragment in the training set is fed into the text layout feature embedding learning network to extract the text layout features and obtain the text layout features of each Dunhuang manuscript fragment image;

[0035] B23: Send the text layout features of the two Dunhuang manuscript fragments into the contrastive learning module for learning;

[0036] B24: Using the positive and negative samples in the training set, the text layout feature embedding learning network is trained until the error of the text layout feature embedding learning network reaches a set threshold, or the number of iterative training reaches a set threshold of iterations; finally, the trained text layout feature embedding learning network is obtained;

[0037] In the process of training with positive samples, contrastive learning is used to shorten the Euclidean distance of the text layout features of two images of Dunhuang manuscript fragments that have been determined to be able to be spliced ​​together; in the process of training with negative samples, contrastive learning is used to shorten the Euclidean distance of the text layout features of two images of Dunhuang manuscript fragments that have been determined to be unable to be spliced ​​together.

[0038] The residual edge feature embedding learning network is trained according to the following method:

[0039] B31: Construct a training set, which contains multiple groups of Dunhuang manuscript fragment images that have been determined to be able to be pieced together, and multiple groups of Dunhuang manuscript fragment images that have been determined to be unable to be pieced together. Each image group contains two images of the edge of the Dunhuang manuscript fragments. The image groups that can be pieced together are used as positive samples, and the image groups that cannot be pieced together are used as negative samples.

[0040] B32: Send the residual edge images of the Dunhuang manuscript fragments in the training set to the residual edge feature embedding learning network to extract the residual edge features and obtain the residual edge features of each Dunhuang manuscript fragment image;

[0041] B33: Send the edge features of the two Dunhuang manuscript fragments into the contrast learning module for learning;

[0042] B34: Using the positive and negative samples in the training set, the residual edge feature embedding learning network is trained until the error of the residual edge feature embedding learning network reaches a set threshold, or the number of iterative training reaches a set threshold of iterations; finally, the trained residual edge feature embedding learning network is obtained;

[0043] In the process of training with positive samples, contrastive learning is used to shorten the Euclidean distance between the residual edge features of two images of Dunhuang manuscript fragments that have been determined to be able to be spliced ​​together; in the process of training with negative samples, contrastive learning is used to shorten the Euclidean distance between the residual edge features of two images of Dunhuang manuscript fragments that have been determined to be unspliceable.

[0044] The sentence coherence feature embedding learning network is trained according to the following method:

[0045] B41: Construct a training set containing multiple groups of positive and negative samples;

[0046] B42: Each sentence in the training set is fed into the sentence coherence feature embedding learning network to extract the sentence coherence features and obtain the sentence coherence features of each sentence;

[0047] B43: Use the positive and negative samples in the training set to fine-tune the sentence coherence feature embedding learning network until the number of iterative training reaches the set iteration threshold; finally, the fine-tuned sentence coherence feature embedding learning network is obtained.

[0048] When using the tuned and trained sentence coherence feature embedding learning network to extract the sentence coherence features of the image, the sentence coherence values ​​corresponding to the two Dunhuang manuscript fragment images in various relative position relationships and alignment states are obtained respectively, and then the two corresponding sentences in the two Dunhuang manuscript fragment images corresponding to the highest value of the sentence coherence value are determined, and then the two corresponding sentences are embedded in the learning network using the sentence coherence features to obtain the sentence coherence features of each sentence respectively.

[0049] In step C, the encoded aggregated features obtained by the multi-feature extraction network are learned according to the following steps:

[0050] C1: Construct a training set containing multiple groups of images of Dunhuang manuscript fragments that have been determined to be able to be pieced together, and multiple groups of images of Dunhuang manuscript fragments that have been determined to be unable to be pieced together. Each image group contains two pre-processed images of Dunhuang manuscript fragments. The image groups that can be pieced together are used as positive samples, and the image groups that cannot be pieced together are used as negative samples.

[0051] C2: Each pre-processed image of the Dunhuang manuscript fragment in the training set is fed into a multi-feature extraction network. The heterogeneous feature mining module mines and extracts the writing style features, text layout features, residual edge features, and sentence coherence features of each image. The feature aggregation module then aggregates the obtained writing style features, text layout features, residual edge features, and sentence coherence features, and stacks them in parallel to obtain an aggregated feature based on the four key features for each image. Finally, the feature encoding module fuses and learns the aggregated features, and uses a set encoder to obtain the encoded aggregated features of each image. The encoded aggregated features contain both global and local features.

[0052] C3: The aggregated features after encoding the two images are fed into the contrastive learning module for learning;

[0053] C4: Use the positive and negative samples in the training set to train the multi-feature extraction network. After each round of training, the contrastive learning module optimizes the multi-feature extraction network through error backpropagation until the error of the multi-feature extraction network reaches the set threshold or the number of iterative training reaches the set iteration threshold. Finally, the trained multi-feature extraction network is obtained.

[0054] In the process of training the multi-feature extraction network with positive samples, contrastive learning is used to shorten the Euclidean distance of the aggregated features after encoding the two images of Dunhuang manuscript fragments that have been determined to be able to be spliced ​​together; in the process of training the multi-feature extraction network with negative samples, contrastive learning is used to shorten the Euclidean distance of the aggregated features after encoding the two images of Dunhuang manuscript fragments that have been determined to be unspliced ​​together.

[0055] In the step D, the pre-processed Dunhuang manuscript images A and B to be spliced ​​are simultaneously input into the optimized multi-feature extraction network to obtain the encoded aggregate features of images A and B respectively, and then the obtained global features are used to calculate the Euclidean distance between the encoded aggregate features of images A and B; similarly, the Euclidean distances between the encoded aggregate features of image A and the remaining images are obtained in sequence; finally, the top K aggregate features with the smallest Euclidean distances are selected, and the Dunhuang manuscript images corresponding to the top K aggregate features are used as the K Dunhuang manuscript images with the highest splicing matching degree with Dunhuang manuscript image A.

[0056] The present invention explores three physical characteristics of Dunhuang manuscript images, namely writing style characteristics, text layout characteristics and residual edge characteristics, as well as the intrinsic logical characteristics of the text content in Dunhuang manuscript images, such as sentence fluency characteristics, and organically integrates the above four characteristics to synergistically constrain the characteristic expression of Dunhuang manuscript images, thereby effectively improving the splicing accuracy of Dunhuang manuscript fragment images. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0058] The present invention is described in detail below with reference to the accompanying drawings and embodiments:

[0059] like Figure 1 As shown, the multi-feature mining and collaborative constraint Dunhuang manuscript fragment splicing method of the present invention includes the following steps:

[0060] A: Preprocessing of the images of the Dunhuang manuscript fragments to be spliced, including image resizing, text string selection, binarization of text and non-text areas, image edge delineation, image text extraction, and the construction of a text string set.

[0061] B: Construct a multi-feature extraction network based on a Transformer encoder. The multi-feature extraction network is used to mine multiple key features of the Dunhuang manuscript fragment images and obtain the aggregated features of each image after encoding through feature aggregation and feature encoding based on the Transformer encoder. The multi-feature extraction network includes a heterogeneous feature mining module, a feature aggregation module, and a feature encoding module.

[0062] The heterogeneous feature mining module includes a writing style feature embedding learning network, a text layout feature embedding learning network, a residual edge feature embedding learning network, and a sentence coherence feature embedding learning network, which are used to mine and extract four key features from images: writing style features, text layout features, residual edge features, and sentence coherence features.

[0063] Among them, writing style features refer to the handwriting characteristics of the writer, including pen stroke characteristics, pen movement characteristics, thickness characteristics, and connected stroke characteristics; text layout features refer to the regularity of the arrangement of word columns, including the width of word columns and the gaps between word columns; residual edge features refer to the contour shape characteristics of residual edges; sentence coherence features refer to the semantic correctness of two sentences after being connected, including logical correctness and grammatical correctness;

[0064] The feature aggregation module is used to aggregate the obtained writing style features, text layout features, residual edge features and sentence coherence features, and obtain the aggregated features based on the four key features for each image by stacking them side by side;

[0065] The feature encoding module is used to fuse the aggregated features and obtain the aggregated features of each image after encoding through the Transformer encoder. The encoded aggregated features include global features (c0, that is, the corresponding vector obtained after the additional [CLS] token added to the sequence corresponding to the image block passes through the Transformer encoder) and local features (c1, ..., cm ).

[0066] For the Transformer encoder, before the image or feature is sent to the encoder, it is usually divided into multiple image blocks or feature blocks. The feature vector corresponding to each image block or feature block is recorded as c1,...,c m . With c1,…,c m The difference is that the c0 feature vector does not initially correspond to a certain image block in the image, but through the self-attention mechanism in the Transformer encoder, the feature vectors c1, ..., c m If the c0 feature vector is selected and sent to the final classification module or other task module, then with the iterative training and weight update of the neural network, c0 will eventually become a compact representation of the global features of the entire image, while c1, ..., c m It is the local feature representation of the image.

[0067] C: Use the contrastive learning module to learn the encoded aggregate features obtained by the multi-feature extraction network to further optimize the multi-feature extraction network;

[0068] D: The pre-processed images of the Dunhuang manuscript fragments to be spliced ​​are input into the optimized multi-feature extraction network to extract key features, obtain the encoded aggregated features of each image, and use the encoded aggregated features to determine the splicing matching degree of multiple Dunhuang manuscript fragments to be spliced.

[0069] Considering that the text layout feature embedding learning network, the residual edge feature embedding learning network, and the sentence coherence feature embedding learning network need to be adjusted accordingly before they can be used as inputs of the embedding learning network when performing feature mining and extraction, the present invention pre-processes the image of the Dunhuang manuscript fragments in step A through the following steps:

[0070] A1: Adjust the image size uniformly;

[0071] A standard value for the character size is set, and then a character is selected from each of the images of the Dunhuang manuscript fragments to be spliced, and the size of the character is adjusted to the set standard value by scaling the image, thereby obtaining a uniformly sized image;

[0072] A2: Select the text string in the image;

[0073] If there is only one column of text in the image, then this column of text is selected as the column of text to be segmented; if there are multiple columns of text in the image, then each column of text is treated as a separate column of text to be segmented;

[0074] A3: Binarization of text and non-text areas in the image;

[0075] Based on whether the area in the image contains text, the image is processed by a binarization algorithm to obtain a binary image of the image, where the area containing text is white and the area without text is black;

[0076] A4: Describe the residual edge of the image;

[0077] Describing the residual edges of the resized image; the residual edges refer to the edges of the broken parts of the Dunhuang manuscript fragments to be pieced together;

[0078] A5: Extract text from images and construct text sequence sets;

[0079] According to the method described in step G2-2 of the disclosed invention patent with patent number "2022112760024" and patent name "Dunhuang manuscript fragment image splicing method based on sentence fluency", the text extraction in the Dunhuang manuscript fragment image and the construction of the text column set are performed in sequence. The text column set in this application is the text column in the patent "Dunhuang manuscript fragment image splicing method based on sentence fluency". The specific process will not be repeated here.

[0080] Handwriting style is one of the key features that determines whether two images of Dunhuang manuscripts can be spliced ​​together. If the text in the two images can be confirmed to be different from the handwriting of the same person, the probability of the two images being spliced ​​together is extremely low. Therefore, by constructing a learning network for embedding both text-independent and text-dependent writing style features, this method identifies whether the handwriting features of the text in the two images are consistent, thus eliminating a large number of irrelevant splicing candidates.

[0081] In step B, the constructed writing style feature embedding learning network, including both a text-independent writing style feature embedding learning network and a text-dependent writing style feature embedding learning network, can utilize existing Transformer neural networks or convolutional neural networks. Feature extraction is performed on the image of the Dunhuang manuscript fragment using the text-independent writing style feature embedding learning network and / or the text-dependent writing style feature embedding learning network to ultimately obtain the writing style features of the image.

[0082] Among them, the text-independent writing style feature embedding learning network is used to obtain the writing style features of the image of the Dunhuang manuscript fragment; the text-related writing style feature embedding learning network is used to obtain the writing style features of the text image of a single text;

[0083] In the present invention, when training the text-independent writing style feature embedding learning network, the following method is used:

[0084] B111: Construct a training set containing multiple fragments of Dunhuang manuscripts written by the same person and multiple fragments of Dunhuang manuscripts written by different people. The fragments of Dunhuang manuscripts written by the same person are used as positive samples, while the fragments of Dunhuang manuscripts written by different people are used as negative samples.

[0085] B112: Segment the images corresponding to all text columns separately. Use a sliding window of the same size to continuously move according to a set step size to segment the image corresponding to each text column into n non-overlapping image blocks.

[0086] In this embodiment, the sliding window may be moved in a direction of sliding horizontally first and then vertically, or in a direction of sliding vertically first and then horizontally.

[0087] B113: The obtained n image blocks are fed into a text-independent writing style feature embedding learning network to extract writing style features. Finally, n image features are obtained, with each image block corresponding to one image feature.

[0088] B114: Randomly divide n image features into two groups of image feature groups, and randomly select The image features are fused to generate a fused handwriting style feature. The two handwriting style features are then fed into a fully connected layer for dimensionality reduction to obtain two length-transformed handwriting style features. Feature fusion refers to finding the average or maximum value of the elements at each identical position in multiple vectors.

[0089] B115: Send the two handwriting style features after dimensionality reduction operation to the contrastive learning module for learning;

[0090] B116: Using the positive and negative samples in the training set, the character-independent writing style feature embedding learning network is trained until the error of the character-independent writing style feature embedding learning network reaches a set threshold, or the number of iterative training reaches a set threshold number of iterations; ultimately, a trained character-independent writing style feature embedding learning network is obtained;

[0091] In this embodiment, during the feature fusion process of the positive sample, the writing style features are generated by fusing the image features in the following composition:

[0092] 1. For a character sequence to be segmented in an image of a Dunhuang manuscript fragment written by the same person, the character-independent writing style features are embedded into the image features generated by the learning network. During training, contrastive learning is used to narrow the Euclidean distance between the two writing style features.

[0093] 2. For different character sequences to be segmented in one or more images of Dunhuang manuscript fragments written by the same person, the character-independent writing style features are embedded into the image features generated by the learning network. During training, contrastive learning is used to narrow the Euclidean distance between the two writing style features.

[0094] During the feature fusion process of negative samples, the writing style features are generated by fusing image features in the following composition:

[0095] 1. For different character sequences to be segmented in images of Dunhuang manuscript fragments written by different authors, the character-independent writing style features are embedded into the image features generated by the learning network. During training, contrastive learning is used to increase the Euclidean distance between the two writing style features.

[0096] In the present invention, when embedding the writing style features related to the text into the learning network, the following method is used:

[0097] B121: Construct a training set containing multiple images of the same character in one or more Dunhuang manuscript fragments. Two images of the same character written by the same person in the same Dunhuang manuscript fragment are used as positive samples, while two images of the same character written by different people in two Dunhuang manuscript fragments are used as negative samples.

[0098] B122: Input each character image in the training set into the character-related writing style feature embedding learning network;

[0099] B123: The text-related writing style feature embedding learning network extracts image features from the input text image. After processing through the fully connected layer, the content features and writing style features of the input text image are obtained through decoupled learning.

[0100] B124: Send the writing style features of two identical characters to the comparative learning module for learning;

[0101] B125: Using the positive and negative samples in the training set, the character-related writing style feature embedding learning network is trained until the error of the character-related writing style feature embedding learning network reaches a set threshold, or the number of iterative training reaches a set threshold number of iterations; finally, a trained character-related writing style feature embedding learning network is obtained;

[0102] In the process of training the learning network for embedding the writing style features related to the characters using positive samples, contrastive learning is used to shorten the Euclidean distance between the writing style features of the two positive samples; in the process of training the learning network for embedding the writing style features related to the characters using negative samples, contrastive learning is used to shorten the Euclidean distance between the writing style features of the two negative samples;

[0103] In the present invention, regardless of whether the two images of Dunhuang manuscript fragments contain the same text, the trained text-independent writing style features can be used to embed the learning network to extract the writing style features of the images. The pre-processed images of the Dunhuang manuscript fragments to be spliced ​​are subjected to feature extraction according to the method of steps B112 to B114 to obtain the writing style features of the images.

[0104] When two images of Dunhuang manuscript fragments contain the same text, the trained text-related writing style features can be used to embed the learning network to extract the writing style features of individual characters. If the two pre-processed Dunhuang manuscript fragment images to be spliced ​​contain one identical text, the corresponding two text images will be subjected to feature extraction according to the methods in steps B122 to B123 to obtain the writing style features of the text images respectively.

[0105] In step B, the text layout feature embedding learning network constructed can use an existing Transformer neural network or a convolutional neural network. For images of Dunhuang manuscript fragments, the text layout feature embedding learning network can be used to extract features from the corresponding binary image, ultimately obtaining the image's text layout features.

[0106] When splicing Dunhuang manuscript fragments, if two fragment images are to be joined, their text layout characteristics (the arrangement of text columns), such as the width of the text columns and the spacing between them, must be essentially identical. Therefore, the present invention selects text layout characteristics as one of the key features for image splicing.

[0107] The present invention uses a text region binarization image to preprocess the Dunhuang manuscript images, which can effectively shield the interference of redundant information such as background and text content, so that the constructed text layout feature embedding learning network can focus on comparing whether the text column arrangement patterns of two Dunhuang manuscript images are consistent, thereby improving the accuracy of text layout pattern comparison.

[0108] In the present invention, when training the text layout feature embedding learning network, the following method is used:

[0109] B21: Construct a training set, which contains multiple groups of images of Dunhuang manuscript fragments that have been determined to be able to be pieced together, and multiple groups of images of Dunhuang manuscript fragments that have been determined to be unable to be pieced together. Each image group contains two binary images of Dunhuang manuscript fragments. The image groups that can be pieced together are used as positive samples, and the image groups that cannot be pieced together are used as negative samples.

[0110] Since the number of two different Dunhuang manuscript fragments that can be spliced ​​together is limited, in order to increase the number of positive samples, a Dunhuang manuscript fragment can also be rotated and scaled to generate a transformed Dunhuang manuscript fragment image, and then the binarized images of the images before and after the transformation are used as positive samples; or a Dunhuang manuscript fragment can be randomly cut into two parts, and then rotated and scaled to generate transformed Dunhuang manuscript fragment images respectively, and then the binarized images of the two transformed images are used as positive samples;

[0111] B22: The binarized image of each Dunhuang manuscript fragment in the training set is fed into the text layout feature embedding learning network to extract the text layout features and obtain the text layout features of each Dunhuang manuscript fragment image;

[0112] B23: Send the text layout features of the two Dunhuang manuscript fragments into the contrastive learning module for learning;

[0113] B24: Using the positive and negative samples in the training set, the text layout feature embedding learning network is trained until the error of the text layout feature embedding learning network reaches a set threshold, or the number of iterative training reaches a set threshold of iterations; finally, the trained text layout feature embedding learning network is obtained;

[0114] In this embodiment, in the process of training the text layout feature embedding learning network using positive samples, contrastive learning is used to shorten the Euclidean distance between the text layout features of two images of Dunhuang manuscript fragments that have been determined to be capable of splicing; in the process of training the text layout feature embedding learning network using negative samples, contrastive learning is used to shorten the Euclidean distance between the text layout features of two images of Dunhuang manuscript fragments that have been determined to be uncapable of splicing.

[0115] In the present invention, when the trained text layout feature embedding learning network is used to extract the text layout features of the image, the pre-processed binarized image of the Dunhuang manuscript fragments to be spliced ​​is subjected to feature extraction according to the method in step B22 to obtain the text layout features of the image;

[0116] In the present invention, the residual edge feature embedding learning network constructed can adopt an existing Transformer neural network or a convolutional neural network. For the image of the Dunhuang manuscript fragment to be spliced, the residual edge feature embedding learning network is used to extract features, ultimately obtaining the residual edge features of the image.

[0117] Residual edge features, or the degree of matching between the broken edges of two images, are a common feature used in image splicing. This method, by mining and extracting these residual edge features, can further improve the splicing accuracy of Dunhuang manuscript fragments by combining them with features of writing style, text layout, and sentence coherence.

[0118] In step B, when training the residual edge feature embedding learning network, proceed as follows:

[0119] B31: Construct a training set, which contains multiple groups of Dunhuang manuscript fragment images that have been determined to be able to be pieced together, and multiple groups of Dunhuang manuscript fragment images that have been determined to be unable to be pieced together. Each image group contains two images of the edge of the Dunhuang manuscript fragments. The image groups that can be pieced together are used as positive samples, and the image groups that cannot be pieced together are used as negative samples.

[0120] B32: Send the residual edge images of the Dunhuang manuscript fragments in the training set to the residual edge feature embedding learning network to extract the residual edge features and obtain the residual edge features of each Dunhuang manuscript fragment image;

[0121] B33: Send the edge features of the two Dunhuang manuscript fragments into the contrast learning module for learning;

[0122] B34: Using the positive and negative samples in the training set, the residual edge feature embedding learning network is trained until the error of the residual edge feature embedding learning network reaches a set threshold, or the number of iterative training reaches a set threshold of iterations; finally, the trained residual edge feature embedding learning network is obtained;

[0123] In this embodiment, in the process of training the residual edge feature embedding learning network using positive samples, contrastive learning is used to shorten the Euclidean distance between the residual edge features of two images of Dunhuang manuscript fragments that have been determined to be able to be spliced ​​together; in the process of training the residual edge feature embedding learning network using negative samples, contrastive learning is used to shorten the Euclidean distance between the residual edge features of two images of Dunhuang manuscript fragments that have been determined to be unable to be spliced ​​together.

[0124] In the present invention, when extracting the residual edge features of an image using the trained residual edge feature embedding learning network, the residual edge image of the pre-processed Dunhuang manuscript fragment image is subjected to feature extraction according to the method in step B32 to obtain the residual edge features of the image;

[0125] Sentence coherence features examine the grammatical correctness and logical rationality of the textual content in the two images to be joined, based on their logic and grammar. By mining and extracting these features, the present invention can further eliminate a large number of irrelevant candidate joins based on writing style, text layout, and residual edge features, thereby improving join accuracy.

[0126] In step B, the sentence coherence feature embedding learning network constructed can use the existing BERT pre-trained language model. The sentence coherence feature embedding learning network extracts features from the text sequence in the Dunhuang manuscript fragment image, ultimately obtaining the image's sentence coherence features.

[0127] In the present invention, when training the sentence coherence feature embedding learning network, the following method is used:

[0128] B41: Construct a training set containing multiple groups of positive and negative samples;

[0129] The method for constructing the training set can be carried out according to the method described in step G2-1 of the published invention patent with patent number "2022112760024" and patent name "Dunhuang manuscript fragment image splicing method based on sentence fluency", which will not be repeated here;

[0130] B42: Each sentence in the training set is fed into the sentence coherence feature embedding learning network to extract the sentence coherence features and obtain the sentence coherence features of each sentence;

[0131] B43: Using the positive and negative samples in the training set, the sentence fluency feature embedding learning network is fine-tuned and trained until the number of iterative training reaches the set iteration threshold; finally, the fine-tuned and trained sentence fluency feature embedding learning network is obtained;

[0132] In the present invention, when using the tuned and trained sentence coherence feature embedding learning network to extract the sentence coherence features of the image, the pre-processed Dunhuang manuscript fragment image is first obtained according to the method described in step G2 of the disclosed invention patent with patent number "2022112760024" and patent name "Dunhuang manuscript fragment image splicing method based on sentence coherence", and the corresponding sentence coherence values ​​of Dunhuang manuscript fragment image A and Dunhuang manuscript fragment image B in various relative position relationships and aligned states are obtained respectively, and then the two corresponding sentences in the two Dunhuang manuscript fragment images corresponding to the highest value of the sentence coherence value are determined, and then the two corresponding sentences are used using the method described in step B42 to obtain the sentence coherence features of each sentence respectively.

[0133] In step C, when using the contrastive learning module to learn the encoded aggregated features obtained by the multi-feature extraction network, the following steps are followed:

[0134] C1: Construct a training set containing multiple groups of images of Dunhuang manuscript fragments that have been determined to be able to be pieced together, and multiple groups of images of Dunhuang manuscript fragments that have been determined to be unable to be pieced together. Each image group contains two pre-processed images of Dunhuang manuscript fragments. The image groups that can be pieced together are used as positive samples, and the image groups that cannot be pieced together are used as negative samples.

[0135] C2: Each pre-processed image of the Dunhuang manuscript fragment in the training set is fed into a multi-feature extraction network based on a Transformer encoder; the heterogeneous feature mining module mines and extracts the writing style features, text layout features, residual edge features, and sentence coherence features of each image respectively; then the feature aggregation module aggregates the obtained writing style features, text layout features, residual edge features, and sentence coherence features, and obtains the aggregated features of each image based on the four key features by stacking them in parallel; finally, the feature encoding module fuses and learns the aggregated features, and obtains the encoded aggregated features of each image through the set encoder; the encoded aggregated features include the global feature c0 and the local features c1, ..., c m .

[0136] C3: The aggregated features after encoding the two images are fed into the contrastive learning module for learning;

[0137] C4: Use the positive and negative samples in the training set to train the multi-feature extraction network. After each round of training, the contrastive learning module optimizes the multi-feature extraction network through error backpropagation until the error of the multi-feature extraction network reaches the set threshold or the number of iterative training reaches the set iteration threshold. Finally, the trained multi-feature extraction network is obtained.

[0138] In the process of training the multi-feature extraction network with positive samples, contrastive learning is used to shorten the Euclidean distance of the aggregated features after encoding the two images of Dunhuang manuscript fragments that have been determined to be able to be spliced ​​together; in the process of training the multi-feature extraction network with negative samples, contrastive learning is used to shorten the Euclidean distance of the aggregated features after encoding the two images of Dunhuang manuscript fragments that have been determined to be unspliced ​​together.

[0139] In the step D, the pre-processed Dunhuang manuscript images A and B to be spliced ​​are simultaneously input into the optimized multi-feature extraction network to obtain the encoded aggregate features of images A and B respectively, and then the obtained global feature c0 is used to calculate the Euclidean distance between the encoded aggregate features of images A and B; similarly, the Euclidean distances between the encoded aggregate features of image A and the remaining images are obtained in sequence; finally, the top K aggregate features with the smallest Euclidean distances are selected, and the Dunhuang manuscript images corresponding to the top K aggregate features are used as the K Dunhuang manuscript images with the highest splicing matching degree with Dunhuang manuscript image A.

Claims

1. A method for splicing Dunhuang manuscript fragments based on multi-feature mining and collaborative constraints, characterized by: The following steps are involved: A: Preprocessing of the images of the Dunhuang manuscript fragments to be spliced, including image resizing, text string selection, binarization of text and non-text areas, image edge delineation, image text extraction, and the construction of a text string set. B: Construct a multi-feature extraction network based on a Transformer encoder. The multi-feature extraction network is used to mine multiple key features of the Dunhuang manuscript fragment images and obtain the aggregated features of each image after encoding through feature aggregation and feature encoding based on the Transformer encoder. The multi-feature extraction network includes a heterogeneous feature mining module, a feature aggregation module, and a feature encoding module. The heterogeneous feature mining module includes a writing style feature embedding learning network, a text layout feature embedding learning network, a residual edge feature embedding learning network, and a sentence coherence feature embedding learning network, which are used to mine and extract writing style features, text layout features, residual edge features, and sentence coherence features from images, respectively. The feature aggregation module is used to aggregate the obtained writing style features, text layout features, residual edge features and sentence coherence features, and obtain the aggregated features based on the four key features for each image by stacking them side by side; The feature encoding module is used to fuse the aggregated features and obtain the encoded aggregated features of each image through the Transformer encoder. The encoded aggregated features include global features and local features. C: Use the contrastive learning module to learn the encoded aggregate features obtained by the multi-feature extraction network to further optimize the multi-feature extraction network; D: The pre-processed images of the Dunhuang manuscript fragments to be spliced ​​are input into the optimized multi-feature extraction network to extract key features, obtain the encoded aggregated features of each image, and use the encoded aggregated features to determine the splicing matching degree of multiple Dunhuang manuscript fragments to be spliced.

2. The multi-feature mining and collaborative constraint-based Dunhuang manuscript fragment splicing method according to claim 1 is characterized by: The writing style feature embedding learning network includes a text-independent writing style feature embedding learning network and a text-related writing style feature embedding learning network; the text-independent writing style feature embedding learning network is used to obtain the writing style features of the images of Dunhuang manuscript fragments; the text-related writing style feature embedding learning network is used to obtain the writing style features of the text images of individual characters.

3. The multi-feature mining and collaborative constraint Dunhuang manuscript fragment splicing method according to claim 2 is characterized in that: The text-independent writing style feature embedding learning network is trained as follows: B111: Construct a training set containing multiple fragments of Dunhuang manuscripts written by the same person and multiple fragments of Dunhuang manuscripts written by different people. The fragments of Dunhuang manuscripts written by the same person are used as positive samples, while the fragments of Dunhuang manuscripts written by different people are used as negative samples. B112: Segment the images corresponding to all text columns separately. Use a sliding window of the same size to continuously move according to a set step size to segment the image corresponding to each text column into n non-overlapping image blocks. B113: The obtained n image blocks are fed into a text-independent writing style feature embedding learning network to extract writing style features. Finally, n image features are obtained, with each image block corresponding to one image feature. B114: Randomly divide n image features into two groups of image feature groups, and randomly select The image features are fused to generate a fused writing style feature, and then the two writing style features are sent to the fully connected layer for dimensionality reduction to obtain two length-transformed writing style features. B115: Send the two handwriting style features after dimensionality reduction operation to the contrastive learning module for learning; B116: Using the positive and negative samples in the training set, the character-independent writing style feature embedding learning network is trained until the error of the character-independent writing style feature embedding learning network reaches a set threshold, or the number of iterative training reaches a set threshold number of iterations; ultimately, a trained character-independent writing style feature embedding learning network is obtained; In the process of training with positive samples, contrastive learning is used to shorten the Euclidean distance of the writing style features of two images of Dunhuang manuscript fragments that have been determined to be written by the same person; in the process of training with negative samples, contrastive learning is used to shorten the Euclidean distance of the edge features of two images of Dunhuang manuscript fragments that have been determined to be written by different people.

4. The multi-feature mining and collaborative constraint Dunhuang manuscript fragment splicing method according to claim 3 is characterized in that: The text-related writing style feature embedding learning network is trained as follows: B121: Construct a training set containing multiple images of the same character from one or more Dunhuang manuscript fragments. Two images of the same character written by the same person are used as positive samples, while two images of the same character written by different people are used as negative samples. B122: Input each character image in the training set into the character-related writing style feature embedding learning network; B123: The text-related writing style feature embedding learning network extracts image features from the input text image. After processing through the fully connected layer, the content features and writing style features of the input text image are obtained through decoupled learning. B124: Send the writing style features of two identical characters to the comparative learning module for learning; B125: Using the positive and negative samples in the training set, the character-related writing style feature embedding learning network is trained until the error of the character-related writing style feature embedding learning network reaches a set threshold, or the number of iterative training reaches a set threshold number of iterations; finally, a trained character-related writing style feature embedding learning network is obtained; In the process of training with positive samples, contrastive learning is used to shorten the Euclidean distance between the writing style features of two text images of the same character written by the same person; in the process of training with negative samples, contrastive learning is used to shorten the Euclidean distance between the writing style features of two text images of the same character written by two different people.

5. The multi-feature mining and collaborative constraint Dunhuang manuscript fragment splicing method according to claim 1 is characterized in that: The text layout feature embedding learning network is trained according to the following method: B21: Construct a training set, which contains multiple groups of images of Dunhuang manuscript fragments that have been determined to be able to be pieced together, and multiple groups of images of Dunhuang manuscript fragments that have been determined to be unable to be pieced together. Each image group contains two binary images of Dunhuang manuscript fragments. The image groups that can be pieced together are used as positive samples, and the image groups that cannot be pieced together are used as negative samples. B22: The binarized image of each Dunhuang manuscript fragment in the training set is fed into the text layout feature embedding learning network to extract the text layout features and obtain the text layout features of each Dunhuang manuscript fragment image; B23: Send the text layout features of the two Dunhuang manuscript fragments into the contrastive learning module for learning; B24: Using the positive and negative samples in the training set, the text layout feature embedding learning network is trained until the error of the text layout feature embedding learning network reaches a set threshold, or the number of iterative training reaches a set threshold of iterations; finally, the trained text layout feature embedding learning network is obtained; In the process of training with positive samples, contrastive learning is used to shorten the Euclidean distance of the text layout features of two images of Dunhuang manuscript fragments that have been determined to be able to be spliced ​​together; in the process of training with negative samples, contrastive learning is used to shorten the Euclidean distance of the text layout features of two images of Dunhuang manuscript fragments that have been determined to be unable to be spliced ​​together.

6. The multi-feature mining and collaborative constraint Dunhuang manuscript fragment splicing method according to claim 1 is characterized in that: The residual edge feature embedding learning network is trained according to the following method: B31: Construct a training set, which contains multiple groups of Dunhuang manuscript fragment images that have been determined to be able to be pieced together, and multiple groups of Dunhuang manuscript fragment images that have been determined to be unable to be pieced together. Each image group contains two images of the edge of the Dunhuang manuscript fragments. The image groups that can be pieced together are used as positive samples, and the image groups that cannot be pieced together are used as negative samples. B32: Send the residual edge images of the Dunhuang manuscript fragments in the training set to the residual edge feature embedding learning network to extract the residual edge features and obtain the residual edge features of each Dunhuang manuscript fragment image; B33: Send the edge features of the two Dunhuang manuscript fragments into the contrast learning module for learning; B34: Using the positive and negative samples in the training set, the residual edge feature embedding learning network is trained until the error of the residual edge feature embedding learning network reaches a set threshold, or the number of iterative training reaches a set threshold of iterations; finally, the trained residual edge feature embedding learning network is obtained; In the process of training with positive samples, contrastive learning is used to shorten the Euclidean distance between the residual edge features of two images of Dunhuang manuscript fragments that have been determined to be able to be spliced ​​together; in the process of training with negative samples, contrastive learning is used to shorten the Euclidean distance between the residual edge features of two images of Dunhuang manuscript fragments that have been determined to be unspliceable.

7. The multi-feature mining and collaborative constraint Dunhuang manuscript fragment splicing method according to claim 1 is characterized in that: The sentence coherence feature embedding learning network is trained according to the following method: B41: Construct a training set containing multiple groups of positive and negative samples; B42: Each sentence in the training set is fed into the sentence coherence feature embedding learning network to extract the sentence coherence features and obtain the sentence coherence features of each sentence; B43: Use the positive and negative samples in the training set to fine-tune the sentence coherence feature embedding learning network until the number of iterative training reaches the set iteration threshold; finally, the fine-tuned sentence coherence feature embedding learning network is obtained.

8. The multi-feature mining and collaborative constraint-based Dunhuang manuscript fragment splicing method according to claim 1 is characterized by: When using the tuned and trained sentence coherence feature embedding learning network to extract the sentence coherence features of the image, the sentence coherence values ​​corresponding to the two Dunhuang manuscript fragment images in various relative position relationships and alignment states are obtained respectively, and then the two corresponding sentences in the two Dunhuang manuscript fragment images corresponding to the highest value of the sentence coherence value are determined, and then the two corresponding sentences are embedded in the learning network using the sentence coherence features to obtain the sentence coherence features of each sentence respectively.

9. The multi-feature mining and collaborative constraint Dunhuang manuscript fragment splicing method according to claim 1 is characterized in that: In step C, the encoded aggregated features obtained by the multi-feature extraction network are learned according to the following steps: C1: Construct a training set containing multiple groups of images of Dunhuang manuscript fragments that have been determined to be able to be pieced together, and multiple groups of images of Dunhuang manuscript fragments that have been determined to be unable to be pieced together. Each image group contains two pre-processed images of Dunhuang manuscript fragments. The image groups that can be pieced together are used as positive samples, and the image groups that cannot be pieced together are used as negative samples. C2: Each pre-processed image of the Dunhuang manuscript fragment in the training set is fed into a multi-feature extraction network; the heterogeneous feature mining module mines and extracts the writing style features, text layout features, residual edge features, and sentence coherence features of each image respectively; the feature aggregation module then aggregates the obtained writing style features, text layout features, residual edge features, and sentence coherence features, and obtains the aggregated features of each image based on the four key features by stacking them in parallel; finally, the feature encoding module fuses and learns the aggregated features, and obtains the encoded aggregated features of each image through a set encoder; the encoded aggregated features include the global feature c0 and the local features c1, ..., c m ); C3: The aggregated features after encoding the two images are fed into the contrastive learning module for learning; C4: Use the positive and negative samples in the training set to train the multi-feature extraction network. After each round of training, the contrastive learning module optimizes the multi-feature extraction network through error backpropagation until the error of the multi-feature extraction network reaches the set threshold or the number of iterative training reaches the set iteration threshold. Finally, the trained multi-feature extraction network is obtained. In the process of training the multi-feature extraction network with positive samples, contrastive learning is used to shorten the Euclidean distance of the aggregated features after encoding the two images of Dunhuang manuscript fragments that have been determined to be able to be spliced ​​together; in the process of training the multi-feature extraction network with negative samples, contrastive learning is used to shorten the Euclidean distance of the aggregated features after encoding the two images of Dunhuang manuscript fragments that have been determined to be unspliced ​​together.

10. The multi-feature mining and collaborative constraint Dunhuang manuscript fragment splicing method according to claim 1 is characterized in that: In the aforementioned step D, the pre-processed Dunhuang manuscript images A and B to be spliced ​​are simultaneously input into the optimized multi-feature extraction network to obtain the encoded aggregate features of images A and B, respectively, and then the obtained global features are used to calculate the Euclidean distance between the encoded aggregate features of images A and B; Similarly, the Euclidean distances between the encoded aggregate features of image A and the remaining images are obtained in sequence; finally, the top K aggregate features with the smallest Euclidean distances are selected, and the Dunhuang manuscript images corresponding to the top K aggregate features are used as the K Dunhuang manuscript images with the highest splicing matching degree with the Dunhuang manuscript image A.

Images