Conjunctive dunhuang fragment screening method based on combinatorial optimization and multi-modal large model

Abstract

The application discloses a combinable Dunhuang fragment screening method based on combination optimization and a multimodal large model, and specifically comprises the following steps: using a twin neural network based on a variant ResNet18 of a convolutional neural network to extract a feature vector of Dunhuang fragments, predicting a matching degree score of the fragments, forming a Dunhuang fragment pair with a matching degree score greater than a threshold value into an undirected graph, using a graph neural network fused with an optimal transmission layer to screen a combinable small fragment set, and inputting a combinable Dunhuang fragment candidate splicing set and a predefined prompt word into a multimodal large language model to judge an error fragment, removing a vertex and an edge corresponding to the error fragment in the undirected graph, and re-screening to obtain a new combinable candidate splicing set. The application can be applied to a Dunhuang small fragment splicing scene, and can improve the efficiency of research in the field of ancient linguistics and social studies.

Description

Technical Field

[0001] This invention belongs to the field of cultural relic restoration, and in particular relates to a method for screening combinatorial Dunhuang fragments based on combinatorial optimization and multimodal large model. Background Technology

[0002] In the fields of culture and archaeology, the restoration of cultural relic fragments is a significant undertaking, with the restoration of fragments from the Mogao Grottoes in Dunhuang being a prime example in China. Current computer vision and deep learning methods can now be used to piece together and restore large fragments, improving the efficiency of archaeologists' work.

[0003] Current fragment restoration efforts primarily focus on large fragments. Small fragments, due to their sheer number, limited text, and insufficiently distinct features, are largely beyond the capabilities of existing computer methods. Furthermore, manually piecing together small fragments is prohibitively expensive and often fails. In ancient fragment collections, the proportion of small fragments is often significant. For example, in the Dunhuang fragment dataset currently available at Zhejiang University, approximately 10% of the fragments are small fragments containing fewer than 10 complete characters. Restoring these small fragments is crucial for understanding key information in pieced-together fragments, but currently, there is no technological solution to this problem. Summary of the Invention

[0004] The purpose of this invention is to solve the problems existing in the prior art and to provide a method for screening combinatorial Dunhuang fragments based on combinatorial optimization and multimodal large model.

[0005] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution:

[0006] A method for screening combinatorial Dunhuang fragments based on combinatorial optimization and multimodal large models includes the following steps:

[0007] S1. Obtain all possible pairs of fragments from the Dunhuang fragments to be screened, and extract the mask from the fragment pairs. Each fragment pair corresponds to a mask pair.

[0008] S2. Obtain a trained ensemble optimization solver, which consists of a Siamese neural network, an optimal graph neural network, and a multimodal large language model. The optimal graph neural network is composed of a graph neural network and an optimal transport layer cascaded together. During training, the ensemble optimization solver only updates the parameters of the Siamese neural network and the optimal graph neural network, while the parameters of the multimodal large language model remain fixed. When updating the parameters of the optimal transport layer, the top K largest probabilities output by the optimal transport layer are maximized, and the remaining probabilities output by the optimal transport layer are minimized.

[0009] S3. Each time, a mask pair is input into the twin neural network to obtain the feature vector of the Dunhuang fragment corresponding to the mask pair and the matching degree score of each mask pair; wherein, the twin neural network consists of two sub-networks for feature extraction and a classification module;

[0010] S4. Take each feature vector obtained from the Siamese neural network as a vertex, add an edge between fragment pairs whose matching degree score is greater than the preset matching degree threshold, and use the corresponding matching degree score as the weight of the edge. After all fragment pairs have been traversed, an undirected graph is obtained.

[0011] S5. Input the undirected graph obtained in step S4 into the optimal graph neural network. First, the graph neural network predicts the initial probability of the Dunhuang fragments corresponding to each vertex in the undirected graph in the candidate splicing set. After the initial probability is passed through the optimal transmission layer, the optimal probability is obtained. Select the top K Dunhuang fragments with the highest probability values ​​in the optimal probability to form the candidate splicing set.

[0012] S6. After combining two Dunhuang fragments in the candidate splicing set into a candidate Dunhuang fragment pair, input each candidate Dunhuang fragment pair and the pre-constructed prompt words into the multimodal large language model. The multimodal large language model scores each candidate Dunhuang fragment pair and outputs the score result of each candidate Dunhuang fragment pair. If the score result is less than the preset score threshold, the corresponding candidate Dunhuang fragment pair is added to the erroneous small fragment subset.

[0013] S7. In the undirected graph obtained in step S4, all feature vector vertices corresponding to the subset of erroneous fragments and all edges connected to the feature vector vertices are removed to obtain a pruned undirected graph. The pruned undirected graph is then re-inputted into the optimal graph neural network to predict a new candidate splicing set containing K Dunhuang fragments, which is used for splicing Dunhuang fragments.

[0014] Based on the above scheme, each step can be implemented in the following preferred manner.

[0015] Preferably, in the twin neural network, the first fragment mask in the i-th mask pair is input into the first sub-network to obtain the first feature vector of the Dunhuang fragment, and the second fragment mask in the i-th mask pair is input into the second sub-network to obtain the second feature vector of the Dunhuang fragment. The first feature vector and the second feature vector are concatenated, and the concatenated vector is input into the classification module to output the matching degree score of the i-th mask pair.

[0016] Preferably, the first subnetwork and the second subnetwork have the same structure and the two subnetworks have the same weights.

[0017] Preferably, the classification module is composed of a first linear layer, a ReLU activation function, a second linear layer, and a Sigmoid activation function cascaded in sequence.

[0018] Preferably, the matching score is between 0 and 1, where 0 indicates that the mask pair cannot be matched and 1 indicates that the mask pair has the highest matching degree.

[0019] Preferably, the score threshold is set to 60.

[0020] Preferably, the matching degree threshold is 0.2.

[0021] Preferably, the multimodal large language model adopts the GPT-4V model.

[0022] Preferably, the graph neural network is a graph convolutional network (GCN).

[0023] Preferably, when training the combinatorial optimization solver, the outline of the Dunhuang fragments is used as the dividing line segment, and each complete Dunhuang fragment is torn into several small Dunhuang fragments. Adjacent small Dunhuang fragments are used as pairs of positive samples, while non-adjacent small Dunhuang fragments are pairs of negative samples. The constructed positive and negative sample pairs are used as the training dataset, which contains 40,000 pairs of positive samples and 80,000 pairs of negative samples.

[0024] Compared with the prior art, the present invention has the following advantages:

[0025] This invention proposes a method for screening combinatorial Dunhuang fragments based on combinatorial optimization and a multimodal large-scale model. The combinatorial optimization solver utilizes the contour features of small fragments and solves the combinatorial explosion problem of the multimodal large-scale language model. The multimodal large-scale language model utilizes the remaining features of small fragments and can serve as an auxiliary tool for archaeologists, greatly reducing the manpower and time spent on piecing together small fragments. It can be applied to the scenario of piecing together small Dunhuang fragments, as well as to the fields of ancient linguistics and social research, thereby improving the efficiency of research in these fields. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating the overall solution of the present invention;

[0027] Figure 2 This is a schematic diagram illustrating the mask extraction process for Dunhuang fragments according to the present invention;

[0028] Figure 3 This is an architectural diagram of the twin neural network of the present invention;

[0029] Figure 4 This is a schematic diagram of the classification module of the twin neural network of the present invention;

[0030] Figure 5 A flowchart for constructing a simulated shredded dataset for this invention;

[0031] Figure 6 This is a diagram of the architecture of the convolutional network of this invention;

[0032] Figure 7 This is a schematic diagram illustrating an example of the use of the multimodal large language model of this invention;

[0033] Figure 8 This is a schematic diagram of some results obtained from an embodiment of the present invention. Detailed Implementation

[0034] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0035] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature.

[0036] Considering the vast number of small fragments of Dunhuang manuscripts disclosed by various collecting institutions, with varying sizes, preservation conditions, and textual integrity—reaching over 50,000 fragments in total—manually piecing them together is extremely difficult, time-consuming, and labor-intensive. To address these issues, in a preferred embodiment of this invention, a method for selecting collable Dunhuang fragments based on combinatorial optimization and a multimodal large-scale model is proposed. This method includes a Siamese neural network composed of convolutional neural networks and a combinatorial optimization solver composed of a graph neural network incorporating an optimal transport layer. Fragment pairs are input into this system to obtain a candidate fragment piecing set. Then, a multimodal large-scale language model selects and removes erroneous small fragment subsets, reconstructs an undirected graph, and inputs it into the graph neural network incorporating the optimal transport layer to obtain the final candidate fragment piecing set.

[0037] Existing technologies primarily address the piecing together of large fragments, while this invention focuses on piecing together small Dunhuang fragments. In this invention, "Dunhuang fragments" refers to these small fragments. These small fragments contain fewer than 10 complete characters and suffer from severe edge erosion, rendering the piecing methods used for larger fragments unsuitable. Approximately 10% of the Dunhuang manuscripts fall into the category of small fragments, some of which contain crucial textual information essential for the complete joining of the fragments.

[0038] like Figure 1 As shown, in a preferred embodiment of the present invention, the above-mentioned method for screening combinatorial Dunhuang fragments based on combinatorial optimization and multimodal large model includes the following steps S1 to S7. The specific implementation process of each step will be described in detail below.

[0039] S1. Obtain all possible pairs of fragments from the Dunhuang fragments to be screened, and extract the mask from the fragment pairs. Each fragment pair corresponds to a mask pair.

[0040] It should be noted that, in this embodiment, as Figure 2 As shown, this invention uses the binarization and closure operation functions provided in the OpenCV library. First, binarization is performed, and then the closure operation function is used to process the data, finally extracting the mask pairs corresponding to the Dunhuang fragment pairs.

[0041] S2. Obtain a trained ensemble optimization solver, which consists of a Siamese neural network, an optimal graph neural network, and a multimodal large language model. The optimal graph neural network is composed of a graph neural network (GNN) and an optimal transport layer (OT layer) cascaded together. During training, the ensemble optimization solver only updates the parameters of the Siamese neural network and the optimal graph neural network; the parameters of the multimodal large language model remain fixed. When updating the parameters of the optimal transport layer, it maximizes the top K largest probabilities output by the optimal transport layer and minimizes the remaining probabilities.

[0042] It should be noted that in this embodiment, the top K largest probabilities in the output of the optimal transport layer are maximized, and the remaining probabilities in the output of the optimal transport layer are minimized. Maximization and minimization here mean that, through optimization by the Sinkhorn algorithm, the K largest probabilities in the output of the optimal transport layer are as close as possible to the maximum value p_max, and the remaining probabilities are as close as possible to the minimum value p_min. p_max and p_min are automatically calculated by the optimal transport layer based on all probabilities output by the graph neural network.

[0043] It should be noted that, in this embodiment, the aforementioned combined optimization solver includes a Siamese neural network, an optimal graph neural network, and a multimodal large language model. Specifically, in the Siamese neural network, a variant of the convolutional neural network (CNN), ResNet18, is used as a sub-network for feature extraction; in the optimal graph neural network, an optimal transport layer is incorporated, which is composed of a graph neural network (GNN) and an optimal transport layer (OT layer) cascaded sequentially.

[0044] S3. Each time, a mask pair is input into the above twin neural network to obtain the feature vector of the Dunhuang fragment corresponding to the mask pair and the matching degree score of each mask pair; wherein, the above twin neural network consists of two sub-networks for feature extraction and a classification module.

[0045] It should be noted that in the above-mentioned Siamese neural network, the first fragment mask in the i-th mask pair is input into the first sub-network to obtain the first feature vector of the Dunhuang fragment, and the second fragment mask in the i-th mask pair is input into the second sub-network to obtain the second feature vector of the Dunhuang fragment. The first feature vector and the second feature vector are concatenated, and the concatenated vector is input into the classification module to output the matching degree score of the i-th mask pair. The first sub-network and the second sub-network have the same structure and the two sub-networks have the same weights.

[0046] It should be noted that the above classification module is composed of a first linear layer, a ReLU activation function, a second linear layer, and a Sigmoid activation function cascaded in sequence.

[0047] In this embodiment, to extract the contour information of the Dunhuang fragments and predict the matching degree between pairs of Dunhuang fragments, this invention uses ResNet18, a variant of Convolutional Neural Network (CNN), as a sub-network of a self-supervised Siamese neural network. Each time, a mask pair is input into the trained Siamese neural network, which consists of two ResNet18 sub-networks with the same weights and structure. Figure 3As shown, these two sub-networks extract feature vectors from the two input masks, representing the contour information of the Dunhuang fragments. In this embodiment, the feature vector is a 512-dimensional vector. After obtaining the feature vector of each mask corresponding to the Dunhuang fragment, the feature vectors output by the two sub-networks are concatenated. The concatenated feature vector is then input into a classification module. In the classification module, the image first passes through a fully connected layer (i.e., the first linear layer and the second linear layer), and then the output of the fully connected layer is calculated using the Sigmoid activation function. The Sigmoid activation function is used to assign a normalized matching score to each input mask pair, representing the degree of matching between the original Dunhuang fragment images. The matching score is between 0 and 1, where 0 indicates that the mask pair cannot match, and 1 indicates the highest degree of matching. The structure and settings of the Siamese network classification module are as follows: Figure 4 As shown.

[0048] S4. Using each feature vector obtained from the Siamese neural network as a vertex, add an edge between fragment pairs whose matching score is greater than the preset matching score threshold, and use the corresponding matching score as the weight of the edge. After all fragment pairs have been traversed, an undirected graph is obtained.

[0049] It should be noted that step S3 involves constructing an undirected graph, using the feature vector of each Dunhuang fragment obtained from the Siamese neural network as a vertex, adding edges between all fragment pairs with matching scores greater than a preset matching threshold, and using the matching score as the weight of the edge.

[0050] To best adapt to the extraction results of the Siamese neural network obtained in this embodiment, the matching degree threshold is set to 0.2 in this embodiment.

[0051] S5. Input the undirected graph obtained in step S4 into the above-mentioned optimal graph neural network. First, the graph neural network (GNN) predicts the initial probability of the Dunhuang fragments corresponding to each vertex in the undirected graph in the candidate splicing set. After the initial probability is passed through the above-mentioned optimal transport layer (OT Layer), the optimal probability is obtained. Select the top K Dunhuang fragments with the highest probability values ​​among the optimal probabilities to form the candidate splicing set.

[0052] It should be noted that this embodiment of the invention uses a Graph Convolutional Network (GCN) as the architecture of a Graph Neural Network (GNN). The optimal GNN is trained to learn the relationship between the feature vectors corresponding to vertices in the undirected graph, the weights of edges, and the corresponding selected Dunhuang fragments and candidate splicing sets. Ultimately, it outputs a probability value between 0 and 1 for each vertex in the undirected graph. The architecture of the Graph Convolutional Network is as follows: Figure 6 As shown, Figure 6 The normalized probability in the equation is the initial probability.

[0053] S6. After combining two Dunhuang fragments in the candidate splicing set into a candidate Dunhuang fragment pair, input each candidate Dunhuang fragment pair and the pre-constructed prompt into the multimodal large language model. The multimodal large language model scores each candidate Dunhuang fragment pair and outputs the score result of each candidate Dunhuang fragment pair. If the score result is less than the preset score threshold, the corresponding candidate Dunhuang fragment pair is added to the erroneous small fragment subset.

[0054] In this invention, the score threshold can be set according to actual conditions. In this embodiment, the score threshold is set to 60.

[0055] It should be noted that this invention mainly uses the method of API calls and the method of uploading images and prompts on web pages to use a multimodal large language model. That is, remote calls are made through API programming interfaces, and manual interaction is performed through web pages with UI interfaces. The prompts used in this invention are as follows: "Suppose you are an expert in solving jigsaw puzzles. Now please help me solve the problem of piecing together fragments of an ancient artifact. I will provide you with a series of fragments, some of which can be pieced together and some of which cannot. When piecing together, the following rules must be followed: (1) Ignore straight edges, and straight edges cannot be pieced together; (2) Use visual ability, do not use scripts to complete the task; (3) Give a score between 1 and 100 and determine the orientation of the small fragments that can be pieced together."

[0056] In this embodiment, several multimodal large language models, including LLaVA, QwenVM, Erine, and GPT-4V, were tested. OpenAI's GPT-4V performed best, therefore, GPT-4V was used as the multimodal large language model in this embodiment. An example of using a multimodal large language model in Top-4 is shown below. Figure 7 As shown.

[0057] S7. In the undirected graph obtained in step S4, all feature vector vertices corresponding to the subset of erroneous fragments and all edges connected to the feature vector vertices are removed to obtain a pruned undirected graph. The pruned undirected graph is then re-inputted into the optimal graph neural network to predict a new candidate splicing set containing K Dunhuang fragments, which is used for splicing Dunhuang fragments.

[0058] It's important to note that using a multimodal large language model alone can lead to the combinatorial explosion problem. When the candidate splice set is fed into the multimodal large language model all at once, the model may be unable to make a correct judgment due to the sheer number of possible combinations. Adding the preliminary results (candidate splice set) from the combinatorial optimization solver to the multimodal large language model for judgment can avoid the combinatorial explosion problem and make fuller use of the information in the image.

[0059] It should also be noted that when training the combinatorial optimization solver, the parameters of the multimodal large language model are fixed and do not need to be updated. This invention uses the following loss function as the loss function Loss of the combinatorial optimization solver:

[0060] Loss=αL siamese +βL gnn +γL co

[0061] Among them, L siamese It is the loss function of the Siamese neural network, used to measure the incorrect predictions of small fragment pairs; L gnn L represents the binary cross-entropy loss function when the optimal graph neural network selects K maximum probabilities; co The parameters are calculated by the probability transformation of the optimal transport layer; α, β, and γ are three predefined hyperparameters, which can be set according to actual needs and are not restricted in this invention.

[0062] The following section details the parameter settings and training dataset construction methods for the Siamese neural network and the optimal graph neural network in the combinatorial optimization solver.

[0063] In the training process of Siamese neural networks, this invention uses the binary cross-entropy loss function as the loss function L of the Siamese neural network. siamese :

[0064] L siamese =-[ylogp t +(1-y)log(1-p t )]

[0065] Where, p t Let represent the probability of a predicted input mask pair matching, and y represent the true value of whether the mask pair matches. The parameter settings for the Siamese neural network during training are shown in Table 1.

[0066] Table 1. Parameter settings for Siamese neural networks

[0067]

[0068] To calculate the loss function of the optimal graph neural network on an undirected graph, this invention inputs the probability distribution of the probabilities of the small fragments corresponding to each vertex predicted by the graph convolutional network into the optimal transport layer (OT Layer), and uses the Gumbel-Softmax trick and the Sinkhorn approximation method to implement the optimal transport layer (OT Layer).

[0069] Table 2. Parameter settings for graph convolutional networks

[0070]

[0071]

[0072] When training the combinatorial optimization solver, the outline of the Dunhuang fragments is used as the dividing line segment. Each complete Dunhuang fragment is torn into several small Dunhuang fragments. Adjacent small Dunhuang fragments are used as pairs of positive samples, while non-adjacent small Dunhuang fragments are pairs of negative samples. The constructed positive and negative sample pairs are used as the training dataset. The training dataset contains 40,000 pairs of positive samples and 80,000 pairs of negative samples.

[0073] Specifically, due to the limited number of real pairwise matching images of Dunhuang manuscript fragments, this invention employs a simulated shredded dataset to train a high-performing Siamese neural network and an optimal graph neural network. The method for constructing this dataset is as follows: Figure 5 As shown.

[0074] In the process of constructing the simulated shredded dataset, the existing outlines of Dunhuang fragments are used as dividing lines. A complete Dunhuang fragment is torn into smaller fragments. Adjacent fragments can be used as positive sample pairs, while non-adjacent fragments are negative sample pairs. After selecting a set of mask fragments from the same torn mask, fragments from other masks are randomly added to form a set of small fragments containing candidate subsets that can be spliced.

[0075] This invention adjusts the composition of the training data, conducting experiments with different total amounts of training data and different ratios of positive and negative samples. The dataset composition, accuracy of the Siamese neural network (SNN) recognition results, and accuracy of the optimal graph neural network (GNN) in selecting the Top-K are shown in Table 3. Verification shows that when the total amount of training data is 120,000 pairs and the ratio of positive to negative samples is 1:2, both the SNN and the optimal GNN achieve the best results. Therefore, this embodiment uses this dataset composition to train the combinatorial optimization solver, obtaining all weight parameters in both the SNN and the optimal GNN.

[0076] Table 3. Training dataset used in this invention

[0077]

[0078]

[0079] To better demonstrate the specific implementation and technical effects of the present invention, the method for screening combinatorial Dunhuang fragments based on combinatorial optimization and multimodal large models shown in steps S1 to S7 of the above preferred implementation is applied to a specific example.

[0080] Example

[0081] In this embodiment, validation was conducted on a real dataset. The accuracy was tested using only the multimodal large language model GPT-4V, the combinatorial optimization solver of this invention, and the method of this invention. Tests were performed under several scenarios: correct candidate fragment distributions of 2×2 and 3×3, and the size of the set to be filtered was 10, 15, 20, and 25. The accuracy rates were calculated and confirmed by archaeologists, and the accuracy rates obtained for each method are shown in Table 4.

[0082] Table 4. Accuracy on Real Datasets

[0083]

[0084] The candidate splicing set obtained from this embodiment can be spliced ​​into some complete small fragment splicing results, some of which are as follows: Figure 8 As shown.

[0085] In summary, the method of this invention searches for all possible combinations of small Dunhuang fragments to be screened, and inputs the mask pairs corresponding to all pairs of fragments to be screened into a Siamese neural network with ResNet18 as a subnetwork to extract the feature vector of each fragment and the matching degree score of each pair of fragments. Using the feature vector of each fragment and the fragment pairs with matching degree scores greater than a threshold, an undirected graph is constructed. A graph neural network with an optimal transport layer is used to predict the selection probability of the fragment corresponding to each vertex, and the K results with the highest probabilities are selected to form a candidate splicing set. The fragments in the candidate splicing set are input into a multimodal large language model, and using predefined prompt words, incorrect results are selected. Simultaneously, the vertices and edges corresponding to the incorrect results are removed from the undirected graph. The undirected graph after removal is input again into the graph neural network with an optimal transport layer to obtain the final candidate splicing set composed of K results.

[0086] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A method for screening combinatorial Dunhuang fragments based on combinatorial optimization and a multimodal large model, characterized in that, Includes the following steps: S1. Obtain all possible pairs of fragments from the Dunhuang fragments to be screened, and extract the mask from the fragment pairs. Each fragment pair corresponds to a mask pair. S2. Obtain a trained ensemble optimization solver, which consists of a Siamese neural network, an optimal graph neural network, and a multimodal large language model. The optimal graph neural network is composed of a graph neural network and an optimal transport layer cascaded together. During training, the ensemble optimization solver only updates the parameters of the Siamese neural network and the optimal graph neural network, while the parameters of the multimodal large language model remain fixed. When updating the parameters of the optimal transport layer, the top K largest probabilities output by the optimal transport layer are maximized, and the remaining probabilities output by the optimal transport layer are minimized. S3. Each time, a mask pair is input into the twin neural network to obtain the feature vector of the Dunhuang fragment corresponding to the mask pair and the matching degree score of each mask pair; wherein, the twin neural network consists of two sub-networks for feature extraction and a classification module; S4. Take each feature vector obtained from the Siamese neural network as a vertex, add an edge between fragment pairs whose matching degree score is greater than the preset matching degree threshold, and use the corresponding matching degree score as the weight of the edge. After all fragment pairs have been traversed, an undirected graph is obtained. S5. Input the undirected graph obtained in step S4 into the optimal graph neural network. First, the graph neural network predicts the initial probability of the Dunhuang fragments corresponding to each vertex in the undirected graph in the candidate splicing set. After the initial probability is passed through the optimal transmission layer, the optimal probability is obtained. Select the top K Dunhuang fragments with the highest probability values ​​in the optimal probability to form the candidate splicing set. S6. After combining two Dunhuang fragments in the candidate splicing set into a candidate Dunhuang fragment pair, input each candidate Dunhuang fragment pair and the pre-constructed prompt words into the multimodal large language model. The multimodal large language model scores each candidate Dunhuang fragment pair and outputs the score result of each candidate Dunhuang fragment pair. If the score result is less than the preset score threshold, the corresponding candidate Dunhuang fragment pair is added to the erroneous small fragment subset. S7. In the undirected graph obtained in step S4, all feature vector vertices corresponding to the subset of erroneous fragments and all edges connected to the feature vector vertices are removed to obtain a pruned undirected graph. The pruned undirected graph is then re-inputted into the optimal graph neural network to predict a new candidate splicing set containing K Dunhuang fragments, which is used for splicing Dunhuang fragments.

2. The method for screening combinatorial Dunhuang fragments based on combinatorial optimization and multimodal large model as described in claim 1, characterized in that, In the twin neural network, the first fragment mask in the i-th mask pair is input into the first sub-network to obtain the first feature vector of the Dunhuang fragment, and the second fragment mask in the i-th mask pair is input into the second sub-network to obtain the second feature vector of the Dunhuang fragment. The first feature vector and the second feature vector are concatenated and then input into the classification module to output the matching degree score of the i-th mask pair.

3. The method for screening combinatorial Dunhuang fragments based on combinatorial optimization and multimodal large model as described in claim 2, characterized in that, The first subnetwork and the second subnetwork have the same structure and the same weights.

4. The method for screening combinatorial Dunhuang fragments based on combinatorial optimization and multimodal large model as described in claim 1, characterized in that, The classification module is composed of a first linear layer, a ReLU activation function, a second linear layer, and a Sigmoid activation function cascaded in sequence.

5. The method for screening combinatorial Dunhuang fragments based on combinatorial optimization and multimodal large model as described in claim 1, characterized in that, The matching score is between 0 and 1, where 0 indicates that the mask pair cannot be matched and 1 indicates that the mask pair has the highest matching degree.

6. The method for screening combinatorial Dunhuang fragments based on combinatorial optimization and multimodal large model as described in claim 1, characterized in that, The score threshold is set to 60.

7. The method for screening combinatorial Dunhuang fragments based on combinatorial optimization and multimodal large model as described in claim 1, characterized in that, The matching degree threshold is 0.

2.

8. The method for screening combinatorial Dunhuang fragments based on combinatorial optimization and multimodal large model as described in claim 1, characterized in that, The multimodal large language model adopts the GPT-4V model.

9. The method for screening combinatorial Dunhuang fragments based on combinatorial optimization and multimodal large model as described in claim 1, characterized in that, The graph neural network described uses a graph convolutional network (GCN).

10. The method for screening combinatorial Dunhuang fragments based on combinatorial optimization and multimodal large model as described in claim 1, characterized in that, When training the combinatorial optimization solver, the outline of the Dunhuang fragments is used as the dividing line segment. Each complete Dunhuang fragment is torn into several small Dunhuang fragments. Adjacent small Dunhuang fragments are used as pairs of positive samples, while non-adjacent small Dunhuang fragments are pairs of negative samples. The constructed positive and negative sample pairs are used as the training dataset, which contains 40,000 pairs of positive samples and 80,000 pairs of negative samples.

Images