A model training method and system for archaeological relics

By constructing multimodal data sets and training the archaeological cultural relics data and inference combined with user problems, the existing models have solved the problems of insufficient scale and quality of data sets, difficulty in integrating multimodal information, and insufficient utilization of prior knowledge in archaeological cultural relics research, and more efficient and accurate cultural relics information processing and analysis are achieved.

CN119443284BActive Publication Date: 2025-04-29海看网络科技(山东)股份有限公司 +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202510024663.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2025-01-08
Publication Date
2025-04-29
Estimated Expiration
2045-01-08

AI Technical Summary

Technical Problem

In the research of archaeological relics, existing deep learning models face problems such as limited data set scale, uneven data quality, difficulty in integrating multimodal information, and insufficient utilization of prior knowledge, resulting in insufficient training effect and interpretability of the model.

Method used

By processing the original archaeological cultural relics data, an archaeological cultural relics knowledge base and multimodal data set are constructed, multimodal large models are used for training, and inference is carried out in combination with user problems. Specific steps include identifying and parsing the original document data, acquiring image-text pairs based on location and content alignment, building high-quality multimodal data sets, and fine-tuning pre-trained large models to enable them to better understand and generate natural language descriptions related to archaeological artifacts.

Benefits of technology

It realizes efficient processing and analysis of archaeological cultural relics data, improves the training effect and interpretability of the model, can more accurately dig up the historical information contained in cultural relics, and provides more reliable and academic value cultural relics information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119443284B_ABST
    Figure CN119443284B_ABST
Patent Text Reader

Abstract

The present invention discloses a model training method and system for archaeological relics, which mainly relates to the field of artificial intelligence technology. It includes: processing original archaeological relics data, constructing an archaeological relics knowledge base, and establishing a multimodal data set; training a multimodal large model based on the established multimodal data set and a public multimodal data set of relics; and reasoning based on user questions and in combination with the archaeological relics knowledge base. The beneficial effect of the present invention is that it can better process and analyze archaeological relics data, and provide a sustainable model training method for archaeological relics research.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model training method and system for archaeological relics. Background Art

[0002] With the continuous deepening of archaeological research, the number and variety of archaeological relics are huge, which puts higher requirements on the research of relics. Traditional manual analysis methods are inefficient and difficult to fully and accurately mine the historical information contained in relics. In recent years, artificial intelligence technology, especially deep learning and large model technology, has made significant progress in image recognition, natural language processing and other fields, providing new ideas and methods for the study of archaeological relics.

[0003] The application of existing deep learning models in archaeological research still faces many challenges. First, the scale of archaeological datasets is limited, and the data quality is uneven. Most of them lack sufficient annotations and diversity, which limits the training effect of the model. Secondly, archaeological relics contain rich multimodal information such as vision and text. How to effectively integrate this information, construct multimodal datasets and train corresponding models is a key problem that needs to be solved urgently. In addition, archaeological research usually requires the combination of historical documents, archaeological excavation information and other knowledge. Existing deep learning models are difficult to effectively use this prior knowledge, resulting in insufficient interpretability and credibility of the model.

[0004] Therefore, there is an urgent need for an efficient and accurate multimodal dataset construction method and model training method that can make full use of the multimodal information of archaeological relics, combine prior knowledge, improve the efficiency and accuracy of archaeological research, and provide strong support for cultural relic protection and cultural inheritance, so as to better process and analyze archaeological data and improve the depth and breadth of research. Summary of the invention

[0005] The purpose of the present invention is to provide a model training method and system for archaeological relics, which can better process and analyze archaeological relics data and provide a sustainable model training method for archaeological relics research.

[0006] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0007] In one aspect, the present invention provides a model training method for archaeological relics, comprising the following steps:

[0008] S1: Process the original archaeological artifact data, build an archaeological artifact knowledge base, and establish a multimodal dataset;

[0009] S2: training a multimodal large model based on the multimodal dataset established in step S1 and the publicly available multimodal dataset of cultural relics;

[0010] S3: Reasoning based on user questions and the archaeological artifacts knowledge base.

[0011] Preferably, the step S1 comprises the following steps:

[0012] S11: Identify original document data and parse the original document;

[0013] S12: segmenting the parsed text content;

[0014] S13: Obtaining archaeological artifact image-text pairs based on position alignment;

[0015] S14: Use the multimodal large model to extract the image content, and calculate the text similarity through the Caps-BiLSTM model to obtain the archaeological relics image-text pairs based on content alignment. The two parts of the image-text pairs constitute a partial archaeological relics multimodal dataset;

[0016] S15: The original text and vectorized representation of the parsed and segmented archaeological artifact data are stored in the archaeological artifact knowledge base.

[0017] Preferably, in step S11, the original document includes a word format document and a PDF format document;

[0018] The parsing of the original document includes:

[0019] For word format documents, use python-docx to traverse each paragraph, check whether there is an image in the paragraph, if there is an image, save the image and extract the text of the previous and next paragraphs;

[0020] For PDF documents, use pdfplumber to obtain the objects corresponding to the text and images on each page. The text object LTTextBox represents a set of text block rectangular areas. Use this object to obtain the bounding box coordinates of the text. LTImage represents an image object. Use this object to obtain the bounding box coordinates of the image and set the offset to extract the text above and below the image coordinates.

[0021] Preferably, in step S11, the original document includes a word format document and a PDF format document;

[0022] The parsing of the original document includes:

[0023] For word format documents, use python-docx to traverse each paragraph, check whether there is an image in the paragraph, if there is an image, save the image and extract the text of the previous and next paragraphs;

[0024] For PDF documents, use pdfplumber to obtain the objects corresponding to the text and images on each page. The text object LTTextBox represents a set of text block rectangular areas. Use this object to obtain the bounding box coordinates of the text. LTImage represents an image object. Use this object to obtain the bounding box coordinates of the image and set the offset to extract the text above and below the image coordinates.

[0025] Preferably, in step S13, the obtaining of the archaeological cultural relic image-text pair based on position alignment is specifically:

[0026] According to the bounding box coordinates of each image obtained during parsing, an offset is set to extract the text paragraphs above and below the image, forming an archaeological artifact image-text pair based on position alignment.

[0027] Preferably, the step S14 is specifically:

[0028] Each image obtained during parsing is input into the multimodal large model to obtain the text of the multimodal large model's understanding of the image content, and the text content output by the multimodal large model is converted into With segmented content Convert into sentences After input The model calculates text similarity. If two vectors are similar, then The value tends to 1. If the two vectors are not similar, then The value is close to 0, and the calculation formula is as follows:

[0029]

[0030] in, and Represents two sentences, It is a model The last hidden state of the sentence global feature will be the highest score The paragraphs are matched as the text corresponding to the pictures to form archaeological cultural relics picture-text pairs based on content alignment, and the archaeological cultural relics picture-text pairs based on content alignment are combined with the archaeological cultural relics picture-text pairs based on content alignment to constitute a partial archaeological cultural relics multimodal dataset.

[0031] Preferably, the step S15 is specifically:

[0032] The bce-embedding model is used in the knowledge base to convert the parsed text paragraphs into vector representations. Taking the content of each file as the unit, the original archaeological cultural relic data and the vectorized data representation are stored in the archaeological cultural relic knowledge base.

[0033] Preferably, the step S2 comprises:

[0034] Process the resulting multimodal dataset into a suitable fine-tuning format:

[0035] Fine-tune the model using the training data.

[0036] Preferably, the step S3 comprises:

[0037] S31: User input processing and model output generation;

[0038] S32: inputting the responses generated by the model into the archaeological artifact knowledge base for matching;

[0039] S33: Conducting confidence assessment by hybrid strategy retrieval in archaeological artifact knowledge base.

[0040] On the other hand, a training system based on the above-mentioned model training method of archaeological relics is provided, comprising:

[0041] The data processing module is used to: process the original archaeological cultural relic data, build an archaeological cultural relic knowledge base, and establish a multimodal data set;

[0042] A model training module is used to: train a multimodal large model according to the multimodal dataset established in the steps and the public multimodal dataset of cultural relics;

[0043] The model reasoning module is used to perform reasoning based on user questions and in combination with the archaeological artifact knowledge base.

[0044] Compared with the prior art, the beneficial effects of the present invention are:

[0045] The present invention provides a method for constructing a multimodal dataset of archaeological relics and training the model based on a multimodal large model. The method improves the storage, retrieval and reasoning capabilities of archaeological relics information through systematic steps to meet the needs of academic research and public education:

[0046] 1. Identify and parse the original archaeological artifact document data, use the tool library to locate the image position in the document and segment the document data, align the artifact image and related text in the document by position and content, and ensure the accuracy and consistency of the data;

[0047] 2. The aligned image and text pairs are processed by the system to construct a high-quality multimodal data set. The text data is cleaned and vectorized, and the original text and the vectorized text representation are stored in the knowledge base for subsequent retrieval and verification;

[0048] 3. Use the constructed multimodal archaeological artifact dataset to conduct joint training with the existing public multimodal archaeological artifact dataset. By fine-tuning the pre-trained large model, it can better understand and generate natural language descriptions related to archaeological artifacts. During the training process, the model not only learns the relationship between text and images, but also improves its knowledge and ability in the field of archaeological artifacts;

[0049] 4. During the user questioning stage, the system can perform intelligent reasoning based on the user input and the archaeological cultural relic knowledge base. Through a hybrid retrieval strategy, the system quickly locates relevant cultural relic information and verifies the answer based on the set confidence level. If the generated answer does not meet the expected accuracy, it will be handed over to human experts for judgment to ensure that the information provided is reliable and has academic value. The correct results given by the experts and the user input questions are combined into question-answer pairs, and the question-answer pairs are processed. The processed data is expanded to the multimodal data set of archaeological cultural relics and the archaeological cultural relic knowledge base to continuously improve the accuracy and reliability of the system.

[0050] In summary, the multimodal dataset construction and model training method based on the multimodal large model provided by the present invention systematically organizes a large amount of information in the field of archaeological relics, and can be applied to the fields of archaeological research, museum exhibitions, education and training through continuous optimization and iteration. It provides researchers and the public with accurate and rich cultural relic information, promotes the development of archaeological disciplines through intelligent reasoning and retrieval, and enhances the public's awareness of cultural heritage cognition and protection. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flow chart of the method of the present invention;

[0052] Figure 2 It is a flowchart of the original archaeological cultural relic data processing of the present invention;

[0053] Figure 3 It is a flowchart of the image-text alignment and data set construction of the present invention;

[0054] Figure 4 It is a multi-modal large model training and reasoning flow chart of the present invention;

[0055] Figure 5 is a similarity calculation flow chart based on content alignment of the present invention;

[0056] Figure 6 is a flow chart of the knowledge base hybrid retrieval strategy of the present invention;

[0057] Figure 7 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0058] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the application equally.

[0059] In the present invention, terms such as "upper", "lower", "left", "right", "front", "back", "vertical", "horizontal", "side", "bottom", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings. They are relational words determined only for the convenience of describing the structural relationships of the various parts or elements of the present invention, and do not specifically refer to any part or element in the present invention and should not be understood as limitations on the present invention.

[0060] Example:

[0061] like Figure 1 As shown, this embodiment provides a model training method for archaeological relics, comprising the following steps:

[0062] S1: Processing of original archaeological artifact data: building an archaeological artifact knowledge base and multimodal datasets;

[0063] S2: Multimodal large model training: Use the obtained multimodal dataset and the public multimodal dataset of cultural relics for training;

[0064] S3: Multimodal large model reasoning: Reasoning based on user questions combined with the archaeological artifacts knowledge base;

[0065] The specific details of step S1 are as follows Figure 2 , Figure 3 As shown, the data processing steps mainly include the following:

[0066] S11: Identify the original document data and parse the original document using a tool;

[0067] S12: segmenting the parsed text content;

[0068] S13: Obtaining archaeological artifact image-text pairs based on position alignment;

[0069] S14: Use the multimodal large model to extract the image content, and calculate the text similarity through the Caps-BiLSTM model to obtain the archaeological relics image-text pairs based on content alignment. The two parts of the image-text pairs constitute a partial archaeological relics multimodal dataset;

[0070] S15: storing the original text and vectorized representation of the parsed and segmented archaeological cultural relic data in an archaeological cultural relic knowledge base;

[0071] The original document data is identified and the original document is parsed using a tool, specifically:

[0072] Original documents usually include two categories: Word format and PDF format. Word format is generally the original file, and PDF file is a protected file. Use python-docx and pdfplumber third-party libraries to parse Word files and PDF files respectively.

[0073] For Word files, use python-docx to traverse each paragraph, check whether there is a picture in the paragraph, and if there is a picture, save the picture and extract the text of the previous and next paragraphs.

[0074] For PDF files, use pdfplumber to get the objects corresponding to the text and images on each page. The text object LTTextBox represents a set of text block rectangular areas. This object can be used to obtain the bounding box coordinates of the text. LTImage represents an image object. The image can be in JPEG or other formats. This object can be used to obtain the bounding box coordinates of the image. By setting an offset, you can extract the text above and below the image coordinates.

[0075] The parsed text content is segmented, specifically:

[0076] When parsing the text content, the main purpose is to better organize the information and relate the image to other text content. Segmentation is done according to a logical structure (e.g. chapters, sections) or based on the relevance of artifacts as indicated by the title.

[0077] For the content in the Word document, the python-docx library has been used to obtain the content of each paragraph and whether it contains images. If there is an image before or after a paragraph, the paragraph is considered as a separate part because it may be related to the image content. If there is no image in the text context, it is segmented according to the title level (such as first-level title, second-level title, etc.), and the document data is processed to form structured data.

[0078] For the content in the PDF document, the pdfplumber library is used to segment the content based on the layout information on the page. For each page, the positions of all the images on the page can be determined first, and then the page content can be divided into several regions based on these positions. Each region may include one or more images and the text surrounding them.

[0079] Get the archaeological artifact image and text pairs based on position alignment, specifically:

[0080] Image-text pair based on position alignment: First, the bounding box coordinates of each image are obtained during parsing, including the positions of its upper left corner and lower right corner. Next, an offset is set to extract the text paragraphs above and below the image, which are usually descriptions of the image or related information. When the document content structure is clear, the method based on position alignment of image and text effectively ensures the contextual consistency of the image and text pair.

[0081] The multimodal large model is used to extract the image content, and the text similarity is calculated through the Caps-BiLSTM model to obtain the archaeological relics image-text pairs based on content alignment. The two parts of the image-text pairs constitute part of the archaeological relics multimodal data set, specifically:

[0082] Image-text pairing based on content alignment: Figure 5 As shown in the figure, each image obtained during parsing is input into the multimodal model to obtain the text of the multimodal model's understanding of the image content. With segmented content Convert into sentences After input The model calculates text similarity. If two vectors are similar, then The value tends to 1, if they are not similar, then The value is close to 0, and the calculation formula is as follows:

[0083]

[0084] in, express The target value of and Indicates two sentences. It is a model The last hidden state of the sentence global feature. The highest score The paragraphs are matched as the text corresponding to the pictures. The content-based image-text alignment method ensures the relevance of the content of the image-text pair. The above two parts of image-text pairs are combined to form a multimodal dataset of some archaeological relics.

[0085] The original text and vectorized representation of the parsed and segmented archaeological cultural relic data are stored in the archaeological cultural relic knowledge base, specifically:

[0086] The bce-embedding model is used in the knowledge base to convert the parsed text paragraphs into vector representations. Taking the content of each file as a unit, the original archaeological artifact data and the vectorized data representation are stored in the archaeological artifact knowledge base.

[0087] According to the multimodal large model training described in step S2: use the obtained multimodal dataset and the public cultural relics multimodal dataset for training. The details are as follows Figure 3 As shown, the data processing steps mainly include the following:

[0088] Process the resulting multimodal dataset into a suitable fine-tuning format;

[0089] Fine-tune the model using the training data;

[0090] The obtained multimodal dataset is processed into a suitable fine-tuning format, specifically:

[0091] The aligned image and text pairs in the previous steps are saved in the same folder. Data preprocessing is required to convert them into structured data required by the Qwen-VL model. Traverse all samples in the folder and put them into a list and save them in a json file. Each sample corresponds to a dictionary, including id and conversation, the latter of which is a list. The processed training data samples are as follows:

[0092] {

[0093] "id": "identity_12035",

[0094] "conversations": [

[0095] {

[0096] "from": "user",

[0097] "value": "Picture 12035: assets / 52007d606039d5ba.jpg\nWhat kind of cultural relic is in the picture? "

[0098] },

[0099] {

[0100] "from": "assistant",

[0101] "value": "The picture shows a diamond-shaped net-patterned painted pottery pot\nThe Neolithic Dawenkou culture mud red pottery. Diameter 7, belly 12.2, bottom diameter 4.9, height 16. Wide mouth, long neck, deep round belly, small flat bottom, belly covered with black and white double-channel diamond patterns."

[0102] } ]

[0104] }

[0105] in, , are special tokens that identify the address of the picture. The content of "value" can be expressed as Picture id: img_path\n{prompt}, where id indicates the image number in the conversation. "img_path" can be a local image or a network address, and "prompt" is the image-related content.

[0106] The model is fine-tuned using training data, specifically:

[0107] Using the training data obtained in the previous steps, the large model is fine-tuned to enable it to better generate responses to cultural relics content that meet the user's wishes. Fine-tuning can be done by using the generated data on the existing large model for additional training and adjusting the model parameters to improve its performance and adaptability. The full name of fine-tuning training is Parameter-Efficient Fine-tuning (PEFT), which refers to fine-tuning the basic large model with a small number of parameters, which can save costs.

[0108] Commonly used and efficient parameter fine-tuning include:

[0109] (1) FREE, i.e. parameter freezing, freezes some parameters of the original basic large model and only trains some parameters, so that the basic large model can be trained on a single card.

[0110] (2) LORA, freezes the pre-trained model weights and injects the trainable rank decomposition matrix into each weight of the Transformer layer, greatly reducing the number of trainable parameters for downstream tasks.

[0111] (3) P-Tuning: add prompt parameters to each layer of the transformer for fine-tuning.

[0112] Through the above steps, the image-text pair can be processed into structured data and converted into a multimodal training dataset. Then, the obtained partial multimodal training data is combined with the open source multimodal dataset to fine-tune the large model. Such a process can help generate natural language responses related to cultural relics to provide better answers and descriptions.

[0113] In step S3, according to the multimodal large model reasoning described in step S2: reasoning is performed based on the user's question and the archaeological cultural relics knowledge base. The specific details are as follows Figure 4 As shown, it mainly includes the following steps:

[0114] S31: User input processing and model output generation;

[0115] S32: inputting the responses generated by the model into the archaeological artifact knowledge base for matching;

[0116] S33: Confidence assessment by hybrid strategy retrieval in archaeological artifacts knowledge base;

[0117] Question-answer pairs with low confidence levels are evaluated by experts, and the correct answers are given. The results are then expanded to the multimodal dataset of archaeological relics and the knowledge base of archaeological relics.

[0118] Specifically:

[0119] The user input processing and model output generation are specifically as follows:

[0120] The user's question input can be text or a combination of text and images. The user's question is input into the trained multimodal large model, which generates answers based on the user's input and outputs preliminary text results.

[0121] The responses generated by the model are input into the archaeological cultural relics knowledge base for matching, specifically:

[0122] Due to the particularity of the field of archaeological relics, the answers expected by users have high requirements for the accuracy of knowledge, and key archaeological relics information must be traced back to the source of the data. Therefore, it is necessary to bring the answers generated by the model into the knowledge base for retrieval, quickly locate relevant cultural relics entries through a hybrid retrieval strategy, and sort the two types of query results according to relevance to select the results that best match the user's questions.

[0123] The hybrid strategy retrieval in the archaeological cultural relics knowledge base is performed to perform confidence assessment, specifically:

[0124] like Figure 6 As shown in the figure, the hybrid retrieval strategy combines traditional databases and vector databases, performs full-text retrieval and vector retrieval at the same time, and uses a cross encoder model to re-rank the results of the candidate results of the full-text keywords and vector search, and selects the best result that matches the user's question from the two types of query results. If the confidence of the best result is greater than the set confidence p, it means that the answer generated by the model is accurate, and the reply text is output and the relevant cultural relics entry in the knowledge base is given for reference. If the confidence is less than the set p value, it means that the answer generated by the model is inaccurate. At this time, the question entered by the user and the answer generated by the model are handed over to human experts for judgment.

[0125] The question-answer pairs with low confidence levels are evaluated by expert judgment, and the correct answers are given. The results are expanded to the multimodal dataset of archaeological relics and the knowledge base of archaeological relics, specifically:

[0126] The correct answer given by the expert and the question input by the user form a question-answer pair, and the question-answer pair is analyzed as follows: Figure 2The data processing shown is carried out, and the processed data is expanded to the multimodal data set of archaeological relics and the archaeological relics knowledge base to continuously improve the accuracy and reliability of the system.

[0127] like Figure 7 As shown, this embodiment also provides a training system based on the above-mentioned model training method of archaeological relics, including:

[0128] The data processing module is used to: process the original archaeological cultural relic data, build an archaeological cultural relic knowledge base, and establish a multimodal data set;

[0129] A model training module is used to: train a multimodal large model according to the multimodal dataset established in the steps and the public multimodal dataset of cultural relics;

[0130] The model reasoning module is used to perform reasoning based on user questions and in combination with the archaeological artifact knowledge base.

[0131] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A model training method for archaeological relics, characterized in that: The following steps are involved: S1: Processing the original archaeological artifact data, building an archaeological artifact knowledge base, and establishing a multimodal dataset includes the following steps: S11: Identify original document data and parse the original document; S12: segmenting the parsed text content; S13: Obtaining archaeological artifact image and text pairs based on position alignment, specifically: According to the bounding box coordinates of each image obtained during parsing, an offset is set to extract the text paragraphs above and below the image to form an archaeological artifact image-text pair based on position alignment; S14: Use the multimodal large model to extract the image content, and calculate the text similarity through the Caps-BiLSTM model to obtain the archaeological relics image-text pairs based on content alignment. The two parts of the image-text pairs constitute part of the archaeological relics multimodal data set, specifically: Input each image obtained during parsing into the multimodal large model to obtain the text of the multimodal large model's understanding of the image content, and output the text content of the multimodal large model. With segmented content Convert into sentences After input The model calculates text similarity. If two vectors are similar, then The value tends to 1. If the two vectors are not similar, then The value is close to 0, and the calculation formula is as follows: in, express The target value of and Represents two sentences, It is a model The last hidden state of the sentence global feature will be the highest score The paragraphs are matched as the text corresponding to the pictures to form the archaeological cultural relics picture-text pairs based on content alignment, and the archaeological cultural relics picture-text pairs based on content alignment are combined with the archaeological cultural relics picture-text pairs based on position alignment to form a partial archaeological cultural relics multimodal dataset; S15: storing the original text and vectorized representation of the parsed and segmented archaeological cultural relic data in an archaeological cultural relic knowledge base; S2: training a multimodal large model based on the multimodal dataset established in step S1 and the publicly available multimodal dataset of cultural relics; S3: Based on the user input question and combined with the archaeological artifact knowledge base, reasoning is performed, including: S31: User input processing and model output generation; S32: inputting the responses generated by the model into the archaeological artifact knowledge base for matching; S33: A hybrid retrieval strategy is used in the archaeological artifact knowledge base to perform confidence assessment. The hybrid retrieval strategy is specifically: combining the traditional database and the vector database, performing full-text retrieval and vector retrieval at the same time, combining the full-text keywords and the candidate results of the vector search using a cross-encoder model to re-rank the results, and selecting the best result that matches the user's question from the two types of query results.

2. The model training method for archaeological relics according to claim 1 is characterized in that: In step S11, the original document includes a word format document and a PDF format document; The parsing of the original document includes: For word format documents, use python-docx to traverse each paragraph, check whether there is an image in the paragraph, if there is an image, save the image and extract the text of the previous and next paragraphs; For PDF documents, use pdfplumber to obtain the objects corresponding to the text and images on each page. The text object LTTextBox represents a set of text block rectangular areas. Use this object to obtain the bounding box coordinates of the text. LTImage represents an image object. Use this object to obtain the bounding box coordinates of the image and set the offset to extract the text above and below the image coordinates.

3. A model training method for archaeological relics according to claim 2, characterized in that: In step S12, the parsed text content is segmented, specifically: For Word documents, if there is a picture before or after a paragraph of text, the paragraph of text is considered as a separate part. If there is no picture in the text context, it is segmented according to the title level, and the document data is processed to form structured data. For PDF documents, segmentation is performed based on the layout information on the page.

4. The model training method for archaeological relics according to claim 1 is characterized in that: The step S15 is specifically as follows: The bce-embedding model is used in the knowledge base to convert the parsed text paragraphs into vector representations. Taking the content of each file as the unit, the original archaeological cultural relic data and the vectorized data representation are stored in the archaeological cultural relic knowledge base.

5. The model training method for archaeological relics according to claim 1 is characterized in that: The step S2 comprises: Process the resulting multimodal dataset into a suitable fine-tuning format: Fine-tune the model using the training data.

6. A training system based on the model training method of archaeological relics as claimed in claim 1, characterized in that: include: The data processing module is used to: process the original archaeological cultural relic data, build an archaeological cultural relic knowledge base, and establish a multimodal data set; A model training module is used to: train a multimodal large model according to the multimodal dataset established in the steps and the public multimodal dataset of cultural relics; The model reasoning module is used to perform reasoning based on user questions and in combination with the archaeological artifact knowledge base.

Citation Information

Patent Citations

  • Model training method, system and equipment for program recommendation and medium

    CN118200673A

  • Image-text fusion question and answer method, device and equipment based on large language model

    CN118861229A