Oracle-bone inscription image recognition method, device, equipment and medium based on multimodal large model

Abstract

A method, device, equipment and medium for oracle bone inscription image recognition based on a multimodal large model, relating to the field of computer vision technology. Among them, the oracle bone inscription image recognition method first extracts oracle bone inscription single-character images from the OBIMD data set, and retrieves basic information from the mirror original oracle bone platform to form a single-character data set. Then, a first dialogue data set containing information such as text interpretation is generated using a large language model, and the Qwen2VL model is fine-tuned to obtain an initial recognition model. Then, a second fine-tuning is performed through the location information dialogue data set to obtain a positioning sequence recognition model, which can recognize characters and complete positioning annotations. Further, a modern Chinese translation dialogue data set is generated, and the model is fine-tuned for the third time to obtain a semantic recognition model, which can combine characters into sentences and translate. Finally, the oracle bone inscription image to be recognized is input, and the model outputs character identification, interpretation or combined sentences according to whether the image is a single word or multiple words, and translates it into modern Chinese.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an oracle bone inscription image recognition method, device, equipment and medium based on a multimodal large model. Background Art

[0002] The automatic recognition technology of oracle bone inscriptions faces major challenges. Due to their age, oracle bone inscriptions often have blurred handwriting, broken strokes, and missing fonts. These degradation problems seriously affect the accuracy of automatic recognition of oracle bone inscriptions. In addition, the scarcity of oracle bone inscription samples also leads to problems of small sample learning, zero sample learning, and imbalanced class samples, which increases the complexity of research.

[0003] Early research on oracle bone inscription recognition mainly relied on graph theory and topology theory to extract artificial features, laying the foundation for oracle bone inscription recognition. Subsequently, the introduction of machine learning algorithms made feature expression more abstract and advanced. In recent years, the development of deep learning technology has brought new breakthroughs in oracle bone inscription recognition, especially in handling complex image recognition tasks.

[0004] Nevertheless, existing methods still have significant limitations in dealing with the highly degraded problems of oracle bone images, the scarcity of samples, and the in-depth understanding of the semantics of oracle bone inscriptions, and are unable to accurately recognize oracle bone inscriptions. These challenges highlight the necessity of developing more advanced and comprehensive oracle bone inscription recognition technologies in order to improve recognition accuracy and deeply understand the semantic content of oracle bone inscriptions. Summary of the invention

[0005] The present invention provides a method, device, equipment and medium for oracle bone inscription image recognition based on a multimodal large model to improve at least one of the above-mentioned technical problems.

[0006] In a first aspect, the present invention provides an oracle bone inscription image recognition method based on a multimodal large model, which comprises steps S01 to S11.

[0007] S01. Obtain oracle bone fragment images from the OBIMD dataset, cut and crop them, and extract oracle bone inscriptions. At the same time, the basic information of a single oracle bone script is retrieved from the Jingyuan Oracle Bone Platform Then, maintain a unique character ID for each synonym , to form a single word data set .

[0008] S02, based on the single word data set, generate a first dialogue data set including word interpretation, definition, and component composition through a large language model .

[0009] S03, based on the first dialogue data set The Qwen2VL model is first fine-tuned to obtain an initial recognition model, wherein the initial recognition model can associate oracle bone inscription image features and semantic information.

[0010] S04. Based on the oracle bone rubbings, templates, and the positioning information and sequence information of each oracle bone character in the OBIMD dataset, a location information dialogue dataset is obtained.

[0011] S05, performing a second fine-tuning training on the initial recognition model according to the position information dialogue data set to obtain a positioning sequence recognition model. The positioning sequence recognition model can recognize oracle bone characters from an image and complete the positioning information and sequence annotation of the characters according to the position information of the characters.

[0012] S06, based on the location information dialogue data set and basic information of a single oracle bone character ,Through a large language model, a single oracle sentence and its corresponding modern Chinese translation are generated, and a modern Chinese translation dialogue dataset is obtained.

[0013] S07: Perform a third fine-tuning training on the positioning sequence recognition model according to the modern Chinese translation dialogue data set to obtain a semantic recognition model, wherein the semantic recognition model can combine the recognized characters into sentences in order and generate corresponding modern Chinese interpretations.

[0014] S08, obtaining the oracle bone script image to be recognized and inputting it into the semantic recognition model.

[0015] S09. The semantic recognition model determines whether the oracle bone script image is a single-word image or a multi-word image.

[0016] S10: When it is determined that the oracle bone inscription image is a single-character image, the visual features are extracted, and the position information is encoded, and then the visual and language modal information are fused and processed to output the character identifier. and textual interpretation.

[0017] S11, when it is determined that the oracle bone script image is a multi-character image, identifying the character identifier of each oracle bone script in turn and the position of oracle bone inscriptions in the image , and then combine these characters to form sentences and generate corresponding modern Chinese interpretations.

[0018] In an optional implementation, step S01 specifically includes steps S011 to S013.

[0019] S011. Read the original json file of the OBIMD data set, and crop the original oracle bone image according to the position field of the oracle bone inscription words in the OBIMD data set to obtain the oracle bone inscription word image. .

[0020] S012. Identify the characters in the original data set Images with oracle bone characters Association, get image text pair .

[0021] S013, according to the character identification Search for basic information corresponding to the characters in the Jingyuan Oracle Platform font library , forming a single word data set Among them, the basic information Including text interpretation , wide-style , strict affiliation ,part 、Prefix .

[0022] In an optional implementation, step S02 specifically includes steps S021 to S023.

[0023] S021. First, an initial prompt template is provided for the large language model to generate dialogue data, wherein the initial prompt template contains information of a single word data set and guiding principles for dialogue generation.

[0024] S022. Then, according to the dialogue data generated each time, adjust the prompt to guide the large language model to generate richer and more accurate dialogue content.

[0025] S023. Use the adjusted prompt to call the API of the large language model to batch generate the first dialogue dataset Among them, the first dialogue data set Contains text interpretation, definition and component composition.

[0026] In an optional implementation, step S03 is specifically as follows: when fine-tuning based on the Qwen2-VL model, firstly, the first dialogue data set is As input data, the image of a single oracle bone script is processed by Vision Encoder to generate visual tokens that the model can understand, while the basic information is semantically analyzed and encoded by the Qwen2VL model. By processing the image of a single oracle bone script and combining it with the relevant historical background and character interpretation, basic feature learning is performed to associate oracle bone script image features and semantic information. Among them, the model dynamically packages oracle bone script images of different resolutions. At the same time, when passing through the last layer of Vision Encoder, adjacent 2×2 visual tokens are compressed into a single token to reduce the number of tokens in the overall image.

[0027] In an optional implementation, step S04 is specifically as follows: based on the oracle bone rubbings, templates, and the identification, positioning information, and sequence information of each oracle bone character in the OBIMD data set, the location information dialogue data set of all oracle bone characters is obtained through the large language model. The location information dialogue data set is a formatted text, and the format is: <|object_ref_start|> target character <|object_ref_end|><|box_start|>(x1,y1),(x2,y2)<|box_end|>, (x1,y1),(x2,y2) respectively correspond to the coordinates of the lower left corner and the upper right corner of the detection box.

[0028] In an optional embodiment, step S05 is specifically: performing a second fine-tuning training on the initial recognition model according to the position information dialogue data set to obtain a positioning sequence recognition model. Among them, the positioning sequence recognition model can recognize oracle bone characters from the image, and complete the positioning information and sequence annotation of the characters according to the position information of the characters. During the second fine-tuning training, the model inputs the oracle bone image, and outputs the formatted text of the list of each character in the oracle bone image and the corresponding bounding box coordinates of the characters, the format is: <|object_ref_start|> target character <|object_ref_end|><|box_start|>(x1,y1),(x2,y2)<|box_end|>, (x1,y1),(x2,y2) respectively correspond to the coordinates of the lower left corner and the upper right corner of the detection box.

[0029] In an optional implementation, step S06 specifically includes step S061 to step S062.

[0030] S061. Based on the location information dialogue data set and the basic information of a single oracle bone character retrieved from the Jingyuan oracle bone platform , through the large language model, generate the combined oracle bone inscription sentence. Among them, the basic information Includes sentence structure information.

[0031] S062, based on the oracle bone inscription sentence and the basic information of the corresponding single oracle bone inscription , through a large language model, a modern Chinese translation dialogue dataset of the oracle bone inscription single sentence is generated. Among them, the basic information Includes the meaning, typing and component composition information of individual oracle bone characters.

[0032] In an optional implementation, step S07 specifically includes: performing a third fine-tuning training on the positioning sequence recognition model according to the modern Chinese translation dialogue data set to obtain a semantic recognition model. During the third fine-tuning training, the semantic recognition model combines multiple characters into sentences based on the information output during the second fine-tuning training, and then generates the corresponding modern Chinese interpretation.

[0033] In an optional implementation, step S10 specifically includes step S101 to step S102.

[0034] S101. When it is determined that the oracle bone script image is a single-word image, firstly extract visual features through Vision Encoder, and then use M-RoPE to encode position information.

[0035] S102, then the visual and language modal information is processed by multimodal fusion through Grouped Query Attention, and the final output contains character identifiers and the inferences from the literal interpretation.

[0036] In an optional implementation, step S11 specifically includes steps S111 to S114.

[0037] S111. When it is determined that the oracle bone inscription image is a multi-character image, the Dual Chunk Attention technology is used to process the long sequence and divide the image into manageable blocks.

[0038] S112, identifying the character identifier of each oracle bone script in sequence and the position of oracle bone inscriptions in the image .

[0039] S113. The model uses YARN technology to rescale the attention weights.

[0040] S114, according to the order of characters Combine to generate a complete sentence and generate the corresponding modern Chinese interpretation of the sentence.

[0041] In the second aspect, the present invention provides an oracle bone script image recognition device based on a multimodal large model, which includes a single-word data acquisition module, a first dialogue data acquisition module, a first fine-tuning training module, a position information dialogue data acquisition module, a second fine-tuning training module, a translation dialogue data acquisition module, a third fine-tuning training module, an input module, an image judgment module, a single-word processing module and a multi-word processing module.

[0042] The single-word data acquisition module is used to obtain oracle bone fragment images from the OBIMD data set, and perform cutting and cropping to extract oracle bone inscription single-word images. At the same time, the basic information of a single oracle bone script is retrieved from the Jingyuan Oracle Bone Platform Then, maintain a unique character ID for each synonym , to form a single word data set .

[0043] The first dialogue data acquisition module is used to generate a first dialogue data set including word interpretation, definition, and component composition based on the single word data set through a large language model. .

[0044] A first fine-tuning training module is used to adjust the first dialogue data set The Qwen2VL model is first fine-tuned to obtain an initial recognition model, wherein the initial recognition model can associate oracle bone inscription image features and semantic information.

[0045] The location information dialogue data acquisition module is used to obtain the location information dialogue data set based on the oracle bone rubbings, templates, and the positioning information and sequence information of each oracle bone character in the OBIMD data set.

[0046] The second fine-tuning training module is used to perform a second fine-tuning training on the initial recognition model according to the position information dialogue data set to obtain a positioning sequence recognition model. The positioning sequence recognition model can recognize oracle bone characters from an image and complete the positioning information and sequence annotation of the characters according to the position information of the characters.

[0047] A translation dialogue data acquisition module is used to obtain the translation dialogue data based on the location information dialogue data set and the basic information of a single oracle bone character. ,Through a large language model, a single oracle sentence and its corresponding modern Chinese translation are generated, and a modern Chinese translation dialogue dataset is obtained.

[0048] The third fine-tuning training module is used to perform a third fine-tuning training on the positioning sequence recognition model according to the modern Chinese translation dialogue data set to obtain a semantic recognition model. The semantic recognition model can combine the recognized characters into sentences in order and generate corresponding modern Chinese interpretations.

[0049] The input module is used to obtain the oracle bone script image to be recognized and input it into the semantic recognition model.

[0050] The image judgment module is used for the semantic recognition model to judge whether the oracle bone script image is a single-word image or a multi-word image.

[0051] The single-word processing module is used to extract visual features and encode position information when the oracle bone inscription image is determined to be a single-word image, and then fuse the visual and language modal information to output the character identifier. and textual interpretation.

[0052] A multi-word processing module is used to identify the character identifier of each oracle bone script in turn when it is determined that the oracle bone script image is a multi-word image. and the position of oracle bone inscriptions in the image , and then combine these characters to form sentences and generate corresponding modern Chinese interpretations.

[0053] In a third aspect, the present invention provides an oracle bone inscription image recognition device based on a multimodal large model, comprising a processor, a memory, and a computer program stored in the memory. The computer program can be executed by the processor to implement an oracle bone inscription image recognition method based on a multimodal large model as described in any paragraph of the first aspect.

[0054] In a fourth aspect, the present invention provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute a method for oracle bone script image recognition based on a multimodal large model as described in any paragraph of the first aspect.

[0055] By adopting the above technical solution, the present invention can achieve the following technical effects:

[0056] The oracle bone script image recognition method based on a multimodal large model of the present invention significantly improves the accuracy and depth of oracle bone script recognition. The oracle bone script image recognition method not only improves the recognition rate of a single oracle bone script character by integrating advanced visual-language models and knowledge in specific fields of oracle bone script, especially when dealing with degradation problems such as image blur, broken strokes or missing fonts, but also comprehensively analyzes the entire oracle bone script, including character positioning, sentence combination and interpretation generation. In addition, the method enhances the model's recognition ability for new glyphs and improves the generalization performance of the model through a two-stage fine-tuning strategy, so that it can flexibly adapt to small sample and zero-sample learning scenarios. These technological breakthroughs provide strong technical support for the automated research and translation of oracle bone scripts and promote the progress of paleography research. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the specific implementation methods of the present invention. It should be understood that the following drawings only show certain specific implementation methods of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1 The present invention is a flowchart of an oracle bone inscription image recognition method.

[0059] Figure 2 is a schematic diagram of step S04.

[0060] Figure 3 It is a schematic diagram of step S06.

[0061] Figure 4 It is a schematic diagram of step S08 to step S10. DETAILED DESCRIPTION

[0062] The technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0063] Example 1, please refer to Figures 1 to 4 The first embodiment of the present invention provides an oracle bone script image recognition method based on a multimodal large model, which can be performed by an oracle bone script image recognition device based on a multimodal large model (hereinafter referred to as: oracle bone script image recognition device). In particular, it is performed by one or more processors in the oracle bone script image recognition device to implement steps S01 to S11.

[0064] It is understandable that the oracle bone script image recognition device can be an electronic device with computing capabilities, such as a portable notebook computer, a desktop computer, a server, a smart phone or a tablet computer.

[0065] S01. Obtain oracle bone fragment images from the OBIMD dataset, cut and crop them, and extract oracle bone inscriptions. At the same time, the basic information of a single oracle bone script is retrieved from the Jingyuan Oracle Bone Platform Then, maintain a unique character ID for each synonym , to form a single word data set Preferably, step S01 specifically includes steps S011 to S013. Steps S011 to S013. The basic information of the oracle bone inscriptions is obtained by automated script preprocessing.

[0066] S011. Read the original json file of the OBIMD data set, and crop the original oracle bone image according to the position field of the oracle bone inscription words in the OBIMD data set to obtain the oracle bone inscription word image. .

[0067] The OBIMD (Oracle Bone Inscriptions Multi-modal Dataset) dataset is a dataset specifically for oracle bone inscription research. It contains multimodal information of oracle bone inscriptions, such as oracle bone images, rubbings, copies, and related text information. This dataset provides rich resources for the automatic recognition and research of oracle bone inscriptions, allowing researchers and developers to use this data to train and test oracle bone recognition models.

[0068] S012. Identify the characters in the original data set Images with oracle bone characters Association, get image text pair .

[0069] S013, according to the character identification Search for basic information corresponding to the characters in the Jingyuan Oracle Platform font library , forming a single word data set Among them, the basic information Including text interpretation , wide-style , strict affiliation ,part 、Prefix By designing a crawler program, according to Search for basic information corresponding to the characters in the Jingyuan Oracle Platform font library Finally, a complete set of single-word data containing oracle bone inscriptions is obtained. .

[0070] The cropped single-word images and the extracted basic information of the glyphs are formatted to ensure that each oracle bone inscription image The corresponding basic information Can be matched accurately. The specific processing includes and Stored in a standardized format, each synonymous glyph maintains a unique identifier: .

[0071] S02, based on the single word data set, generate a first dialogue data set including word interpretation, definition, and component composition through a large language model Preferably, Figure 2 As shown, step S02 specifically includes steps S021 to S023.

[0072] S021. First, provide an initial prompt template for the large language model to generate dialogue data. The initial prompt template contains a single word data set. Provides information and dialogue generation guidelines.

[0073] S022. Then, according to the dialogue data generated each time, adjust the prompt to guide the large language model to generate richer and more accurate dialogue content.

[0074] S023. Use the adjusted prompt to call the API of the large language model to batch generate the first dialogue dataset Among them, the first dialogue data set Contains text interpretation, definition and component composition.

[0075] In step S2, in order to ensure that the first conversation data set is generated The quality and diversity of the prompts are improved by using multiple rounds of iterations to interact with ChatGPT. First, an initial prompt template is designed, which contains a single-word data set. The information provided and the guiding principles for dialogue generation. Then, based on the results of each generation, the prompt is dynamically adjusted to guide ChatGPT to generate richer and more accurate dialogue content. Finally, the ChatGPT API is called with the adjusted prompt to batch the first dialogue dataset. .

[0076] Specifically, first provide GPT with oracle bone inscriptions single word images and basic information GPT organizes relevant knowledge based on these inputs and generates the first dialogue dataset in the form of simulated dialogue ,in Each member in corresponds to a specific and , each member contains one or more rounds of dialogue information, such as Figure 2 shown.

[0077] This iterative approach not only ensures the quality of the generated data, but also captures the diversity and complexity of oracle bone inscriptions, providing rich and accurate training materials for subsequent model training. In addition, through multiple rounds of interaction, we can also generate targeted data for specific oracle bone inscription features or difficulties, thereby improving the performance of the model in processing complex oracle bone inscriptions.

[0078] Prompt is designed to guide the large language model to generate the first dialogue dataset D1 related to Oracle image recognition. Prompt needs to contain enough information to stimulate the model to generate dialogues, while also following certain guidelines to ensure the quality and relevance of the generated dataset. The following is an example of setting up Prompt.

[0079] Initial Prompt template (Oracle image recognition dialogue generation):

[0080] Purpose:

[0081] Generate a conversational dataset related to oracle image recognition.

[0082] enter:

[0083] oracle-image dimg: [insert link or description of oracle image here].

[0084] Basic information of oracle bone inscriptions dinfo: [Fill in the basic information of oracle bone inscriptions here].

[0085] Text interpretation ddef: [Fill in the meaning of the oracle bone inscriptions here].

[0086] Wide-style clerical script dws: [Fill in the wide-style clerical script of oracle bone script here].

[0087] Strict official script dss: [Fill in the strict official script of oracle bone script here].

[0088] Component dcomp: [Fill in the component composition of oracle bone inscriptions here].

[0089] Head: [Fill in the head of the oracle bone script here].

[0090] Dialogue Generation Guidelines:

[0091] 1. The conversation should include descriptions and inquiries about the oracle image.

[0092] 2. The dialogue should involve the interpretation and historical background of the oracle bone inscriptions.

[0093] 3. The conversation should simulate the queries and responses that real users might have.

[0094] 4. A conversation should contain at least one round of query and response.

[0095] Sample dialogue:

[0096] User: What is this symbol? What does it mean?

[0097] Assistant: This symbol is the word "year" in oracle bone script. Its shape presents a curved line, like a person carrying a long stalk of wheat. This image depicts the scene of ancient farmers carrying ripe grains during the harvest. In oracle bone script, the word "year" was first used to represent the annual harvest season, so the origin of the word "year" is closely related to agriculture, symbolizing the maturity and harvesting process of crops.

[0098] Generate a dialogue:

[0099] User: [Here the large language model will generate the user’s question].

[0100] Assistant: [Here the large language model will generate the assistant’s answer].

[0101] Dynamically adjust Prompt: In practical applications, according to the results generated by the large language model, Prompt may need to be dynamically adjusted to improve the quality and diversity of the conversation. For example, if the generated conversation is too simple or lacks depth, Prompt can be adjusted to include more detailed background information or more complex query scenarios. The following is an example of an adjusted Prompt.

[0102] Adjusted Prompt template (Oracle Image Recognition Dialog Generation).

[0103] Purpose:

[0104] Generate a deeper and more specific Oracle image recognition conversation dataset.

[0105] enter:

[0106] oracle-image dimg: [insert link or description of oracle image here].

[0107] Basic information of oracle bone inscriptions dinfo: [Fill in the basic information of oracle bone inscriptions here].

[0108] Text interpretation ddef: [Fill in the meaning of the oracle bone inscriptions here].

[0109] Wide-style clerical script dws: [Fill in the wide-style clerical script of oracle bone script here].

[0110] Strict official script dss: [Fill in the strict official script of oracle bone script here].

[0111] Component dcomp: [Fill in the component composition of oracle bone inscriptions here].

[0112] Head: [Fill in the head of the oracle bone script here].

[0113] Dialogue Generation Guidelines:

[0114] 1. The conversation should include a detailed description and in-depth questioning of the oracle bone image.

[0115] 2. The dialogue should explore the evolution of oracle bone inscriptions, their cultural significance and historical background.

[0116] 3. Conversations should simulate expert-level queries and responses, involving specialized terminology and in-depth analysis.

[0117] 4. The conversation should include at least two rounds of query and response to demonstrate the complexity of the problem and the depth of the solution.

[0118] Generate a dialogue:

[0119] User: [Here the large language model will generate the user's question, which may involve grapheme evolution or cultural background].

[0120] Assistant: [Here the large language model will generate the assistant’s responses, providing in-depth analysis and explanations].

[0121] In this way, Prompt’s design and tuning help guide large language models to generate richer and more accurate conversational datasets, thereby improving the training results of Oracle’s image recognition models.

[0122] S03, based on the first dialogue data set The Qwen2VL model is first fine-tuned to obtain an initial recognition model, wherein the initial recognition model can associate oracle bone inscription image features and semantic information.

[0123] Preferably, step S03 is specifically as follows: when fine-tuning based on the Qwen2-VL model, firstly, the pre-processed oracle bone inscription image and the corresponding text description (ie, the first dialogue data set) are ) is used as input data. The image of a single oracle bone script is processed by Vision Encoder to generate visual tokens that the model can understand, while the basic information is semantically analyzed and encoded by the Qwen2VL model. By processing the image of a single oracle bone script and combining it with the relevant historical background and character interpretation, basic feature learning is performed to associate the oracle bone script image features and semantic information.

[0124] The model dynamically packages oracle bone images of different resolutions to optimize resource usage and reduce processing time. At the same time, when passing through the last layer of the Vision Encoder, adjacent 2×2 visual tokens are compressed into a single token to reduce the number of tokens in the overall image.

[0125] Specifically, based on the Qwen2-VL model, the first dialogue dataset that has been verified and sorted is used. Perform preliminary fine-tuning to give the model basic knowledge of oracle image features and semantic associations. It can be understood that Vision Encoder and Qwen2 are the existing encoders of the Qwen2-VL model. The network structure can be improved by those skilled in the art as needed.

[0126] S04. Based on the oracle bone rubbings, templates, and the positioning information and sequence information of each oracle bone character in the OBIMD dataset, a location information dialogue dataset is obtained.

[0127] Preferably, Figure 3 As shown in the first stage generated content in the upper right corner of , step S04 is specifically as follows: based on the oracle bone rubbings, templates, and the identification, positioning information and sequence information of each oracle bone character in the OBIMD dataset, the location information dialogue dataset of all oracle bone characters is obtained through the large language model. Among them, the location information dialogue dataset is formatted text. The format is: <|object_ref_start|> target character <|object_ref_end|><|box_start|>(x1,y1),(x2,y2)<|box_end|>. Among them, (x1,y1) and (x2,y2) correspond to the coordinates of the lower left corner and upper right corner of the detection box respectively.

[0128] In this embodiment, the position of the oracle bone characters in the image provided by OBIMD and order , design the first round of question-answering data, and fine-tune the model to have the ability to locate characters and combine sentences. Specifically, OBIMD provides an image of the entire oracle bone, including rubbings and copies, and contains the grounding information of each oracle bone character and its order in the sentence. The embodiment of the present invention uses this to construct a position information dialogue dataset that includes character recognition and position positioning, ensuring that the model can accurately recognize oracle bone characters and understand the position of each character in the sentence. At the same time, the sentence structure information in OBIMD supports the model to learn the syntactic rules of oracle bone characters, providing an important basis for generating complete sentences.

[0129] S05, performing a second fine-tuning training on the initial recognition model according to the position information dialogue data set to obtain a positioning sequence recognition model. The positioning sequence recognition model can recognize oracle bone characters from an image and complete the positioning information and sequence annotation of the characters according to the position information of the characters.

[0130] Preferably, step S05 is specifically as follows: performing a second fine-tuning training on the initial recognition model according to the position information dialogue data set to obtain a positioning sequence recognition model. Among them, the positioning sequence recognition model can recognize oracle bone characters from the image, and complete the positioning information and sequence annotation of the characters according to the position information of the characters. During the second fine-tuning training, the model inputs the oracle bone image, and outputs the formatted text of the list of each character in the oracle bone image and the corresponding bounding box coordinates of the characters. The format is: <|object_ref_start|> target character <|object_ref_end|><|box_start|>(x1,y1),(x2,y2)<|box_end|>. Among them, (x1,y1) and (x2,y2) correspond to the coordinates of the lower left corner and the upper right corner of the detection box respectively.

[0131] In step S5, the second fine-tuning training is dialogue fine-tuning, and the model learns how to recognize oracle bone characters from images and The information completes the grounding and sequential annotation of the characters. In the second fine-tuning training, the model learns how to accurately identify characters and locate their positions from oracle bone images through a large number of samples. Specifically, the model receives an oracle bone image containing multiple oracle bone characters as input, and then generates a list containing each character and its corresponding bounding box coordinates (as shown in the formatted text above).

[0132] S06, based on the location information dialogue data set and basic information of a single oracle bone character ,Through a large language model, a single oracle sentence and its corresponding modern Chinese translation are generated, and a modern Chinese translation dialogue dataset is obtained.

[0133] Preferably, Figure 3 As shown in the dialogue generated in the lower right corner, step S06 specifically includes steps S061 to S062.

[0134] S061. Based on the location information dialogue data set and the basic information of a single oracle bone character retrieved from the Jingyuan oracle bone platform , through the large language model, generate the combined oracle bone inscription sentence. Among them, the basic information Includes sentence structure information.

[0135] S062, based on the oracle bone inscription sentence and the basic information of the corresponding single oracle bone inscription , through a large language model, a modern Chinese translation dialogue dataset of the oracle bone inscription single sentence is generated. Among them, the basic information Includes the meaning, typing and component composition information of individual oracle bone characters.

[0136] The Jingyuan Oracle Platform contains the interpretation, definition, and component composition information of a single oracle bone script. Based on this, the present invention embodiment designs the third fine-tuning training prompt, which provides a combined oracle bone script sentence by calling the ChatGPT API. And the corresponding basic information , generate a modern Chinese translation dialogue dataset on sentence interpretation and modern Chinese translation for the third fine-tuning training.

[0137] This approach not only ensures that the model can accurately understand the meaning of oracle bone inscriptions, but also generates translations that conform to modern language habits, thereby improving the practicality and reliability of the model in actual applications.

[0138] The location information dialogue dataset and the modern Chinese translation dialogue dataset together constitute the first dialogue dataset This dataset not only contains the visual and semantic information of oracle bone inscriptions, but also covers the position and order of characters in sentences, as well as the interpretation of complete sentences and modern Chinese translation. With this comprehensive and detailed dataset, the model is able to better understand and process the complex structure and semantic content of oracle bone inscriptions.

[0139] S07: Perform a third fine-tuning training on the positioning sequence recognition model according to the modern Chinese translation dialogue data set to obtain a semantic recognition model, wherein the semantic recognition model can combine the recognized characters into sentences in order and generate corresponding modern Chinese interpretations.

[0140] Preferably, step S07 specifically comprises: performing a third fine-tuning training on the positioning sequence recognition model according to the modern Chinese translation dialogue data set to obtain a semantic recognition model. During the third fine-tuning training, the semantic recognition model combines multiple characters into sentences according to the information output during the second fine-tuning training, and then generates corresponding modern Chinese interpretations.

[0141] The third fine-tuning training of step S5 is dialogue fine-tuning, and the purpose of this fine-tuning is to combine the characters recognized by the model into complete sentences in order and generate corresponding modern Chinese interpretations. In the third fine-tuning training, the model uses the context information of the second fine-tuning training to respond to the second round of query requirements and combine and translate the oracle bone inscriptions. The model first combines these characters into complete oracle bone inscriptions sentences in the correct order, and then generates corresponding modern Chinese interpretations according to the grammatical rules and context information of oracle bone inscriptions. In this way, the model can fully understand and interpret the oracle bone inscriptions, thereby improving the model's understanding and translation capabilities of oracle bone inscriptions.

[0142] The training method of the semantic recognition model of oracle bone script in the embodiment of the present invention extracts oracle bone script images from the OBIMD data set, and combines the text information of the mirror original oracle bone platform to construct a comprehensive training data set. The basic information of oracle bone script is expanded through preprocessing and data set enhancement, so that the model has a preliminary understanding of text morphology. The method is based on the Qwen2-VL multimodal large model and uses a two-stage fine-tuning and knowledge tuning strategy to enhance the model's understanding of oracle bone script characters and their contextual relationships. The fine-tuning process is divided into two stages, targeting single character recognition and multi-character sentence understanding respectively.

[0143] The first stage of fine-tuning includes the first fine-tuning training, in which the model learns basic features by processing a single oracle bone script image and combining it with the relevant historical background and character interpretation. The dataset in this stage consists of a single or multiple rounds of dialogue, and the model reasoning is guided by the introduction of a thought chain, gradually deriving the character interpretation from the visual features of the character. This method enhances the model's ability to understand the image features of oracle bone script characters and enables the model to perform character recognition and preliminary interpretation in combination with background knowledge.

[0144] The second stage of fine-tuning includes the second fine-tuning training and the third fine-tuning training. The input in this stage is an oracle bone tablet image containing multiple characters. The second fine-tuning training allows the model to recognize each oracle bone character in the oracle bone tablet and determine their position in the image and the order of the characters in the sentence; the third fine-tuning training generates the interpretation of the whole sentence after combining these characters based on the character information recognized in the first round, and outputs its corresponding modern Chinese translation. Through two rounds of dialogue training, the model can not only accurately recognize and locate multiple oracle bone characters, but also understand the contextual relationship between characters and generate a complete sentence interpretation.

[0145] This two-stage fine-tuning method improves the model's performance in oracle bone inscriptions image recognition tasks, enabling the model to have multi-level capabilities from single-character recognition to whole-sentence understanding. It solves the complex image recognition and context understanding problems of oracle bone inscriptions, and provides strong support for the automated research and translation of oracle bone inscriptions.

[0146] The inference phase of the model, such as Figure 4 As shown, it includes steps S08 to S11.

[0147] S08, obtaining the oracle bone script image to be recognized and inputting it into the semantic recognition model.

[0148] S09. The semantic recognition model determines whether the oracle bone script image is a single-word image or a multi-word image.

[0149] S10: When it is determined that the oracle bone inscription image is a single-character image, the visual features are extracted, and the position information is encoded, and then the visual and language modal information are fused and processed to output the character identifier. Preferably, step S10 specifically includes step S101 to step S102.

[0150] S101. When it is determined that the oracle bone script image is a single-word image, firstly extract visual features through Vision Encoder, and then use M-RoPE to encode position information.

[0151] S102, then the visual and language modal information is processed by multimodal fusion through Grouped Query Attention (GQA), and the final output contains character identifiers and the inferences from the literal interpretation.

[0152] S11, when it is determined that the oracle bone script image is a multi-character image, identifying the character identifier of each oracle bone script in turn and the position of oracle bone inscriptions in the image , and then combine these characters to form a sentence, and generate the corresponding modern Chinese interpretation. Preferably, step S11 specifically includes steps S111 to S114.

[0153] S111. When it is determined that the oracle bone image is a multi-character image, the Dual Chunk Attention (DCA) technology is used to process the long sequence and divide the image into manageable blocks. DCA is mainly used to improve the long context processing performance, specifically by effectively capturing the relative position information of tokens within and between blocks, and significantly improving the model's processing ability for complex oracle bone images.

[0154] S112, identifying the character identifier of each oracle bone script in sequence and the position of oracle bone inscriptions in the image .

[0155] S113. The model uses YARN technology to rescale attention weights for better length extrapolation.

[0156] S114, according to the order of characters Combine to generate a complete sentence and generate the corresponding modern Chinese interpretation of the sentence.

[0157] The oracle bone inscription image recognition method based on a multimodal large model in this embodiment significantly improves the accuracy and depth of oracle bone inscription recognition by integrating the powerful capabilities of the multimodal large model with the specific domain knowledge of oracle bone inscriptions, and realizes efficient recognition and interpretation of oracle bone inscription images. By integrating advanced visual-language models and knowledge of the specific domain of oracle bone inscriptions, it not only improves the recognition rate of a single oracle bone inscription character, but also can accurately recognize a single oracle bone inscription character, especially when dealing with degradation problems such as blurred images, broken strokes or missing fonts, and can also fully parse the entire oracle bone inscription, including character positioning, sentence combination and interpretation generation.

[0158] In addition, the method enhances the model's ability to recognize new glyphs through a two-stage fine-tuning strategy. The model gradually masters the complex skills of oracle bone inscription recognition, improves the generalization performance of the model, and enables it to flexibly adapt to small sample and zero-sample learning scenarios. These technological breakthroughs provide strong technical support for the automated research and translation of oracle bone inscriptions and promote the progress of ancient philology research.

[0159] Embodiment 2: The present invention provides an oracle bone script image recognition device based on a multimodal large model, which includes a single word data acquisition module, a first dialogue data acquisition module, a first fine-tuning training module, a position information dialogue data acquisition module, a second fine-tuning training module, a translation dialogue data acquisition module, a third fine-tuning training module, an input module, an image judgment module, a single word processing module and a multi-word processing module.

[0160] The single-word data acquisition module is used to obtain oracle bone fragment images from the OBIMD data set, and perform cutting and cropping to extract oracle bone inscription single-word images. At the same time, the basic information of a single oracle bone script is retrieved from the Jingyuan Oracle Bone Platform Then, maintain a unique character ID for each synonym , to form a single word data set .

[0161] The first dialogue data acquisition module is used to generate a first dialogue data set including word interpretation, definition, and component composition based on the single word data set through a large language model. .

[0162] A first fine-tuning training module is used to adjust the first dialogue data set The Qwen2VL model is first fine-tuned to obtain an initial recognition model, wherein the initial recognition model can associate oracle bone inscription image features and semantic information.

[0163] The location information dialogue data acquisition module is used to obtain the location information dialogue data set based on the oracle bone rubbings, templates, and the positioning information and sequence information of each oracle bone character in the OBIMD data set.

[0164] The second fine-tuning training module is used to perform a second fine-tuning training on the initial recognition model according to the position information dialogue data set to obtain a positioning sequence recognition model. The positioning sequence recognition model can recognize oracle bone characters from an image and complete the positioning information and sequence annotation of the characters according to the position information of the characters.

[0165] A translation dialogue data acquisition module is used to obtain the translation dialogue data based on the location information dialogue data set and the basic information of a single oracle bone character. ,Through a large language model, a single oracle sentence and its corresponding modern Chinese translation are generated, and a modern Chinese translation dialogue dataset is obtained.

[0166] The third fine-tuning training module is used to perform a third fine-tuning training on the positioning sequence recognition model according to the modern Chinese translation dialogue data set to obtain a semantic recognition model. The semantic recognition model can combine the recognized characters into sentences in order and generate corresponding modern Chinese interpretations.

[0167] The input module is used to obtain the oracle bone script image to be recognized and input it into the semantic recognition model.

[0168] The image judgment module is used for the semantic recognition model to judge whether the oracle bone script image is a single-word image or a multi-word image.

[0169] The single-word processing module is used to extract visual features and encode position information when the oracle bone inscription image is determined to be a single-word image, and then fuse the visual and language modal information to output the character identifier. and textual interpretation.

[0170] A multi-word processing module is used to identify the character identifier of each oracle bone script in turn when it is determined that the oracle bone script image is a multi-word image. and the position of oracle bone inscriptions in the image , and then combine these characters to form sentences and generate corresponding modern Chinese interpretations.

[0171] Embodiment 3: The present invention provides an oracle bone inscription image recognition device based on a multimodal large model, comprising a processor, a memory, and a computer program stored in the memory. The computer program can be executed by the processor to implement an oracle bone inscription image recognition method based on a multimodal large model as described in any paragraph of Embodiment 1.

[0172] Embodiment 4: The present invention provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute a method for oracle bone script image recognition based on a multimodal large model as described in any paragraph of Embodiment 1.

[0173] In several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed apparatus and method can also be implemented in other ways. The apparatus and method embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the apparatus, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0174] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0175] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, ReadOnlyMemory), random access memory (RAM, RandomAccessMemory), disk or optical disk and other media that can store program code. It should be noted that in this article, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0176] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0177] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0178] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0179] The "first\second" mentioned in the embodiments is only to distinguish similar objects, and does not represent a specific order for the objects. It is understandable that the "first\second" can be interchanged with the specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0180] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for oracle bone inscription image recognition based on a multimodal large model, characterized in that: Include: The oracle bone images are obtained from the OBIMD dataset, and the images of the oracle bone characters are cut and cropped to extract the images. Meanwhile, the basic information of the individual oracle bone characters is retrieved from the original oracle bone platform. Then, a unique character identifier is maintained for each synonymous glyph to form a single-character data set. Based on the single word data set, a first dialogue data set including word interpretation, definition, and component composition is generated through a large language model; Performing a first fine-tuning training on the Qwen2VL model according to the first dialogue data set to obtain an initial recognition model; wherein the initial recognition model can associate oracle bone image features and semantic information; Based on the oracle bone rubbings, templates, and the location information and sequence information of each oracle bone character in the OBIMD dataset, we compiled the location information dialogue dataset. Performing a second fine-tuning training on the initial recognition model according to the position information dialogue data set to obtain a positioning sequence recognition model; wherein the positioning sequence recognition model can recognize oracle bone characters from an image and complete the positioning information and sequence annotation of the characters according to the position information of the characters; Based on the location information dialogue dataset and the basic information of a single oracle bone script, a single oracle bone script sentence and its corresponding modern Chinese translation are generated through a large language model to obtain a modern Chinese translation dialogue dataset; Performing a third fine-tuning training on the positioning sequence recognition model according to the modern Chinese translation dialogue data set to obtain a semantic recognition model; wherein the semantic recognition model can combine the recognized characters into sentences in order and generate corresponding modern Chinese interpretations; Acquire the oracle bone inscription image to be recognized and input it into the semantic recognition model; The semantic recognition model determines whether the oracle bone inscription image is a single-word image or a multi-word image; When it is determined that the oracle bone inscription image is a single-character image, the visual features are extracted and the position information is encoded, and then the visual and language modal information are fused and processed to output the character identification and the text interpretation; When it is determined that the oracle bone script image is a multi-character image, the character identifier of each oracle bone script and the position of the oracle bone script in the image are identified in turn, and then these characters are combined to form a sentence, and the corresponding modern Chinese interpretation is generated.

2. The oracle bone inscription image recognition method based on a multimodal large model according to claim 1, characterized in that: Obtain oracle bone images from the OBIMD dataset, cut and crop them, and extract the images of oracle bone characters; at the same time, retrieve the basic information of individual oracle bone characters from the mirror original oracle bone platform; Then, a unique character identifier is maintained for each synonymous glyph to form a single-word data set, including: Read the original json file of the OBIMD dataset, and crop the original oracle bone image according to the position field of the oracle bone inscription words in the OBIMD dataset to obtain the oracle bone inscription word image; Associating the character identifiers in the original data set with the oracle bone inscription single word images to obtain image-text pairs; According to the character identifier, the basic information corresponding to the character is retrieved in the Jingyuan Oracle Platform character library to form a single-character data set; wherein, the basic information includes the character interpretation, loose-style official definition, strict-style official definition, components, and character heads.

3. The oracle bone inscription image recognition method based on a multimodal large model according to claim 1, characterized in that: Based on the single word data set, a first dialogue data set including word interpretation, definition, and component composition is generated through a large language model, specifically including: First, an initial prompt template is provided for the large language model to generate dialogue data; wherein the initial prompt template contains information provided by the single-word data set and guiding principles for dialogue generation; Then, based on the conversation data generated each time, the prompt is adjusted to guide the large language model to generate richer and more accurate conversation content; Using the adjusted prompt to call the API of the large language model to batch generate a first dialogue data set; wherein the first dialogue data set includes word interpretation, definition and component composition; The Qwen2VL model is first fine-tuned and trained according to the first dialogue data set to obtain an initial recognition model; wherein the initial recognition model can associate oracle bone image features and semantic information, specifically including: When fine-tuning based on the Qwen2VL model, first use the first dialogue dataset As input data, the image of a single oracle bone character is processed by the Vision Encoder to generate visual tokens that the model can understand, while the basic information is semantically analyzed and encoded by the Qwen2VL model. By processing the image of a single oracle bone character and combining it with the relevant historical background and character interpretation, basic feature learning is performed to associate the oracle bone image features and semantic information. Among them, the model dynamically packages oracle bone images of different resolutions. At the same time, when passing through the last layer of the Vision Encoder, adjacent 2×2 visual tokens are compressed into a single token to reduce the number of tokens in the overall image.

4. The oracle bone inscription image recognition method based on a multimodal large model according to claim 1, characterized in that: Based on the oracle bone rubbings, templates, and the location information and sequence information of each oracle bone character in the OBIMD dataset, we have compiled the location information dialogue dataset, which includes: Based on the oracle bone rubbings and templates in the OBIMD dataset, as well as the identification, positioning and sequence information of each oracle bone character, the location information dialogue dataset of all oracle bone characters is obtained through the large language model. Among them, the location information dialogue dataset is formatted text, and the format is: <|object_ref_start|> target character<|object_ref_end|><|box_start|>(x1,y1),(x2,y2)<|box_end|>, (x1,y1),(x2,y2) correspond to the coordinates of the lower left corner and upper right corner of the detection box respectively; Based on the location information dialogue dataset and the basic information of a single oracle bone script , through the large language model, generate oracle bone inscriptions and their corresponding modern Chinese translations, and obtain a modern Chinese translation dialogue dataset, including: Based on the location information dialogue dataset and the basic information of a single oracle bone script retrieved from the Jingyuan Oracle Bone Platform , through the large language model, generate the combined oracle bone inscription sentence; wherein the basic information Includes sentence structure information; Based on the oracle bone inscriptions and the basic information of the corresponding single oracle bone characters , through a large language model, generate a modern Chinese translation dialogue dataset of the oracle bone inscription single sentence; wherein the basic information Includes the meaning, typing and component composition information of individual oracle bone characters.

5. The oracle bone inscription image recognition method based on a multimodal large model according to claim 1, characterized in that: The initial recognition model is subjected to a second fine-tuning training according to the position information dialogue data set to obtain a positioning sequence recognition model; wherein the positioning sequence recognition model can recognize oracle bone characters from an image and complete the positioning information and sequence annotation of the characters according to the position information of the characters, specifically including: The initial recognition model is subjected to a second fine-tuning training according to the position information dialogue data set to obtain a positioning sequence recognition model; wherein the positioning sequence recognition model can recognize oracle bone characters from an image, and complete the positioning information and sequence annotation of the characters according to the position information of the characters; during the second fine-tuning training, the model inputs an oracle bone image, and outputs a formatted text of a list of coordinates of each character in the oracle bone image and corresponding bounding box of the characters, in the format of: <|object_ref_start|> target character <|object_ref_end|><|box_start|>(x1,y1),(x2,y2)<|box_end|>, (x1,y1),(x2,y2) respectively correspond to the coordinates of the lower left corner and the upper right corner of the detection box; The positioning sequence recognition model is subjected to a third fine-tuning training according to the modern Chinese translation dialogue data set to obtain a semantic recognition model; wherein the semantic recognition model can combine the recognized characters into sentences in order and generate corresponding modern Chinese interpretations, specifically including: The positioning sequence recognition model is subjected to a third fine-tuning training according to the modern Chinese translation dialogue data set to obtain a semantic recognition model; wherein, during the third fine-tuning training, the semantic recognition model combines multiple characters into sentences based on the information output during the second fine-tuning training, and then generates corresponding modern Chinese interpretations.

6. The oracle bone inscription image recognition method based on a multimodal large model according to claim 1, characterized in that: When it is determined that the oracle bone inscription image is a single-word image, the visual features are extracted and the position information is encoded, and then the visual and language modal information are fused and processed to output the character identification and text interpretation, which specifically includes: When it is determined that the oracle bone inscription image is a single-character image, the visual features are first extracted by Vision Encoder, and the position information is encoded by M-RoPE; Then, the visual and language modal information is fused through Grouped Query Attention to finally output the reasoning result including character identification and text interpretation. When it is determined that the oracle bone script image is a multi-character image, the character identifier of each oracle bone script and the position of the oracle bone script in the image are identified in turn, and then these characters are combined to form a sentence, and the corresponding modern Chinese interpretation is generated, which specifically includes: When it is determined that the oracle bone inscription image is a multi-character image, the Dual Chunk Attention technique is used to process the long sequence and divide the image into manageable blocks; Then, identify the character identifier of each oracle bone script in turn and the location of oracle bone inscriptions in the image; Then, the model rescales the attention weights using YARN technology; Finally, a complete sentence is generated based on the sequential combination of characters, and the corresponding modern Chinese interpretation of the sentence is generated.

7. An oracle bone inscription image recognition device based on a multimodal large model, characterized in that: Include: The single-word data acquisition module is used to obtain oracle bone images from the OBIMD data set, and perform cutting and cropping to extract oracle bone single-word images; at the same time, the basic information of a single oracle bone character is retrieved from the mirror original oracle bone platform; then, a unique character identifier is maintained for each synonymous glyph to form a single-word data set; A first dialogue data acquisition module, for generating a first dialogue data set including word interpretation, definition, and component composition according to the single word data set through a large language model; A first fine-tuning training module, used to perform a first fine-tuning training on the Qwen2VL model according to the first dialogue data set to obtain an initial recognition model; wherein the initial recognition model can associate oracle bone image features and semantic information; The location information dialogue data acquisition module is used to obtain the location information dialogue data set based on the oracle bone rubbings, templates, and the positioning information and sequence information of each oracle bone character in the OBIMD data set; A second fine-tuning training module is used to perform a second fine-tuning training on the initial recognition model according to the position information dialogue data set to obtain a positioning sequence recognition model; wherein the positioning sequence recognition model can recognize oracle bone characters from an image and complete the positioning information and sequence annotation of the characters according to the position information of the characters; A translation dialogue data acquisition module is used to generate a single oracle bone inscription sentence and its corresponding modern Chinese translation through a large language model based on the location information dialogue dataset and basic information of a single oracle bone inscription, and acquire a modern Chinese translation dialogue dataset; A third fine-tuning training module is used to perform a third fine-tuning training on the positioning sequence recognition model according to the modern Chinese translation dialogue data set to obtain a semantic recognition model; wherein the semantic recognition model can combine the recognized characters into sentences in order and generate corresponding modern Chinese interpretations; An input module, used for acquiring the oracle bone inscription image to be recognized and inputting it into the semantic recognition model; An image judgment module, used for the semantic recognition model to judge whether the oracle bone inscription image is a single-word image or a multi-word image; A single-word processing module is used to extract visual features and encode position information when it is determined that the oracle bone inscription image is a single-word image, and then fuse the visual and language modal information to output the character identification and text interpretation; The multi-word processing module is used to identify the character identifier of each oracle bone character and the position of the oracle bone character in the image in turn when it is determined that the oracle bone character image is a multi-word image, then combine these characters to form a sentence and generate the corresponding modern Chinese interpretation.

8. An oracle bone inscription image recognition device based on a multimodal large model, characterized in that: It comprises a processor, a memory, and a computer program stored in the memory; the computer program can be executed by the processor to implement a method for oracle bone script image recognition based on a multimodal large model as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute a method for oracle bone script image recognition based on a multimodal large model as described in any one of claims 1 to 6.

Images