A Knowledge-Based Method and System for Classical Tibetan-Chinese Bilingual Ancient Books

Abstract

The present invention relates to the technical field of ancient book processing, and specifically discloses a method and system for knowledge-based processing of ancient Chinese-Tibetan bilingual books. The method includes: by introducing a Tibetan word expansion table and supporting pre-training tasks, constructing a multi-modal large model that supports both Chinese and Tibetan, enabling it to have cross-language text processing and image understanding capabilities; using the multi-modal large model for layout analysis to automatically identify and distinguish the picture area, Chinese text area, and Tibetan text area in ancient books; respectively performing entity and relationship extraction on the identified Chinese text and Tibetan text to extract the core elements and their interconnections; uniformly mapping the cross-language and cross-modal text and image information into a queryable knowledge graph to form a semantic and associated description of the content of ancient books. The present invention not only effectively solves the recognition problem brought about by the mixed arrangement of Chinese and Tibetan bilingual languages, but also greatly improves the accuracy and efficiency of automatic parsing and knowledge extraction, and promotes the inheritance and dissemination of academic resources.

Description

Technical Field

[0001] The present invention relates to the technical field of ancient book processing, and specifically, to a method and system for the intellectualization of Chinese-Tibetan bilingual ancient books. Background Technique

[0002] With the rise of digital humanities research and cross-language information processing technologies, the need for the digitization and intellectualization of ancient books has become increasingly prominent. Especially for Chinese-Tibetan bilingual ancient books, they contain extremely rich academic resources in aspects such as history, culture, religion, and medicine. However, these documents usually have problems such as complex layouts, diverse characters (including Chinese characters, Tibetan letters, as well as charts, illustrations, etc.), difficult-to-unify typesetting methods, and damaged fonts. In order to comprehensively and efficiently explore and utilize this batch of precious heritage, there is an urgent need for a new solution that integrates modern artificial intelligence, multi-modal large models, and cross-language processing. Based on this, the present invention is committed to researching and developing a method and system for constructing a Chinese-Tibetan bilingual ancient book knowledge graph based on a multi-modal large model, with the expectation of generating important value at the levels of academic research, cultural protection, and industrial applications.

[0003] Currently, the mainstream OCR technology mainly targets modern typesetting or single-language text scenarios. Facing the complex layouts in Chinese-Tibetan bilingual ancient books with interlaced rows and columns, mixed vertical and horizontal typesetting, and inserted illustrations, the recognition accuracy and block segmentation effect are poor, making it difficult to meet the subsequent in-depth processing requirements; most existing multi-modal or multi-language large models are pre-trained for high-resource languages such as English and Chinese, with insufficient support for low-resource languages such as Tibetan, and also lacking processing strategies for the specialized typesetting and diverse fonts of ancient books, resulting in greater difficulties in in-depth cross-language and cross-modal analysis; in addition, the ancient book digitization process often stays at simple text retrieval or image management, unable to automatically extract entities, concepts, and relationships in the literature, and even more difficult to form a multi-modal knowledge graph for in-depth interdisciplinary research. Summary of the Invention

[0004] The purpose of the present invention is to provide a method and system for the intellectualization of Chinese-Tibetan bilingual ancient books to solve the problems raised in the above background technique.

[0005] To achieve the above purpose, the present invention provides the following technical solutions:

[0006] A method for the intellectualization of Chinese-Tibetan bilingual ancient books, the method comprising:

[0007] S1. By introducing a Tibetan word expansion table and a supporting pre-training task, construct a multi-modal large model that supports both Chinese and Tibetan, enabling it to have cross-language text processing and image understanding capabilities;

[0008] S2. Use a multi-modal large model for layout analysis to automatically identify and distinguish the picture areas, Chinese text areas, and Tibetan text areas in ancient books, and separate the illustrations or photos to the picture server for subsequent separate processing and display;

[0009] S3. Perform entity and relationship extraction on the identified Chinese text and Tibetan text respectively, and extract the core elements and their interconnections. Among them, the core elements include characters, place names, time, medicinal materials, systems, etc.;

[0010] S4. Uniformly map the cross-language and cross-modal text and image information to a queryable knowledge base or knowledge graph to form a semantic and associative description of the content of ancient books.

[0011] As a further technical solution of the present invention, the method further includes data preparation:

[0012] Collect a large amount of Tibetan language materials, clean, segment or sub-letter segment them, and combine them with Chinese language materials for unified multi-modal pre-training; for the scanning of ancient book images, collect diverse images with different fonts, degrees of mutilation, and typesetting methods to enable the model to maintain robustness in complex scenarios.

[0013] As a further technical solution of the present invention, the steps of constructing a multi-modal large model that supports both Chinese and Tibetan by introducing a Tibetan word expansion table and a supporting pre-training task, and enabling it to have cross-language text processing and image understanding capabilities include:

[0014] Tibetan word list expansion: Add Tibetan syllables or common subsets of Tibetan words to the original word list of the large model, and randomly or incrementally initialize the corresponding embedding vectors;

[0015] Large model pre-training: Combine the language model and the image-text matching task to fine-tune or incrementally train on "ancient book scenarios + Chinese-Tibetan bilingual";

[0016] Supervised fine-tuning: Add a translation task head to the model, and perform mutual translation or bilingual alignment between Tibetan and Chinese when necessary to solve the mapping problem of the same entity in different languages. To improve the accuracy of cross-language translation and alignment, the following loss function examples can be introduced:

[0017]

[0018] Among them, represents the cross-entropy loss of Chinese-Tibetan translation, can be the cross-language vector alignment loss, and λ1 and λ2 are used to balance the relative weights of the translation and alignment tasks;

[0019] To address the problem of scarce data in low-resource languages, a small amount of real optical character recognition (OCR) corpus in Tibetan can be combined with automatically synthesized Tibetan data to improve the model coverage. An example of the loss function is as follows:

[0020]

[0021] Among them, represents the real low-resource Tibetan OCR corpus set, represents the automatically synthesized Tibetan data set, is the training loss. The parameter β ∈ [0, 1] is used to balance the proportion of real data and synthetic data in actual training, ensuring both the adaptation to real scenarios and the expansion of the diversity of Tibetan word forms, fonts, etc., and improving the coverage and robustness of the model in low-resource environments.

[0022] As a further technical solution of the present invention, the steps of using a multi-modal large model for layout analysis, automatically identifying and distinguishing the picture area, Chinese text area, and Tibetan text area in ancient books, and separating the illustrations or photos to the picture server for subsequent separate processing and display include:

[0023] Graphic and text input: Input an ancient book image of a certain page into the multi-modal large model;

[0024] Chunking result: The multi-modal large model outputs the corresponding segmentation mask or position box to distinguish the picture area, Chinese text area, and Tibetan text area;

[0025] Picture storage: Crop the picture area separately and save it to the server, and the picture can be referenced in the subsequent knowledge graph construction.

[0026] As a further technical solution of the present invention, the steps of respectively performing entity and relationship extraction on the identified Chinese text and Tibetan text, and extracting the core elements and their interconnections include:

[0027] 1), Chinese text: Obtain Chinese characters through OCR and use natural language processing (NLP) tools to identify entities {E h} and relationships {R h};

[0028] 2), Tibetan text: Use a large model to recognize the Tibetan character form to obtain a text sequence, then perform word segmentation or sub-syllable decomposition, and extract entities {E t} and relationships {R t}. If bilingual merging is required, the cosine similarity of similar entity vectors can be judged. The formula is as follows:

[0029]

[0030] When the similarity is greater than the threshold θ, they can be regarded as the same concept.

[0031] As a further technical solution of the present invention, the step of uniformly mapping cross - language and cross - modal text and image information into a queryable knowledge graph to form a semantic and associative description of the ancient book content includes:

[0032] Multi - modal data integration: Using a graph database, nodes are stored in the graph and connected to each other through relationships. The nodes include person names (Person), place names (Location), events (Event), illustration resources (ImageResource), etc., and the relationships include belonging to, occurring in, being located in, etc.;

[0033] Cross - language entity linking: If the entities represented by Tibetan and Chinese texts are the same, they are merged in the graph or a synonym link is established to facilitate cross - language retrieval by users, as shown in the following formula:

[0034] If sim(v t , v h ) > θ, then the two entities are merged;

[0035] Among them, vt and vh respectively represent the embedding vectors of Tibetan and Chinese entities, sim(·) is the cosine similarity function, and θ is the similarity threshold for determining whether to merge or link entities. When the similarity is greater than θ, it indicates that the Tibetan entity and the Chinese entity have a very high correspondence relationship and can be regarded as the same object, so as to perform unified processing or cross - language synonym linking in the knowledge graph;

[0036] Visualization and interface: Provide a visualization page or API to support researchers in conducting searches, inferences, and statistical analyses in the graph.

[0037] Another object of the present invention is to provide a knowledge - based system for Chinese - Tibetan bilingual ancient books, and the system includes:

[0038] A multi - modal large - model construction module, which is used to construct a multi - modal large model that supports both Chinese and Tibetan by introducing a Tibetan word - expansion table and a supporting pre - training task, so that it has the ability of cross - language text processing and image understanding;

[0039] A layout analysis and block recognition module, which is used to perform layout analysis using the multi - modal large model, automatically identify and distinguish the picture area, Chinese text area, and Tibetan text area in the ancient book, and separate the illustrations or photos to the picture server for subsequent separate processing and display;

[0040] The text recognition and entity relationship extraction module is used to perform entity and relationship extraction on the recognized Chinese and Tibetan texts respectively, and extract the core elements and their interconnections. Among them, the core elements include characters, place names, time, medicinal materials, systems, etc.;

[0041] The multi-modal knowledge graph construction module is used to uniformly map cross-language and cross-modal text and image information into a queryable knowledge base or knowledge graph, forming a semantic and associated description of the content of ancient books.

[0042] Compared with the prior art, the beneficial effects of the present invention are:

[0043] By constructing a multi-modal large model supporting Chinese and Tibetan, the present invention can accurately distinguish picture, Chinese and Tibetan regions in complex ancient book layouts, and efficiently complete text recognition and entity relationship extraction, thus significantly reducing the processing threshold for ancient books in low-resource languages. By jointly mapping the recognition results and picture information into a multi-modal knowledge graph, the multi-dimensional content such as culture, history and medicine contained in ancient books can be comprehensively displayed, realizing in-depth support for cross-language and cross-disciplinary research.

[0044] Compared with traditional methods, the present invention not only effectively solves the recognition problems brought by the mixed arrangement of Chinese and Tibetan, but also takes into account the particularities of complex ancient book layout, diverse fonts and interspersed illustrations, greatly improving the accuracy and efficiency of automatic parsing and knowledge extraction. In addition, through the two-way alignment and translation training of Chinese and Tibetan texts, researchers can freely query and compare the literature content in the two languages, promoting the inheritance and dissemination of academic resources, and providing a broader application prospect for digital humanities, museum informatization and cultural exchanges. Through the above innovative design, the present invention demonstrates a feasible and practical new method in the fields of ancient book digitization, knowledgeization and cross-language information processing, bringing significant improvement space to related scientific research, cultural relic protection and education work. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention.

[0046] Figure 1 It is a flowchart of a method for knowledgeization of Chinese and Tibetan bilingual ancient books. DETAILED DESCRIPTION OF THE INVENTION

[0047] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention clearer, the following further describes the present invention in detail with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0048] The present invention aims to use a multi-modal large model to perform graphic layout recognition, bilingual text parsing, and knowledge extraction on Chinese-Tibetan bilingual ancient books, and finally form a multi-modal knowledge graph to support interdisciplinary research. Its process includes: First, based on the existing Chinese multi-modal model, synchronous support for Chinese-Tibetan bilingual texts is achieved through "Tibetan word expansion table + language model pre-training + supervised translation task"; then, for the complex layout of ancient books, the picture, Chinese, and Tibetan regions are segmented, and the image part is stored separately for subsequent associated display; subsequently, OCR and entity relationship extraction are respectively performed on each text region, and Chinese-Tibetan translation or alignment is assisted when necessary; then, the identified cross-language entities and relationships are mapped to the graph structure (such as RDF or graph database), and multi-modal information such as illustrations, geographical coordinates, and time axis are linked; finally, this knowledge graph can be applied to research in fields such as culture, medicine, history, and literature, and can also be used for more in-depth analysis and display in digital humanities and museum systems.

[0049] Please refer to Figure 1 , an embodiment of the present invention provides a method for knowledge-based Chinese-Tibetan bilingual ancient books, and the method includes:

[0050] S1. Collect a large amount of Tibetan language materials, clean, segment, or sub-letter segment them, and then uniformly use them for multi-modal pre-training in combination with Chinese language materials; for the scanning of ancient book images, collect diverse images including different fonts, degrees of mutilation, and typesetting methods, so that the model can maintain robustness in complex scenarios.

[0051] S2. By introducing a Tibetan word expansion table and a supporting pre-training task, construct a multi-modal large model that supports both Chinese and Tibetan, enabling it to have cross-language text processing and image understanding capabilities;

[0052] Tibetan word list expansion: Add Tibetan syllables or a subset of common Tibetan words to the word list of the original large model, and randomly or incrementally initialize the corresponding embedding vectors;

[0053] Large model pre-training: Combine the language model (Chinese + Tibetan) with graphic-text matching tasks (such as text block detection in images, image-text alignment, etc.) to perform fine-tuning or incremental training on "ancient book scenarios + Chinese-Tibetan bilingual";

[0054] Supervised fine-tuning: Add a translation task head to the model, and perform mutual translation or bilingual alignment between Tibetan and Chinese when necessary to solve the mapping problem of the same entity in different languages. To improve the accuracy of cross-language translation and alignment, the following loss function examples can be introduced:

[0055]

[0056] Among them, represents the cross-entropy loss of Chinese-Tibetan translation, It can be the cross - language vector alignment loss (such as the negative logarithm of the vector cosine similarity), and λ1 and λ2 are used to balance the relative weights of the translation and alignment tasks;

[0057] Regarding the problem of lack of data in low - resource languages, a small amount of real OCR corpus can be combined with automatically synthesized Tibetan data to improve the model coverage. An example of its loss function is as follows:

[0058]

[0059] Among them, represents the real low - resource Tibetan OCR corpus set, represents the automatically synthesized Tibetan data set, is the training loss (such as OCR or language model loss). The parameter β ∈ [0, 1] is used to balance the proportion of real data and synthetic data in actual training, ensuring both the adaptation to the real scenario and the expansion of the diversity of Tibetan word forms, fonts, etc., and improving the model coverage and robustness in the low - resource environment.

[0060] S3. Use a multi - modal large model for layout analysis, automatically identify and distinguish the picture area, Chinese text area, and Tibetan text area in ancient books, and separate the illustrations or photos to the picture server for subsequent separate processing and display;

[0061] Graphic and text input: Input an ancient book image page into a multi - modal large model;

[0062] Chunking result: The multi - modal large model outputs the corresponding segmentation mask or position box to distinguish the picture area, Chinese text area, and Tibetan text area;

[0063] Picture storage: Crop and save the picture area separately to the server, and the picture can be referenced for subsequent knowledge graph construction.

[0064] S4. Perform entity and relationship extraction on the identified Chinese text and Tibetan text respectively, and extract the core elements and their inter - relationships. Among them, the core elements include people, place names, time, medicinal materials, systems, etc.;

[0065] 1). Chinese text: Obtain Chinese characters through OCR and use NLP tools (word segmentation, NER, relationship extraction) to identify entities {E h} and relationships {R h};

[0066] 2). Tibetan text: Use a large model to recognize the Tibetan word form to obtain the text sequence, then perform word segmentation or sub - syllable decomposition, and extract entities {E t} and relationships {R t}. If bilingual merging is required, the cosine similarity of similar entity vectors can be judged. The formula is as follows:

[0067]

[0068] When the similarity is greater than the threshold θ, they can be regarded as the same concept.

[0069] S5. Uniformly map cross - language and cross - modal text and image information into a queryable knowledge base or knowledge graph to form a semantic and associated description of the content of ancient books.

[0070] Multi - modal data integration: Using a graph database, store nodes in the graph and connect them to each other through relationships. The nodes include person names (Person), place names (Location), events (Event), illustration resources (ImageResource), etc., and the relationships include belonging to, occurring in, located in, etc.

[0071] Cross - language entity linking: If the entities represented by Tibetan and Chinese texts are the same, they are merged in the graph or a synonymous link is established to facilitate cross - language retrieval by users, as shown in the following formula:

[0072] If sim(v t , v h ) > θ, then merge the two entities;

[0073] Among them, vt and vh respectively represent the embedding vectors of Tibetan and Chinese entities (which can be obtained by a multi - modal large model or other cross - language models), sim(·) is the cosine similarity function, and θ is the similarity threshold for determining whether to merge or link entities. When the similarity is greater than θ, it indicates that the Tibetan entity and the Chinese entity have a very high correspondence relationship and can be regarded as the same object, so as to be uniformly processed or cross - language synonymously linked in the knowledge graph;

[0074] Visualization and interface: Provide a visualization page or API to support researchers in conducting searches, reasoning, and statistical analysis in the graph.

[0075] This graph can not only support in - depth research in fields such as culture, medicine, and history, but also provide accurate data support for applications such as digital humanities, museum informatization, and education popularization. By integrating low - resource language processing, complex layout parsing, and entity relationship extraction, the present invention realizes the efficient knowledge - based conversion of Chinese - Tibetan bilingual ancient books, significantly improving the deficiencies of existing methods in recognition accuracy, cross - language compatibility, and semantic structure mining, and is of great significance for promoting the digitization of ancient books and academic research.

[0076] Another object of the embodiments of the present invention is to provide a knowledge - based system for Chinese - Tibetan bilingual ancient books, and the system includes:

[0077] The multi-modal large model construction module is used to construct a multi-modal large model that supports both Chinese and Tibetan by introducing a Tibetan word expansion table and supporting pre-training tasks, enabling it to have cross-language text processing and image understanding capabilities;

[0078] The layout analysis and block recognition module is used to perform layout analysis using the multi-modal large model, automatically identify and distinguish the picture area, Chinese text area, and Tibetan text area in ancient books, and separate the illustrations or photos to the picture server for subsequent separate processing and display;

[0079] The text recognition and entity relationship extraction module is used to perform entity and relationship extraction on the recognized Chinese text and Tibetan text respectively, and extract the core elements and their interconnections. Among them, the core elements include people, place names, time, medicinal materials, systems, etc.;

[0080] The multi-modal knowledge graph construction module is used to uniformly map cross-language and cross-modal text and image information into a queryable knowledge base or knowledge graph, forming a semantic and associated description of the content of ancient books.

[0081] It should be noted that in this article, the term "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article, or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or elements inherent to such a process, method, article, or system. Without more limitations, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, article, or system including that element.

[0082] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structure or equivalent process transformation made using the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A knowledge-based method for ancient Tibetan-Chinese bilingual books, characterized in that The method includes: S1. By introducing a Tibetan word expansion table and a supporting pre-training task, a multi-modal large model that supports both Chinese and Tibetan is constructed, enabling it to have cross-language text processing and image understanding capabilities; S2. Using the multi-modal large model for layout analysis, automatically identifying and distinguishing the picture area, Chinese text area, and Tibetan text area in ancient books, and separating the illustrations or photos to the picture server for subsequent separate processing and display; S3. Performing entity and relationship extraction on the identified Chinese text and Tibetan text respectively, extracting the core elements and their interconnections, where the core elements include people, place names, time, medicinal materials, and systems; S4. Uniformly mapping the cross-language and cross-modal text and image information into a queryable knowledge graph to form a semantic and associated description of the content of ancient books; The step of constructing a multi-modal large model that supports both Chinese and Tibetan by introducing a Tibetan word expansion table and a supporting pre-training task, enabling it to have cross-language text processing and image understanding capabilities includes: Tibetan word list expansion: Adding Tibetan syllables or a subset of common Tibetan words to the original word list of the large model, and randomly or incrementally initializing the corresponding embedding vectors; Large model pre-training: Combining the language model and the image-text matching task, performing fine-tuning or incremental training on "ancient book scenarios + Chinese-Tibetan bilingual"; Supervised fine-tuning: Adding a translation task head to the model to perform mutual translation or bilingual alignment between Tibetan and Chinese to solve the mapping problem of the same entity in different languages. To improve the accuracy of cross-language translation and alignment, the following loss function is introduced: Among them, represents the cross-entropy loss for Chinese-Tibetan translation, which can be the cross-language vector alignment loss, and λ1 and λ2 are used to balance the relative weights of the translation and alignment tasks; To address the problem of scarce data for low-resource languages, combining real text recognition corpora with automatically synthesized Tibetan data to improve the model coverage, and its loss function is as follows: Among them, represents the real low-resource Tibetan OCR corpus, represents the automatically synthesized Tibetan dataset, is the training loss, and the parameter β ∈ [0, 1] is used to balance the proportion of real data and synthetic data in actual training.

2. The method for knowledge-based processing of ancient Tibetan-Chinese bilingual books according to claim 1, wherein The method further includes data preparation: Collecting a large amount of Tibetan language corpora, cleaning, segmenting, or sub-letter segmenting them, and uniformly using them for multi-modal pre-training in combination with Chinese language corpora; for the scanning of ancient book images, collecting diverse images with different fonts, degrees of mutilation, and layout methods to enable the model to maintain robustness in complex scenarios.

3. The knowledge-based method for Chinese-Tibetan bilingual ancient books according to claim 1, characterized in that The step of using the multi-modal large model for layout analysis, automatically identifying and distinguishing the picture area, Chinese text area, and Tibetan text area in ancient books, and separating the illustrations or photos to the picture server for subsequent separate processing and display includes: Graphic and text input: Inputting an ancient book image of a certain page into the multi-modal large model; Chunking result: The multi-modal large model outputs the corresponding segmentation mask or position box to distinguish the picture area, Chinese text area, and Tibetan text area; Picture storage: Separately cropping and saving the picture area to the server, and the picture can be referenced for subsequent knowledge graph construction.

4. A method for knowledge-based processing of ancient Tibetan-Chinese bilingual books according to claim 1, characterized in that The step of performing entity and relationship extraction on the identified Chinese text and Tibetan text respectively, extracting the core elements and their interconnections includes: 1) Chinese text: Obtain Chinese characters through OCR and use natural language processing tools to identify entities {E h} and relationships {R h}; 2) Tibetan text: Use a large model to recognize Tibetan glyphs, obtain a text sequence, then perform word segmentation or sub-syllable decomposition, and extract entities {E t} and relationships {R t}. If bilingual merging is required, perform a cosine similarity judgment on similar entity vectors; When the similarity is greater than the threshold θ, it can be regarded as the same concept.

5. A knowledge-based system for Chinese-Tibetan bilingual ancient books, characterized in that, The system includes: A multi-modal large model construction module for constructing a multi-modal large model that supports both Chinese and Tibetan by introducing a Tibetan word expansion table and a supporting pre-training task, enabling it to have cross-language text processing and image understanding capabilities; The layout analysis and block recognition module is used to perform layout analysis using a multi-modal large model, automatically identify and distinguish the picture area, Chinese text area, and Tibetan text area in ancient books, and separate the illustrations or photos to the picture server for subsequent separate processing and display; The text recognition and entity relationship extraction module is used to perform entity and relationship extraction on the recognized Chinese text and Tibetan text respectively, and extract the core elements and their interconnections. Among them, the core elements include people, place names, time, medicinal materials, and systems; The multi-modal knowledge graph construction module is used to uniformly map cross-language and cross-modal text and image information into a queryable knowledge graph, forming a semantic and associated description of the content of ancient books; The content of constructing a multi-modal large model that supports both Chinese and Tibetan by introducing a Tibetan word expansion table and a supporting pre-training task, enabling it to have cross-language text processing and image understanding capabilities includes: Tibetan word list expansion: Add Tibetan syllables or common subsets of Tibetan words to the original word list of the large model, and randomly or incrementally initialize the corresponding embedding vectors; Large model pre-training: Combine the language model with the text-image matching task to fine-tune or incrementally train "ancient book scenarios + Chinese-Tibetan bilingual"; Supervised fine-tuning: Add a translation task head to the model to perform mutual translation or bilingual alignment between Tibetan and Chinese to solve the mapping problem of the same entity in different languages. To improve the accuracy of cross-language translation and alignment, the following loss function is introduced: Among them, represents the cross-entropy loss for Chinese-Tibetan translation, which can be the cross-language vector alignment loss, and λ1 and λ2 are used to balance the relative weights of the translation and alignment tasks; To address the problem of scarce data for low-resource languages, combine real text recognition corpora with automatically synthesized Tibetan data to improve model coverage. The loss function is as follows: Among them, represents the real low-resource Tibetan OCR corpus, represents the automatically synthesized Tibetan dataset, is the training loss, and the parameter β ∈ [0, 1] is used to balance the proportion of real data and synthetic data in actual training.

Images