A multi-modal archive intelligent labeling and knowledge graph construction method and system

Abstract

The application discloses a kind of multi-modal archives intelligent marking and knowledge graph construction method and system, it is related to the technical field of archives management, including the following steps, multi-modal archives data is digitally processed, to obtain standard archives text, and the standard archives text is processed, to obtain archive characteristic word;Based on the archive characteristic word, the standard archives text is carried out entity identification, to obtain archive entity, and the archive entity is carried out relationship extraction, to obtain entity relationship;Based on the archive entity and the entity relationship, entity alignment is carried out, to obtain uniform entity identification, and the uniform entity identification is carried out attribute supplement, to obtain entity attribute;Based on the uniform entity identification and the entity attribute, archives knowledge graph is constructed, solves the technical problem that traditional technology is extremely low in identifying archives type, extracting key information, and is difficult to guarantee data consistency.

Description

Technical Field

[0001] This invention relates to the field of archival management technology, and in particular to a method and system for intelligent annotation and knowledge graph construction of multimodal archives. Background Technology

[0002] In the current field of archival management, the number of multimodal archives is rapidly increasing with the progress of digitization. These archives include not only traditional scanned copies of paper documents and electronic Word and PDF files, but also non-textual forms such as historical photos, meeting recordings, and project videos. However, traditional archival processing methods are still mainly manual. Staff need to identify the type of each archive, extract key information, and manually classify and file them. This is not only time-consuming and labor-intensive, but also prone to incomplete information extraction or labeling errors due to differences in individual operating habits and professional knowledge. For example, when organizing research project archives before the decades, an archive had to scan paper contracts into image files, manually enter information such as project leaders, collaborating institutions, and research periods, and write written descriptions for the accompanying experimental videos and meeting recordings. The entire process often required multiple people to collaborate for several months, which was extremely inefficient and made it difficult to ensure data consistency. Summary of the Invention

[0003] The purpose of this invention is to at least partially solve one of the technical problems existing in the prior art.

[0004] To achieve the above objectives, this invention provides a method for intelligent annotation and knowledge graph construction of multimodal archives, comprising the following steps:

[0005] The multimodal archival data is digitized to obtain standard archival text, and the standard archival text is then segmented and labeled to obtain archival feature words.

[0006] Based on the archival feature words, entity recognition is performed on the standard archival text to obtain archival entities, and relation extraction is performed on the archival entities to obtain entity relations;

[0007] Based on the archive entities and the entity relationships, entity alignment is performed to obtain a unified entity identifier, and the unified entity identifier is supplemented with attributes to obtain entity attributes;

[0008] An archive knowledge graph is constructed based on the unified entity identifier and the entity attributes.

[0009] Furthermore, the multimodal archival data is digitized to obtain standard archival text, including:

[0010] The multimodal archive data is format-converted to obtain the original archive text, and the text content in the original archive text is extracted.

[0011] The text content is formatted and standardized to obtain standardized text, and the standardized text is then integrated to obtain standard archive text.

[0012] Furthermore, the standard archival text is segmented and annotated to obtain archival feature words, including:

[0013] The standard archive text is segmented according to sentence boundaries to obtain basic text segments, and word boundary recognition is performed on the basic text segments to obtain an initial word sequence;

[0014] The initial word sequence is matched with an archive dictionary to obtain a part-of-speech tagging table, and keywords are extracted based on the part-of-speech tagging table to obtain archive feature words.

[0015] Furthermore, based on the archival feature words, entity recognition is performed on the standard archival text to obtain archival entities, including:

[0016] The standard archive text is segmented using a context window to obtain local text fragments, and keyword matching is performed on the local text fragments based on the archive feature words to obtain candidate entity fragments;

[0017] Boundary correction is performed on the candidate entity fragments to obtain the start and end positions of the entities, and the entity type is determined based on the content in the start and end positions of the entities to obtain preliminary entity tags;

[0018] The initial entity labels are subjected to terminology consistency verification to obtain standardized entity names. Then, based on the standardized entity names and the archival feature words, a semantic association comparison is performed to obtain the archival entities.

[0019] Furthermore, relation extraction is performed on the aforementioned archive entities to obtain entity relations, including:

[0020] Co-occurrence analysis is performed on the archive entities to obtain a set of entity pairs, and syntactic structure analysis is performed on the set of entity pairs to obtain a syntactic dependency tree;

[0021] Based on the syntactic dependency tree, the entity pair set is subjected to relation type identification to obtain a relation type table, and the relation type table is subjected to temporal constraint analysis to obtain entity relations.

[0022] Furthermore, based on the archive entities and the entity relationships, entity alignment is performed to obtain a unified entity identifier, including:

[0023] The file entity is subjected to string normalization processing to obtain a standardized name string, and the first letter of the pinyin is extracted based on the standardized name string to obtain the pinyin abbreviation code;

[0024] Adjacent entity comparison is performed on the entity relationships corresponding to the archive entities to obtain the associated context set, and cross-validation is performed on the associated context set and the pinyin abbreviation code to obtain candidate alignment groups;

[0025] The candidate alignment groups are matched with historical archive records to obtain an archive referential evidence chain. Based on the archive referential evidence chain, the candidate alignment groups are merged and determined to obtain a unified entity identifier.

[0026] Furthermore, the unified entity identifier is supplemented with attributes to obtain entity attributes, including:

[0027] Based on the unified entity identifier, the original context fragment is located in the standard archive text to obtain the entity context window, and keyword extraction is performed on the entity context window to obtain attribute candidate words;

[0028] Semantic role labeling is performed on the candidate attribute words to obtain a role tag sequence, and the role tag sequence is matched with the category of the unified entity identifier to obtain the entity attribute.

[0029] Furthermore, an archival knowledge graph is constructed based on the unified entity identifier and the entity attributes, including:

[0030] The unified entity identifier is processed to generate a set of entity nodes, and the set of entity nodes is connected by edges based on the entity relationship to obtain an initial graph structure.

[0031] The entity attributes are processed into field structure to obtain attribute key-value pairs, and the attribute key-value pairs are bound to the corresponding entity nodes to obtain attribute graph nodes;

[0032] The attributed graph nodes are topologically integrated with the initial graph structure to obtain an archive knowledge graph.

[0033] Furthermore, the attribute key-value pairs are bound to the corresponding entity nodes to obtain attributed graph nodes, including:

[0034] Based on the unified entity identifier, the attribute key-value pairs are matched to obtain attribute-entity mapping pairs, and the field validity of the attribute-entity mapping pairs is checked to obtain compliant attribute records.

[0035] The compliance attribute records are injected into the attribute container of the corresponding entity node according to the preset attribute template to obtain a node with attribute graph.

[0036] This invention also provides a multimodal archive intelligent annotation and knowledge graph construction system, comprising:

[0037] The annotation module is used to digitize multimodal archival data to obtain standard archival text, and to perform word segmentation and annotation on the standard archival text to obtain archival feature words;

[0038] The identification module is used to perform entity recognition on the standard archive text based on the archive feature words to obtain archive entities, and to extract the relationships between the archive entities to obtain entity relationships;

[0039] The alignment module is used to perform entity alignment based on the archive entity and the entity relationship to obtain a unified entity identifier, and to supplement the unified entity identifier with attributes to obtain entity attributes;

[0040] A construction module is used to construct an archive knowledge graph based on the unified entity identifier and the entity attributes.

[0041] This invention provides a method for intelligent annotation and knowledge graph construction of multimodal archives, comprising the following steps: digitizing multimodal archive data to obtain standard archive text, and performing word segmentation and annotation on the standard archive text to obtain archive feature words; performing entity recognition on the standard archive text based on the archive feature words to obtain archive entities, and extracting relationships from the archive entities to obtain entity relationships; aligning entities based on the archive entities and entity relationships to obtain unified entity identifiers, and supplementing the unified entity identifiers with attributes to obtain entity attributes; and constructing an archive knowledge graph based on the unified entity identifiers and entity attributes. This method solves the technical problems of extremely low efficiency and difficulty in ensuring data consistency in traditional technologies when identifying archive types and extracting key information. It achieves the technical effect of providing standardized and comprehensive entity data for knowledge graph construction, reducing graph quality problems caused by inconsistent entities or missing attributes. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a schematic diagram illustrating the steps of a multimodal archive intelligent annotation and knowledge graph construction method in one embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram illustrating the steps of a multimodal archive intelligent annotation and knowledge graph construction system according to an embodiment of the present invention;

[0045] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0046] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0047] The following describes in detail, with reference to the accompanying drawings, a method for intelligent annotation and knowledge graph construction of multimodal archives according to an embodiment of the present invention. First, the method for intelligent annotation and knowledge graph construction of multimodal archives according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

[0048] Figure 1 This invention provides a method for intelligent annotation and knowledge graph construction of multimodal archives, comprising the following steps:

[0049] Step S1: Digitize the multimodal archive data to obtain standard archive text, and perform word segmentation and annotation on the standard archive text to obtain archive feature words.

[0050] Specifically, when digitizing multimodal archival data, adaptation operations must first be performed for different types of multimodal archival data: for example, when dealing with scanned copies of paper archives, OCR tools (such as Tesseract or Baidu OCR) are used to convert the text in the image into editable text; for audio archives, speech-to-text technology (such as the Whisper model) is used to extract the speech content and generate text; and for electronic documents such as PDF and Word, tools such as PyPDF2 and python-docx are used to parse the format, remove redundant format marks, and extract plain text. The text obtained after these processes will be uniformly standardized into standard archival text with a preset format, such as adjusting the encoding and line breaks of text from different sources to a consistent form. After digitization, the standard archival text is then segmented and annotated: Chinese text is segmented using Jieba or THULAC tools, while English text is segmented using NLTK tools. During the segmentation process, professional dictionaries in the archival field (such as dictionaries containing terms like "archives" and "case number") are used to ensure that professional terms are not split. Then, through TF-IDF or word vector calculation, words that appear frequently in the archival text and have strong semantic connections are selected as archival feature words. For example, in the standard text of scientific research project archives, words such as "project number," "research period," and "cooperating units" are determined as archival feature words after calculation.

[0051] Step S2: Based on the archival feature words, perform entity recognition on the standard archival text to obtain archival entities, and extract relationships from the archival entities to obtain entity relationships.

[0052] Specifically, when performing entity recognition on standard archival text based on archival feature words, the selected archival feature words are first used as key clues. These are then fine-tuned on archival domain data using a pre-trained language model (such as BERT or RoBERTa). The fine-tuned model is then used to traverse the standard archival text, locate and classify the entities within it. For example, in text containing "Zhang San of Peking University presided over a research project in 2023", the model will label archival entities in the categories of "institution", "personal name", and "time" based on archival feature words such as "Peking University", "Zhang San", and "2023". If there is a lack of labeled data, additional rules will be added to assist in recognition. For example, regular expressions will be used to match entities with fixed formats such as ID card numbers, or pseudo-labels will be obtained from knowledge bases such as wikidata through remote supervision to assist in the judgment. After entity recognition is completed and the archival entities are obtained, relation extraction is then performed. At this point, the identified archival entity pairs are combined with their context in the standard archival text as input and fed into a trained classification model (such as BERT+Softmax). The model will determine the type of association between entity pairs based on the semantic context. For example, for the entity pair "Zhang San" and "Peking University", combined with the context of "presiding over scientific research projects", the entity relationship "employed at" can be extracted. For semantically complex cases, PromptTuning is also introduced, using prompts such as "What is the relationship between [entity 1] and [entity 2] in the text?" to guide the model to accurately output the entity relationship.

[0053] Step S3: Based on the archive entity and the entity relationship, perform entity alignment to obtain a unified entity identifier, and supplement the unified entity identifier with attributes to obtain entity attributes.

[0054] Specifically, when performing entity alignment based on archival entities and entity relationships, the similarity of different archival entities will be calculated from three dimensions: name, attribute, and context. For example, when faced with two archival entities, "Peking University" and "Peking University", the name similarity will be calculated first through the edit distance, and then combined with the common attributes such as "Located in Beijing" and "Higher education institution", as well as the context associations such as "Peking University scientific research project" and "Scientific Research Department of Peking University" in the text, to determine that the two are the same entity. Subsequently, it will be mapped to an authoritative database such as the national archival institution directory, and a unified entity identifier such as "INST-001" will be assigned; if there are archival entities with the same name such as "Zhang San", the attribute differences between "Employed by Research Institute A" and "Born in 1985" and "Employed by University B" and "Born in 1990" will be combined, and the DBSCAN clustering algorithm will be used to distinguish them into different groups and assign unified entity identifiers respectively. After completing entity alignment and obtaining unified entity identifiers, attribute supplementation will be carried out. At this time, the detailed information in the standard archival text will be used. For example, for the unified entity identifier "INST-001" (i.e., Peking University), information such as "Establishment time: 1898" and "Supervisory department: Ministry of Education" will be extracted from the text as the entity attributes corresponding to this unified entity identifier.

[0055] Step S4, construct an archival knowledge graph based on the unified entity identifier and the entity attributes.

[0056] Specifically, first, the existing unified entity identifiers and entity attributes will be sorted out and converted into the basic constituent units of the knowledge graph. For example, the "unified entity identifier INST-001 (Peking University)" and the "entity attribute establishment time: 1898" will be corresponded, and then combined with the previously extracted entity relationships to construct triples such as "INST-001 - establishment time - 1898". Then, the schema of the archival knowledge graph will be designed to clarify the core concepts such as "institution" and "archive", define the relationship types such as "subordination" and "custody", and the attribute types such as "number" and "term", ensuring that the associations between different unified entity identifiers and entity attributes comply with the specifications. After that, the conflicts that may exist in multi-source data will be processed. For example, if the "supervisory department" attribute descriptions of the same unified entity identifier from two sources are different, the information from the authoritative archival source will be preferred. Finally, the sorted triples will be combined with the schema to build a complete archival knowledge graph. In the graph, "INST-001 (Peking University)" will be connected to the unified entity identifier of the corresponding scientific research archive through the "associated archive" relationship, along with their respective entity attributes.

[0057] In a specific embodiment, digital processing is performed on multi-modal archival data to obtain standard archival text, including:

[0058] Convert the multi-modal archive data into a format to obtain the original archive text, and extract the text content from the original archive text;

[0059] Perform format normalization processing on the text content to obtain a standardized text, and integrate the content of the standardized text to obtain a standard archive text.

[0060] Specifically, when digitizing multi-modal archive data, first, format conversion is performed on different types of multi-modal archive data. For example, a scanned paper archive is converted into an image format by a scanner, and then the text in the image is converted into an editable original archive text through an OCR tool. Audio archives generate the original archive text with the help of speech-to-text technology. Then, the pure text content is extracted from these original archive texts. For example, from a PDF conversion text containing format markers, non-core information such as headers, footers, and page numbers is stripped off, and only the body text is retained. Next, format normalization processing is performed on the extracted text content. For example, "May 2023" is uniformly adjusted to "2023-05", and "Beijing" is corrected to "Beijing Municipality" to form a standardized text. Then, multiple standardized texts under the same archive theme (such as the project establishment description and approval opinion of a certain project) are integrated in a logical order. For example, the text paragraphs are arranged in chronological order, and finally, a standard archive text with a unified structure and complete content is obtained.

[0061] In a specific embodiment, perform word segmentation and tagging processing on the standard archive text to obtain archive feature words, including:

[0062] Segment the standard archive text at the sentence boundary to obtain basic text segments, and identify word boundaries for the basic text segments to obtain an initial word sequence;

[0063] Match the initial word sequence with an archive dictionary to obtain a词性标注表 (词性标注表 is not clear in the original Chinese and may need further clarification. Here it is left as is for now), and extract keyword features based on the词性标注表 to obtain archive feature words.

[0064] Specifically, when performing word segmentation and annotation on standard archival text, the text is first segmented according to sentence boundaries based on punctuation marks (such as periods, commas, and semicolons) and semantic pauses to obtain basic text segments. For example, in a standard archival text about "archiving of university research projects in 2024," the sentence "In March 2024, Peking University submitted application materials for the research project 'Application of Artificial Intelligence in Archives Management'; in May of the same year, the project was approved by the Ministry of Education" would be segmented into two basic text segments: "In March 2024, Peking University submitted application materials for the research project 'Application of Artificial Intelligence in Archives Management'" and "In May of the same year, the project was approved by the Ministry of Education." During segmentation, special care is taken to avoid splitting phrases containing complete semantics. Project titles like "Application of Artificial Intelligence in Archives Management," even if they contain punctuation, will be retained as a whole in the same basic text segment.

[0065] After obtaining the basic text segments, word segmentation tools (such as Jieba for Chinese and NLTK for English) are used to identify word boundaries for each segment, determining the start and end positions of words to obtain an initial word sequence. Taking the basic text segment "In March 2024, Peking University submitted application materials for the research project 'Application of Artificial Intelligence in Archives Management'" as an example, after word boundary identification, an initial word sequence is generated such as "March 2024", ",", "Peking University", "submit", "research project", "《", "artificial intelligence", "in", "archives management", "in", "of", "application", "》", "application materials". During the identification process, special expressions in the field of archives are taken into account. For example, fixed collocations such as "research project" and "application materials" are split as a whole word rather than into separate words such as "research", "project", "application", and "materials".

[0066] Next, the initial word sequence will be matched with a pre-built archival dictionary. This dictionary contains common archival terms (such as "archives", "file number", "retention period"), institution names (such as "Ministry of Education" and "Peking University"), and project types (such as "research project" and "archived project"). During the matching process, each word in the initial word sequence will be compared one by one to determine whether it falls within the scope of the archival dictionary and its corresponding part of speech will be marked, ultimately forming a part-of-speech tagging table. For example, in the initial word sequence above, "March 2024" will be marked as a "time noun", "Peking University" as an "institution noun", "research project" as a "field noun", "submit" as a "verb", and "application materials" as a "field noun". Punctuation marks such as ",", "《", and "》" will be marked as "punctuation marks". For common words not in the archival dictionary (such as "in", "in", and "of"), they will be marked as "function words" or "general nouns".

[0067] Finally, keyword extraction is performed based on the generated part-of-speech tagging table. Priority is given to selecting words related to the core information of the archives, such as "domain terms," ​​"institutional terms," ​​and "time terms." Simultaneously, the TF-IDF algorithm is used to calculate the frequency and importance of each word in the standard archival text, eliminating "function words," "punctuation marks," and words with extremely low frequency and no key significance to the archival content, thus obtaining archival feature words. For example, in the aforementioned standard archival text of "2024 University Research Project Archiving," after keyword extraction, words such as "March 2024," "Peking University," "research project," "Application of Artificial Intelligence in Archival Management," "Ministry of Education," and "May 2024" are identified as archival feature words. These words accurately reflect the core content of the archive, providing clear clues for subsequent entity recognition.

[0068] In a specific embodiment, entity recognition is performed on the standard archival text based on the archival feature words to obtain archival entities, including:

[0069] The standard archive text is segmented using a context window to obtain local text fragments, and keyword matching is performed on the local text fragments based on the archive feature words to obtain candidate entity fragments;

[0070] Boundary correction is performed on the candidate entity fragments to obtain the start and end positions of the entities, and the entity type is determined based on the content in the start and end positions of the entities to obtain preliminary entity tags;

[0071] The initial entity labels are subjected to terminology consistency verification to obtain standardized entity names. Then, based on the standardized entity names and the archival feature words, a semantic association comparison is performed to obtain the archival entities.

[0072] Specifically, when performing entity recognition on standard archival text based on archival feature words, a fixed-length context window (e.g., 10 words per window) is first set according to the semantic relevance of the text. The standard archival text is then segmented using this context window to obtain local text fragments. For example, for the standard archival text "In March 2024, Peking University submitted application materials for the research project 'Application of Artificial Intelligence in Archival Management'", if the window length is set to 8 words, it will be segmented into local text fragments such as "In March 2024, Peking University submitted application materials for the research project 'Artificial Intelligence'" and "Application materials for the research project 'Application of Artificial Intelligence in Archival Management'". During the segmentation, complete combinations of feature words are avoided. Archival feature words such as "Peking University" and "research project" will not be split into different local text fragments. Then, based on the identified archival feature words (such as "March 2024", "Peking University", "scientific research project"), keyword matching is performed on each local text fragment. If the fragment contains content that is completely consistent with or semantically similar to the archival feature words, it is marked as a candidate entity fragment. For example, in the first local text fragment mentioned above, "March 2024", "Peking University", "scientific research project", and "artificial intelligence" will be matched as candidate entity fragments respectively.

[0073] After obtaining candidate entity fragments, boundary correction is performed. Since context window segmentation may cause incomplete boundaries of candidate entity fragments (such as "Application in Archives Management" in the second fragment above), the boundaries need to be adjusted in combination with the complete semantics and text grammatical structure of the archival feature words. For example, "Application in Archives Management" is supplemented to "Application of Artificial Intelligence in Archives Management". The start and end positions of the entity in "Application of Artificial Intelligence in Archives Management" are determined to be from "《" to "》", avoiding entity breakage caused by window segmentation. Subsequently, based on the content in the start and end positions of the entity, and combined with the entity type library in the field of archives (including types such as "time", "institution", "project name", and "person"), the standard archival text is identified as an entity type: for example, "March 2024" is identified as "time" because it contains specific year and month information, and is marked with the preliminary entity label "[March 2024, time]"; "Peking University" is marked as "[Peking University, institution]" because it belongs to an educational institution; and "Application of Artificial Intelligence in Archival Management" is marked as "[Application of Artificial Intelligence in Archival Management, project name]" because it is the name of a scientific research project.

[0074] Next, the initial entity tags undergo terminology consistency verification. The tags are checked against authoritative archival terminology databases (such as the "Archival Management Terminology Standard") to ensure the terminology is standardized. For example, if "PKU" appears in the initial entity tag, it will be verified as the authoritative term "Peking University," resulting in the standardized entity name "[Peking University, Institution]." If "2024.3" appears, it will be corrected to "March 2024," ensuring consistent terminology. Finally, a semantic association comparison is performed between the standardized entity name and archival feature words. If the semantic similarity between the standardized entity name and the archival feature words reaches a preset threshold (e.g., above 90%), it is determined to be an archival entity. For example, the semantic association between "Application of Artificial Intelligence in Archival Management" (standardized entity name) and "Research Project" (archival feature word) reaches 95%, and the semantic association between "Peking University" and "Institution" type feature words reaches 100%. Ultimately, "[March 2024, Time]," "[Peking University, Institution]," and "[Application of Artificial Intelligence in Archival Management, Project Name]" are determined to be archival entities.

[0075] In a specific embodiment, relation extraction is performed on the archive entities to obtain entity relations, including:

[0076] Co-occurrence analysis is performed on the archive entities to obtain a set of entity pairs, and syntactic structure analysis is performed on the set of entity pairs to obtain a syntactic dependency tree;

[0077] Based on the syntactic dependency tree, the entity pair set is subjected to relation type identification to obtain a relation type table, and the relation type table is subjected to temporal constraint analysis to obtain entity relations.

[0078] Specifically, when extracting relationships from archival entities, the first step is to count the occurrence positions of different archival entities in the standard archival text and perform co-occurrence analysis on the archival entities. If two or more archival entities appear in the same paragraph, the same sentence, or adjacent context windows, they are combined into entity pairs, forming an entity pair set. For example, in the text “In March 2024, Peking University submitted application materials for the research project ‘Application of Artificial Intelligence in Archival Management’”, “[March 2024, time]” and “[Peking University, institution]”, “[Peking University, institution]” and “[“Application of Artificial Intelligence in Archival Management”, project name]”, “[March 2024, time]” and “[“Application of Artificial Intelligence in Archival Management”, project name]” all co-occur in the same sentence and will be combined into three entity pairs: (March 2024, Peking University), (Peking University, “Application of Artificial Intelligence in Archival Management”), and (March 2024, “Application of Artificial Intelligence in Archival Management”), which are then included in the entity pair set. Then, for each entity pair in the sentence, a syntactic analysis tool (such as Stanford Parser) is used to perform syntactic structure analysis on the sentence, disassemble the grammatical components such as subject, verb, object, attributive, adverbial, and complement in the sentence, and construct a syntactic dependency tree. Taking "Peking University submitted a research project 'Application of Artificial Intelligence in Archives Management'" as an example, the syntactic dependency tree will mark the "subject-verb dependency" relationship between "Peking University" (subject) and "submit" (verb), the "verb-object dependency" relationship between "submit" and "research project" (object), and the "attributive-head dependency" relationship between "research project" and "'Application of Artificial Intelligence in Archives Management'" (attributive), clearly presenting the grammatical association between entity pairs and surrounding words.

[0079] Based on the constructed syntactic dependency tree, the system analyzes the grammatical position and related words of entity pairs in the tree, and identifies the relation type of the entity pair set. For example, for the entity pair (Peking University, "Application of Artificial Intelligence in Archives Management"), the syntactic dependency tree reveals that "Peking University" (subject) is associated with "Submit" (predicate) and "Application of Artificial Intelligence in Archives Management" (object). Combining this with the relation type library in the archives field (which includes types such as "Submitted Project", "Presided over Project", and "Approved Project"), the relationship between the two is determined to be "Submitted Project", and it is marked in the relation type table. For the entity pair (March 2024, "Application of Artificial Intelligence in Archives Management"), because "March 2024" (adverbial) modifies "Application of Artificial Intelligence in Archives Management" with "Submit", the relationship is determined to be "Submission Time", and it is also entered into the relation type table. Subsequently, a time-series constraint analysis was performed on the relation type table to verify the rationality of the relationship by comparing it with the time logic and domain rules in the archival text. For example, if the relation type table contains (May 2024, "Application of Artificial Intelligence in Archival Management", submission time), while the text clearly states "submitted in March 2024, approved in May 2024", the time-series constraint analysis will find a contradiction between "May" and "submission time", and correct it to "approval time". If it contains (Peking University, "Application of Artificial Intelligence in Archival Management", approval project), combined with the information of "approval by the Ministry of Education" in the text, it will be corrected to "submission project", and finally the entity relationship that conforms to the facts in the text and the domain logic is obtained, such as (Peking University, submission project, "Application of Artificial Intelligence in Archival Management") and (March 2024, submission time, "Application of Artificial Intelligence in Archival Management").

[0080] In a specific embodiment, entity alignment is performed based on the archive entities and the entity relationships to obtain a unified entity identifier, including:

[0081] The file entity is subjected to string normalization processing to obtain a standardized name string, and the first letter of the pinyin is extracted based on the standardized name string to obtain the pinyin abbreviation code;

[0082] Adjacent entity comparison is performed on the entity relationships corresponding to the archive entities to obtain the associated context set, and cross-validation is performed on the associated context set and the pinyin abbreviation code to obtain candidate alignment groups;

[0083] The candidate alignment groups are matched with historical archive records to obtain an archive referential evidence chain. Based on the archive referential evidence chain, the candidate alignment groups are merged and determined to obtain a unified entity identifier.

[0084] Specifically, when performing entity alignment based on archival entities and entity relationships, first, string normalization is carried out on the names of archival entities, removing special symbols, spaces, or redundant expressions in the names, and unifying the font format and case (if it is in English) to obtain a standardized name string. For example, for the archival entities "Peking University", "Peking University (main campus)", and "PekingUniv.", they will be processed as "Peking University", "Peking University", and "Peking University" respectively; "March 2024" and "March 2024" will be unified as "March 2024". Then, based on these standardized name strings, the initials of the pinyin are extracted to generate a pinyin abbreviation code - the initials of the pinyin of "Peking University" are "BJDXX", and for "March 2024", the initials of the pinyin corresponding to the key time characters are extracted as "NFSNYY", quickly establishing a preliminary association clue between entities through the abbreviation code.

[0085] Subsequently, for each archival entity, other entities with which it has an entity relationship (i.e., adjacent entities) are searched, and the adjacent entity comparison is carried out on the entity relationship corresponding to the archival entity to obtain an associated context set. Taking the archival entity "Peking University" as an example, its associated entity relationship is (Peking University, submitted project, "Application of Artificial Intelligence in Archival Management"), and the adjacent entity is "Application of Artificial Intelligence in Archival Management", and the associated context set then contains the relationship type "submitted project" and the adjacent entity name; if the associated entity relationship of another archival entity "Peking University" is (Peking University, cooperation unit, Tsinghua University), and the adjacent entity is "Tsinghua University", the associated context set will contain "cooperation unit" and "Tsinghua University". Then, cross-verification is carried out based on the associated context set and the pinyin abbreviation code: if the pinyin abbreviation codes of two archival entities are the same (such as "Peking University" and "Peking University" are both "BJDXX"), and there is a common domain in the associated context set (such as both are involved in university scientific research projects), then the two are grouped into a candidate alignment group. For example, "Peking University" and "Peking University" are listed as a group of candidate alignment groups because their abbreviation codes are the same and they belong to entities related to scientific research projects.

[0086] Next, historical archive records are matched for entities in the candidate alignment group. This involves retrieving previously archived texts, tables, or related records from the archives to collect evidence proving they are the same entity, forming an archival referential evidence chain. For example, in 2023 research archives, "PKU" was labeled as "abbreviated as Peking University, participating in the formulation of the 'Archived Digitalization Standard'." In 2024 project application records, the collaborating unit of "PKU" is completely identical to that of "PKU" in 2023. These records together constitute an archival referential evidence chain for "PKU" and "PKU". Finally, based on the archival referential evidence chain, the candidate alignment group is merged: if there are three or more clear referential associations in the evidence chain (such as abbreviation descriptions in historical records, consistent collaborative relationships, and identical institutional addresses), the entities in the candidate alignment group are determined to be the same entity and assigned a unified entity identifier. For example, "PKU" and "PKU" are merged into a unified entity identifier "INST-001" to ensure the uniqueness of entity identifiers in subsequent knowledge graph construction.

[0087] In a specific embodiment, the unified entity identifier is supplemented with attributes to obtain entity attributes, including:

[0088] Based on the unified entity identifier, the original context fragment is located in the standard archive text to obtain the entity context window, and keyword extraction is performed on the entity context window to obtain attribute candidate words;

[0089] Semantic role labeling is performed on the candidate attribute words to obtain a role tag sequence, and the role tag sequence is matched with the category of the unified entity identifier to obtain the entity attribute.

[0090] Specifically, when supplementing attributes for a unified entity identifier, the process first searches the standard archival text for sentences or paragraphs containing the standardized entity name corresponding to the unified entity identifier (e.g., "INST-001" corresponds to "Peking University"). This locates the original context fragment, and then determines an entity context window centered on this fragment, containing 5-8 words before and after it. For example, in the standard archival text "In March 2024, Peking University (located in Haidian District, Beijing) submitted a research project 'Application of Artificial Intelligence in Archival Management.' The university was founded in 1898, and its governing body is the Ministry of Education," for the unified entity identifier "INST-001 (Peking University)," the entire paragraph containing "Peking University" is used as the original context fragment. Then, the preceding and following words are expanded to form an entity context window, with the window content being "In March 2024, Peking University (located in Haidian District, Beijing) submitted a research project 'Application of Artificial Intelligence in Archival Management.' The university was founded in 1898, and its governing body is the Ministry of Education." Then, TF-IDF or keyword extraction tools are used to extract keywords from the entity context window, filtering out words that are related to "Peking University" and can reflect attribute characteristics, and obtaining attribute candidate words. Words such as "Haidian District of Beijing", "1898", "Ministry of Education" and "scientific research project" will be extracted as attribute candidate words, while words such as "March 2024" and "submit" that are not strongly related to the attributes of the entity itself will be removed.

[0091] Next, semantic role labeling is performed on the candidate words for attributes. Semantic analysis tools are used to identify the grammatical role and semantic function of each candidate word in the context window, generating a role label sequence. For example, "Beijing Haidian District" is a location complement in "Located in Beijing Haidian District", labeled as "[Beijing Haidian District, location attribute value]"; "1898" is a time complement in "Founded in 1898", labeled as "[1898, time attribute value]"; "Ministry of Education" is an object in "The competent authority is the Ministry of Education", labeled as "[Ministry of Education, institution attribute value]"; and "Research project" is an object in "Submitted research project", labeled as "[Research project, business attribute value]". These labeling results together constitute the role label sequence. Subsequently, the category of the unified entity identifier was clarified—"INST-001 (Peking University)" belongs to the "Institution" category. Corresponding attribute slots (such as "Institution Address", "Creation Time", "Supervisor", "Main Business") were predefined for this category. Then, the role tag sequence was matched with these attribute slots: "[Haidian District, Beijing, Location Attribute Value]" matched the "Institution Address" slot, "[1898, Time Attribute Value]" matched the "Creation Time" slot, "[Ministry of Education, Institution Attribute Value]" matched the "Supervisor" slot, and "[Research Project, Business Attribute Value]" matched the "Main Business" slot. Finally, the entity attributes "INST-001 (Peking University): Institution Address = Haidian District, Beijing, Creation Time = 1898, Supervisor = Ministry of Education, Main Business = Research Project" were obtained.

[0092] In a specific embodiment, constructing an archive knowledge graph based on the unified entity identifier and the entity attributes includes:

[0093] The unified entity identifier is processed to generate a set of entity nodes, and the set of entity nodes is connected by edges based on the entity relationship to obtain an initial graph structure.

[0094] The entity attributes are processed into field structure to obtain attribute key-value pairs, and the attribute key-value pairs are bound to the corresponding entity nodes to obtain attribute graph nodes;

[0095] The attributed graph nodes are topologically integrated with the initial graph structure to obtain an archive knowledge graph.

[0096] Specifically, when constructing an archival knowledge graph based on unified entity identifiers and entity attributes, the first step is to generate nodes for each unified entity identifier (such as "INST-001 (Peking University)", "PROJ-001 (Application of Artificial Intelligence in Archival Management)", and "TIME-001 (March 2024)") according to preset node generation rules (including basic information such as entity category, unified identifier, and core name). This process transforms each entity into an independent node in the graph, forming an entity node set. For example, "INST-001 (Peking University)" will generate a node labeled "Category: Institution, ID: INST-001, Name: Peking University", and "PROJ-001 (Application of Artificial Intelligence in Archival Management)" will generate a node labeled "Category: Project, ID: PROJ-001, Name: Application of Artificial Intelligence in Archival Management". These nodes together constitute the entity node set. Then, based on the previously extracted entity relationships (such as “INST-001 (Peking University) - Submitted Project - PROJ-001 (《Application of Artificial Intelligence in Archives Management》)” and “TIME-001 (March 2024) - Submission Time - PROJ-001 (《Application of Artificial Intelligence in Archives Management》)”), edge connection operations are performed on the entity node set: an edge with a relationship label is established between two entity nodes that have a relationship. For example, an edge labeled “Submitted Project” is connected between “INST-001” and “PROJ-001”, and an edge labeled “Submission Time” is connected between “TIME-001” and “PROJ-001”, thereby forming an initial graph structure containing nodes and relationship edges.

[0097] Next, the entity attributes corresponding to each unified entity identifier are processed through field structuring, transforming the correspondence between "attribute name and attribute value" into standardized attribute key-value pairs. For example, the entity attributes "Institutional Address = Haidian District, Beijing, Creation Time = 1898, Supervising Unit = Ministry of Education, Main Business = Scientific Research Projects" of "INST-001 (Peking University)" will be processed into the key-value pair "{Institutional Address: Haidian District, Beijing, Creation Time: 1898, Supervising Unit: Ministry of Education, Main Business: Scientific Research Projects}". If "PROJ-001 (《Application of Artificial Intelligence in Archives Management》)" has the attribute "Project Establishment Time = March 2024, Project Type = Scientific Research", it will be processed into the key-value pair "{Project Establishment Time: March 2024, Project Type: Scientific Research}". These attribute key-value pairs are then bound to the corresponding entity nodes: the key-value pair of "INST-001" is associated with the node labeled "INST-001 (Peking University)", and the key-value pair of "PROJ-001" is associated with the corresponding project node, so that each node not only contains basic information, but also comes with complete attribute data, resulting in attribute graph nodes. Finally, the attributed graph nodes are topologically integrated with the initial graph structure: the connection logic between nodes and edges is checked to ensure consistency (e.g., the "Project Initiation Time" attribute value "March 2024" of "PROJ-001" needs to match the "Submission Time" relationship logic of the "TIME-001" node), ensuring that the attribute information and relationship edges are not conflicting. At the same time, the node layout is adjusted according to the knowledge logic of the archival field (e.g., the association hierarchy of "Institution-Project-Time"), ultimately forming an archival knowledge graph with nodes having attributes, edges having relationship labels, and logical coherence. For example, the "INST-001 (Peking University)" node in the graph has complete institutional attributes and is connected to the "PROJ-001" node through the "Submission Project" edge. The "PROJ-001" node is connected to the "TIME-001" node through the "Submission Time" edge, and the attributes and relationships of each node corroborate each other.

[0098] In a specific embodiment, the attribute key-value pairs are bound to the corresponding entity nodes to obtain attributed graph nodes, including:

[0099] Based on the unified entity identifier, the attribute key-value pairs are matched to obtain attribute-entity mapping pairs, and the field validity of the attribute-entity mapping pairs is checked to obtain compliant attribute records.

[0100] The compliance attribute records are injected into the attribute container of the corresponding entity node according to the preset attribute template to obtain a node with attribute graph.

[0101] Specifically, when binding property key-value pairs to corresponding entity nodes, first, with the unified entity identifier as the core index, the attribution matching of property key-value pairs is performed based on the unified entity identifier - each property key-value pair will be accompanied by the entity identifier information associated at the time of generation. For example, "{Organization Address: Haidian District, Beijing, Creation Time: 1898}" is accompanied by the "INST-001" identifier, and "{Project Approval Time: March 2024, Project Type: Scientific Research}" is accompanied by the "PROJ-001" identifier. By comparing these identifiers with the unified entity identifiers in the entity node set (such as "INST-001 (Peking University)" and "PROJ-001 (Application of Artificial Intelligence in Archive Management)"), the property key-value pairs are associated with the corresponding entity nodes to obtain property-entity mapping pairs, such as "INST-001 (Peking University) → {Organization Address: Haidian District, Beijing, Creation Time: 1898, Competent Authority: Ministry of Education, Main Business: Scientific Research Projects}" and "PROJ-001 (Application of Artificial Intelligence in Archive Management) → {Project Approval Time: March 2024, Project Type: Scientific Research}" which are mapping pairs with clear corresponding relationships.

[0102] Subsequently, a field legality check is performed on the property-entity mapping pairs, and the rationality of the key-value pairs is verified according to the property rules of different entity categories: for example, "INST-001" belongs to the "organization" type of entity, and the preset legal property fields include "organization address", "creation time", "competent authority", etc. If a field such as "marital status" that has nothing to do with the organization attributes appears in the mapping pair, it will be determined as an illegal field and excluded; at the same time, it is checked whether the property value format complies with the specification. For example, the "creation time" field requires the format of "YYYY year MM month" or "YYYY-MM". If the property value is "1898 / 01", it will be determined as an illegal format and corrected to "January 1898". After such checks, the property content that complies with the category rules and format requirements is retained to obtain compliant property records. For example, the compliant property record of "INST-001 (Peking University)" is {Organization Address: Haidian District, Beijing, Creation Time: 1898, Competent Authority: Ministry of Education, Main Business: Scientific Research Projects}, and all illegal fields and content with format errors are excluded.

[0103] Next, the preset attribute templates will be invoked—different entity categories correspond to different templates. The template for "Organization" entities includes attribute container partitions such as "Basic Information," "Administrative Affiliation," and "Business Scope." The template for "Project" entities includes partitions such as "Project Basic Information" and "Time Node." The compliance attribute records will be injected into the attribute containers of the corresponding entity nodes according to the template partitions: "Organization Address" and "Creation Time" from the "INST-001" compliance record will be injected into the "Basic Information" container, "Supervisory Unit" into the "Administrative Affiliation" container, and "Main Business" into the "Business Scope" container; "Project Type" from the "PROJ-001" compliance record will be injected into the "Project Basic Information" container, and "Project Establishment Time" into the "Time Node" container. Each attribute field will accurately correspond to the preset position within the container, ensuring that the attribute data is stored in an orderly manner. After injection, the entity node contains both basic identification information and structured attribute data, resulting in an attributed graph node, such as "INST-001 (Peking University) Attributed Graph Node: Category-Institution, ID-INST-001, Name-Peking University; Attribute Container [Basic Information: Institution Address=Haidian District, Beijing, Creation Time=1898; Administrative Affiliation: Supervising Unit=Ministry of Education; Business Scope: Main Business=Scientific Research Projects]".

[0104] The above describes a method for intelligent annotation and knowledge graph construction of multimodal archives according to an embodiment of the present invention. The following describes a system for intelligent annotation and knowledge graph construction of multimodal archives according to an embodiment of the present invention. Please refer to [link / reference]. Figure 2 One embodiment of the multimodal archive intelligent annotation and knowledge graph construction system of the present invention includes:

[0105] The annotation module 21 is used to digitize multimodal archival data to obtain standard archival text, and to perform word segmentation and annotation on the standard archival text to obtain archival feature words;

[0106] The identification module 22 is used to perform entity recognition on the standard archive text based on the archive feature words to obtain archive entities, and to extract the relationships between the archive entities to obtain entity relationships;

[0107] Alignment module 23 is used to perform entity alignment based on the archive entity and the entity relationship to obtain a unified entity identifier, and to supplement the unified entity identifier with attributes to obtain entity attributes;

[0108] Module 24 is used to construct an archive knowledge graph based on the unified entity identifier and the entity attributes.

[0109] In this embodiment, the specific implementation of each module in the above system embodiment is described in the above method embodiment, and will not be repeated here.

Claims

1. A method for intelligent annotation and knowledge graph construction of multimodal archives, characterized in that, Includes the following steps: The multimodal archival data is digitized to obtain standard archival text, and the standard archival text is then segmented and labeled to obtain archival feature words. Based on the archival feature words, entity recognition is performed on the standard archival text to obtain archival entities, and relation extraction is performed on the archival entities to obtain entity relations; Based on the archive entities and the entity relationships, entity alignment is performed to obtain a unified entity identifier, and the unified entity identifier is supplemented with attributes to obtain entity attributes; Construct an archive knowledge graph based on the unified entity identifier and the entity attributes; Relationships are extracted from the aforementioned archive entities to obtain entity relationships, including: Co-occurrence analysis is performed on the archive entities to obtain a set of entity pairs, and syntactic structure analysis is performed on the set of entity pairs to obtain a syntactic dependency tree; Based on the syntactic dependency tree, the entity pair set is subjected to relation type identification to obtain a relation type table, and the relation type table is subjected to temporal constraint analysis to obtain entity relations; Constructing an archive knowledge graph based on the unified entity identifier and the entity attributes, including: The unified entity identifier is processed to generate a set of entity nodes, and the set of entity nodes is connected by edges based on the entity relationship to obtain an initial graph structure. The entity attributes are processed into field structure to obtain attribute key-value pairs, and the attribute key-value pairs are bound to the corresponding entity nodes to obtain attribute graph nodes; The attributed graph nodes are topologically integrated with the initial graph structure to obtain an archive knowledge graph. Binding the attribute key-value pairs to the corresponding entity nodes yields attributed graph nodes, including: Based on the unified entity identifier, the attribute key-value pairs are matched to obtain attribute-entity mapping pairs, and the field validity of the attribute-entity mapping pairs is checked to obtain compliant attribute records. The compliance attribute records are injected into the attribute container of the corresponding entity node according to the preset attribute template to obtain a node with attribute graph.

2. The multimodal archive intelligent annotation and knowledge graph construction method according to claim 1, characterized in that, Multimodal archival data is digitized to obtain standard archival text, including: The multimodal archive data is format-converted to obtain the original archive text, and the text content in the original archive text is extracted. The text content is formatted and standardized to obtain standardized text, and the standardized text is then integrated to obtain standard archive text.

3. The multimodal archive intelligent annotation and knowledge graph construction method according to claim 1, characterized in that, The standard archival text is segmented and annotated to obtain archival feature words, including: The standard archive text is segmented according to sentence boundaries to obtain basic text segments, and word boundary recognition is performed on the basic text segments to obtain an initial word sequence; The initial word sequence is matched with an archive dictionary to obtain a part-of-speech tagging table, and keywords are extracted based on the part-of-speech tagging table to obtain archive feature words.

4. The multimodal archive intelligent annotation and knowledge graph construction method according to claim 1, characterized in that, Based on the archival feature words, entity recognition is performed on the standard archival text to obtain archival entities, including: The standard archive text is segmented using a context window to obtain local text fragments, and keyword matching is performed on the local text fragments based on the archive feature words to obtain candidate entity fragments; Boundary correction is performed on the candidate entity fragments to obtain the start and end positions of the entities, and the entity type is determined based on the content in the start and end positions of the entities to obtain preliminary entity tags; The initial entity labels are subjected to terminology consistency verification to obtain standardized entity names. Then, based on the standardized entity names and the archival feature words, a semantic association comparison is performed to obtain the archival entities.

5. The multimodal archive intelligent annotation and knowledge graph construction method according to claim 1, characterized in that, Based on the archive entities and the entity relationships, entity alignment is performed to obtain a unified entity identifier, including: The file entity is subjected to string normalization processing to obtain a standardized name string, and the first letter of the pinyin is extracted based on the standardized name string to obtain the pinyin abbreviation code; Adjacent entity comparison is performed on the entity relationships corresponding to the archive entities to obtain the associated context set, and cross-validation is performed on the associated context set and the pinyin abbreviation code to obtain candidate alignment groups; The candidate alignment groups are matched with historical archive records to obtain an archive referential evidence chain. Based on the archive referential evidence chain, the candidate alignment groups are merged and determined to obtain a unified entity identifier.

6. The multimodal archive intelligent annotation and knowledge graph construction method according to claim 1, characterized in that, The unified entity identifier is supplemented with attributes to obtain entity attributes, including: Based on the unified entity identifier, the original context fragment is located in the standard archive text to obtain the entity context window, and keyword extraction is performed on the entity context window to obtain attribute candidate words; Semantic role labeling is performed on the candidate attribute words to obtain a role tag sequence, and the role tag sequence is matched with the category of the unified entity identifier to obtain entity attributes.

7. A multimodal archive intelligent annotation and knowledge graph construction system, characterized in that, The method for performing multimodal archive intelligent annotation and knowledge graph construction according to any one of claims 1 to 6 includes: The annotation module is used to digitize multimodal archival data to obtain standard archival text, and to perform word segmentation and annotation on the standard archival text to obtain archival feature words; The identification module is used to perform entity recognition on the standard archive text based on the archive feature words to obtain archive entities, and to extract the relationships between the archive entities to obtain entity relationships; The alignment module is used to perform entity alignment based on the archive entity and the entity relationship to obtain a unified entity identifier, and to supplement the unified entity identifier with attributes to obtain entity attributes; The construction module is used to construct an archive knowledge graph based on the unified entity identifier and the entity attributes; Relationships are extracted from the aforementioned archive entities to obtain entity relationships, including: Co-occurrence analysis is performed on the archive entities to obtain a set of entity pairs, and syntactic structure analysis is performed on the set of entity pairs to obtain a syntactic dependency tree; Based on the syntactic dependency tree, the entity pair set is subjected to relation type identification to obtain a relation type table, and the relation type table is subjected to temporal constraint analysis to obtain entity relations; Constructing an archive knowledge graph based on the unified entity identifier and the entity attributes, including: The unified entity identifier is processed to generate a set of entity nodes, and the set of entity nodes is connected by edges based on the entity relationship to obtain an initial graph structure. The entity attributes are processed into field structure to obtain attribute key-value pairs, and the attribute key-value pairs are bound to the corresponding entity nodes to obtain attribute graph nodes; The attributed graph nodes are topologically integrated with the initial graph structure to obtain an archive knowledge graph. Binding the attribute key-value pairs to the corresponding entity nodes yields attributed graph nodes, including: Based on the unified entity identifier, the attribute key-value pairs are matched to obtain attribute-entity mapping pairs, and the field validity of the attribute-entity mapping pairs is checked to obtain compliant attribute records. The compliance attribute records are injected into the attribute container of the corresponding entity node according to the preset attribute template to obtain a node with attribute graph.

Images