Archives intelligent classification and retrieval method and system based on multi-modal semantic understanding

Abstract

The application provides an archive intelligent classification and retrieval method and system based on multi-modal semantic understanding, and relates to the technical field of data processing. First, multi-modal data is acquired, and deep semantic understanding processing is performed to obtain a text semantic understanding result. Then, archive image content and handwriting style and seal marks in the content are analyzed and understood to obtain an image semantic understanding result. Then, carrier physical information is analyzed to obtain carrier era characteristics and preservation state characteristics. Subsequently, based on formation time and responsible person information in metadata, multi-modal semantic collaborative representation is generated. Finally, according to archive formation background and correlation represented by the multi-modal semantic collaborative representation, a semantic network between archives is constructed to realize archive classification and retrieval operation. The technical scheme provided by the application realizes intelligent classification and correlation retrieval based on archive authenticity and formation background through multi-modal data collaborative analysis and semantic network construction.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an intelligent classification and retrieval method and system for archives based on multimodal semantic understanding. Background Technology

[0002] As the digitization of archival management deepens, the need for efficient and accurate intelligent classification and semantic retrieval of massive, multi-source, and heterogeneous archives is becoming increasingly urgent. This requires technical solutions that can not only process text and image content, but also comprehensively evaluate the physical attributes and background of the archives themselves, in order to support in-depth content understanding and correlation discovery.

[0003] The current technical solution is an archival processing method based on multimodal feature fusion. This method extracts the semantic features of the digitized archival text and the visual features of the scanned images, and then performs feature splicing or early fusion on the two before inputting them into a classification or retrieval model to achieve archival organization and retrieval based on content similarity.

[0004] However, this method still suffers from superficial multimodal information association and cannot effectively assess the intrinsic authenticity and contextual relationships of archives. It only focuses on feature combinations at the content level and fails to utilize archival metadata to conduct in-depth cross-verification and collaborative analysis of information from different sources such as textual records, handwriting, seals, and carrier materials. As a result, the classification and retrieval results are difficult to reflect the deep semantic relationships between archives, such as the hierarchy of authenticity, the relationship of the recorder's position, and the historical context. Summary of the Invention

[0005] This application provides a method and system for intelligent classification and retrieval of archives based on multimodal semantic understanding, in order to solve the problems of insufficient depth of multimodal archive information utilization and inability to support intelligent classification and semantic retrieval based on the authenticity and background of archives in the existing technology.

[0006] Firstly, this application provides an intelligent classification and retrieval method for archives based on multimodal semantic understanding, including: Acquire multimodal data of the target archive, including the archive text content, archive image content, and physical information of the target archive's carrier; Deep semantic understanding processing is performed on the archival text content to obtain text semantic understanding results, which include the recorded events and the subjective bias of the narrator; Visual semantic understanding processing is performed on the content of archival images. The handwriting style and seal marks in the archival image content are analyzed to obtain the image semantic understanding result, which includes writer characteristics and authoritative source characteristics. Material and trace analysis is performed on the physical information of the carrier of the target archive to obtain the characteristics of the carrier's era and preservation status; Based on the formation time and responsible person information in the metadata of the target archive, the subjective inclination of the recorder, the characteristics of the writer, and the era characteristics of the carrier are cross-verified and semantically associated to generate a multimodal semantic collaborative representation of the target archive. The multimodal semantic collaborative representation is used to characterize the authenticity of the archive content, the formation background, and the related relationships. Based on the archival formation background and relationships represented by the multimodal semantic collaborative representation, a semantic network is constructed among the archives, and the classification and retrieval operations of the archives are realized based on the semantic network.

[0007] Optionally, deep semantic understanding processing is performed on the archival text content to obtain text semantic understanding results, which include the described events and the narrator's subjective bias, including: The archival text content is divided into sentences to obtain multiple text sentences, and the target objects and action descriptions appearing in the text sentences are identified. The target objects include names of people, places, and organizations, and the action descriptions include verb phrases. Based on the target object and action description identified in the text sentence, an event description framework with action as the core is constructed. The event description framework includes the action executor, the action object, and the time when the action occurs. Extract evaluative words and mood markers from text sentences, wherein the evaluative words include adjectives and adverbs, and the mood markers include interjections and interrogative words; The event description frames are associated with evaluative words and tone markers to determine the narrator's subjective inclination corresponding to each event description frame. The narrator's subjective inclination includes positive evaluation, negative evaluation, and neutral attitude. By integrating all event description frameworks and corresponding narrators' subjective biases, a textual semantic understanding result is formed.

[0008] Optionally, visual semantic understanding processing is performed on the content of the archival image, and the handwriting style and seal marks in the archival image content are analyzed to obtain the image semantic understanding result. The image semantic understanding result includes writer characteristics and authoritative source characteristics, including: Locate and separate the handwriting area image and the seal area image from the content of the archival image; The text in the handwriting region image is analyzed character by character to extract the stroke shape features of each character and the connection relationship features between strokes; Based on the stroke shape features and connection relationship features of characters, a set of handwriting style features representing individual writing habits is calculated; Contour and internal pattern analysis is performed on the image of the seal area to extract the outer contour shape features and graphic arrangement structure features of the seal; The outer contour shape features and graphic arrangement structure features of the seal are matched with the pre-stored authoritative seal template features. When the match is successful, it is determined to be the authoritative source feature corresponding to the seal imprint. The handwriting style feature set is used as the writer feature, and the identified authoritative source feature is used as the authoritative source feature, together forming the image semantic understanding result.

[0009] Optionally, material and trace analysis is performed on the physical information of the target archive's carrier to obtain the carrier's age characteristics and preservation status characteristics, including: Obtain carrier material samples and carrier surface images from the carrier physical information of the target file. The carrier material samples include carrier color characteristics, texture characteristics and fiber structure characteristics. The carrier material sample is analyzed. Based on the color change characteristics and fiber structure characteristics of the carrier material sample, and combined with the pre-stored historical material characteristic database, the production age range of the carrier material sample is inferred to obtain the first age inference result. Analyze the carrier surface image to identify creases and stains. Based on the direction and depth characteristics of the creases, analyze the folding history and stress conditions of the target file; Based on the color and shape diffusion characteristics of stains, infer the source of the stain's composition and the environment in which it formed; Based on the first-generation inference results, and combined with the stress analysis of crease marks and the inference of the formation environment of stain marks, the carrier age characteristics and preservation status characteristics of the target archives are determined.

[0010] Optionally, based on the formation time and responsible person information in the metadata of the target archive, the subjective inclination of the recorder, the characteristics of the writer, and the era characteristics of the carrier are cross-verified and semantically associated to generate a multimodal semantic collaborative representation of the target archive, including: Extract the creation time information and responsible party information from the metadata of the target file; The formation time information is compared with the action occurrence time contained in the event description framework in the text semantic understanding result. When the action occurrence time is within the time range identified by the formation time information, it is recorded as a time consistency verification result. The responsible person information is compared with the writer features and authoritative source features in the image semantic understanding results. When the responsible person information matches the writer identity identified by the writer features or the seal to which the seal belongs identified by the authoritative source features, it is recorded as the identity consistency verification result. The formation time information is compared with the carrier age characteristics to determine whether the material production age range identified by the carrier age characteristics includes the formation time information. If it includes the formation time information, it is recorded as the material age consistency verification result. Based on the results of time consistency verification, identity consistency verification, and material era consistency verification, we correlate the attitude tendencies reflected by the narrator's subjective inclination, the personal writing habits reflected by the writer's characteristics, and the historical background reflected by the era characteristics of the carrier to generate semantically related records. By integrating the semantic association records, a multimodal semantic collaborative representation of the target file is generated.

[0011] Optionally, based on the results of temporal consistency verification, identity consistency verification, and material era consistency verification, the semantic association records are generated by associating the attitudes reflected in the narrator's subjective inclinations, the personal writing habits reflected in the writer's characteristics, and the historical background reflected in the era characteristics of the medium. These associations include: Based on the time consistency verification results, the degree of consistency between the occurrence time of the events described in the text semantic understanding results and the formation time information in the archive metadata is judged, and the temporal relationship between the described events and the formation of the archive is inferred based on the degree of consistency, which serves as a characteristic of the recording timing. Based on the identity consistency verification results, the matching between the responsible person's information and the writer's characteristics or authoritative source characteristics is judged. Combined with the narrator's subjective tendency in the text semantic understanding results, the narrator's position in the event is inferred as the narration position feature. Based on the consistency verification results of the material era, the logical relationship between the production age of the physical material of the carrier and the time of the formation of the archive is determined. Combined with the historical period information reflected by the characteristics of the carrier era, the physical background constraint of the formation of the archive is constructed as a background authenticity constraint feature. By combining the characteristics of the timing of the record, the characteristics of the recorder's position, and the characteristics of the background authenticity constraint, a semantic association record is generated. The semantic association record includes the assessment conclusion of the authenticity of the archive content, the qualitative description of the recorder's position, and the contextual description of the background of the event.

[0012] Optionally, based on the archival formation background and relationships represented by the multimodal semantic collaborative representation, a semantic network is constructed among the archives, and the classification and retrieval operations of the archives are implemented based on the semantic network, including: Each target file is defined as a file node, and the multimodal semantic collaborative representation of the target file is used as the node attribute of the corresponding file node. The node attribute includes the degree of authenticity of the file content, the recorder's position, and the background context of its formation. Calculate the similarity between the node attributes of any two archive nodes among multiple archive nodes; For any two archive nodes, when the similarity meets the preset association conditions, a semantic association edge is established between the two archive nodes. The semantic association edge is used to characterize the semantic association between the two archives in terms of their formation background, the recorder's position, or the authenticity of the content. Combine all file nodes and all semantically related edges to construct a semantic network between target files; In the semantic network, archive nodes are classified according to their node attributes to form a semantic-based archive classification. When a retrieval request is received, the file node that matches the retrieval request is located in the semantic network, and other target file nodes that have semantic association with the target file node are found based on the semantic association edges and returned as retrieval results.

[0013] Secondly, this application provides an intelligent archival classification and retrieval system based on multimodal semantic understanding, comprising: The acquisition module is used to acquire multimodal data of the target archive, including the archive text content, archive image content, and physical information of the target archive's carrier. The first processing module is used to perform deep semantic understanding processing on the archival text content to obtain text semantic understanding results, which include the recorded events and the subjective tendencies of the recorder. The second processing module is used to perform visual semantic understanding processing on the content of the archival image, analyze the handwriting style and seal marks in the content of the archival image, and obtain the image semantic understanding result, which includes writer features and authoritative source features. The analysis module is used to perform material and trace analysis on the physical information of the carrier of the target archive to obtain the carrier's age characteristics and preservation status characteristics; The generation module is used to cross-verify and semantically associate the subjective inclination of the narrator, the characteristics of the writer, and the era characteristics of the carrier based on the formation time and responsible person information in the metadata of the target archive, and generate a multimodal semantic collaborative representation of the target archive. The multimodal semantic collaborative representation is used to characterize the authenticity of the archive content, the formation background, and the related relationships. The construction module is used to build a semantic network between archives based on the archive formation background and association relationships represented by the multimodal semantic collaborative representation, and to realize archive classification and retrieval operations based on the semantic network.

[0014] Thirdly, this application provides a computing device, including a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are invoked and executed by the processing component to implement the intelligent classification and retrieval method for archives based on multimodal semantic understanding as described in the first aspect above.

[0015] Fourthly, this application provides a computer storage medium storing a computer program, which, when executed by a computer, implements a file intelligent classification and retrieval method based on multimodal semantic understanding as described in the first aspect.

[0016] This application analyzes the recorded events and the subjective inclinations of the narrators from the archival texts, analyzes the handwriting and seals from the archival images to obtain the characteristics of the writer and the authoritative source, and combines the analysis of the physical information of the carrier to obtain the era characteristics of the carrier. Then, it uses the formation time and responsible person information in the archival metadata to cross-verify and deeply semantically associate the above-mentioned multi-source heterogeneous information, generating a multimodal semantic collaborative representation that can characterize the authenticity of the archival content, the formation background and the relationship between them. This process breaks through the limitations of the simple fusion of multimodal information in the existing technology, realizes the deep collaborative analysis of the inherent attributes and formation logic of the archives, and lays the core data foundation for subsequent semantic-based archival organization and discovery.

[0017] Furthermore, by constructing an archival semantic network using multimodal semantic collaborative representation as node attributes, archival classification can be based on the deep semantics contained in the node attributes, such as the degree of authenticity, the recorder's position, and the background context. Archival retrieval can then discover other archives related to the retrieval target in terms of their formation background, recorder's position, or authenticity based on the association edges in the semantic network. This effectively solves the problems of existing methods having a single classification retrieval dimension and failing to reflect the deep semantic relationships between archives. Thus, it achieves more accurate and comprehensive intelligent classification and semantic retrieval based on the authenticity and formation background of archives.

[0018] These or other aspects of this application will become more apparent in the following description of the embodiments. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart of an intelligent classification and retrieval method for archives based on multimodal semantic understanding, provided in this application, is shown. Figure 2 A schematic diagram of the structure of an intelligent classification and retrieval system for archives based on multimodal semantic understanding provided in this application is shown. Figure 3 A schematic diagram of the structure of a computing device provided in this application is shown. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present application, the technical solution of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0022] In some of the processes described in the specification, claims, and accompanying drawings of this application, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not themselves represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a chronological order, nor do they limit "first" and "second" to different types.

[0023] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] Figure 1 This application provides a flowchart of an intelligent classification and retrieval method for archives based on multimodal semantic understanding, as shown below. Figure 1 As shown, the method includes: Step 101: Obtain multimodal data of the target archive, including the archive text content, archive image content, and physical information of the target archive's carrier.

[0025] In this step, multimodal data refers to a collection of different types of data collected to comprehensively describe an archive. In this application, it specifically refers to three types of data: the text content of the archive, the image content of the archive, and the physical information of the carrier. These data are used to provide complete raw materials for subsequent deep semantic analysis. They are obtained by querying and extracting the corresponding digital text files, digital image files, and carrier physical attribute detection data files from the database or dedicated storage interface of the archive management system based on the unique identifier of the target archive.

[0026] The textual content of archives refers to the information recorded in written form in archives, which is used to understand the specific events and viewpoints described in the archives.

[0027] Archival image content refers to image files formed after archival entities have been digitized and scanned, used to analyze visual elements in the archives such as handwriting, printed fonts, and seal patterns.

[0028] Physical information of the carrier refers to the physical attributes and state data of the physical carrier of the archive, such as the material composition, color, texture, fiber structure of paper, as well as traces such as creases and stains on the surface, which are used to help determine the age, authenticity and preservation condition of the archive.

[0029] In this step, a data request is first sent to the document management system based on the unique identifier of the target document. Then, the system responds to the request and returns three core data files associated with the identifier: a digitized text file processed by OCR, a high-resolution digital image file, and a carrier physical attribute data file generated by a dedicated detection device.

[0030] Secondly, text files are read using file parsing technology to extract all character information and obtain structured archival text content. At the same time, digital image files are processed using image decoding technology and loaded into a pixel matrix in memory to obtain archival image content. Then, data parsing technology is used to extract quantitative or descriptive data such as material composition, color, texture, fiber structure, and surface traces from the carrier physical property data file according to a predetermined format to obtain carrier physical information.

[0031] Finally, data integration technology is used to encapsulate and associate the archival text content, archival image content, and carrier physical information according to a predefined structured format, constructing a complete and unified multimodal data object, which serves as the input basis for all subsequent in-depth analysis steps.

[0032] For example, in the historical archives of City A, staff need to process a handwritten document from 1955 concerning the approval for the construction of Factory B, numbered ARC-1955-001. First, they send a data request to the digital resource management system and respond to it by retrieving three core files associated with that number from the database: an OCR-converted text file, a high-resolution scanned image file, and a material data file generated by professional equipment. Next, they read the text file to obtain the document's text content; then, they load and decode the image file to obtain an image containing clear handwriting and seals; simultaneously, they analyze the material data file to extract data such as paper composition, fiber texture, and surface creases, forming the physical information of the carrier; finally, they package the text, image, and physical information data and associate them with the document number to form a complete multimodal data set, preparing for subsequent analysis.

[0033] Step 102: Perform deep semantic understanding processing on the archival text content to obtain text semantic understanding results, which include the recorded events and the narrator's subjective bias.

[0034] Optionally, step 102 may specifically include: Step 1021: Perform sentence-level segmentation on the archive text content to obtain multiple text sentences, and identify the target objects and action descriptions appearing in the text sentences. The target objects include names of people, places, and organizations, and the action descriptions include verb phrases.

[0035] Step 1022: Based on the target object and action description identified in the text sentence, construct an event description framework with action as the core. The event description framework includes the action executor, the action object, and the time when the action occurs.

[0036] Step 1023: Extract evaluative words and mood markers from the text sentence. The evaluative words include adjectives and adverbs, and the mood markers include interjections and interrogative words.

[0037] Step 1024: Associate the event description frames with evaluative words and tone markers to determine the narrator's subjective inclination corresponding to each event description frame. The narrator's subjective inclination includes positive evaluation, negative evaluation, and neutral attitude.

[0038] Step 1025: Integrate all event description frameworks and corresponding narrators' subjective biases to form the text semantic understanding result.

[0039] In this step, the text semantic understanding result refers to the formalized representation obtained after in-depth analysis of the archival text content, which is used to carry the core semantic information parsed from the text.

[0040] A descriptive event refers to a specific event or activity described in a text, including who did what and when, and is obtained by structuring the target objects and action descriptions identified in the text sentences.

[0041] The narrator's subjective inclination refers to the personal attitude or emotional tendency revealed by the author of the text through the choice of words and tone of voice when describing events, such as support, opposition, or neutrality.

[0042] Multiple text sentences refer to independent and complete units of expression obtained by dividing continuous text into sentence boundaries. These units are used for more granular semantic analysis and are obtained through sentence segmentation techniques in natural language processing.

[0043] The target object and action description are the key semantic components identified in the text sentence. The target object includes entities such as names of people, places, and organizations that refer to specific people or things. The action description is mainly verb phrases that express behavior or state, which are obtained through named entity recognition and syntactic analysis techniques.

[0044] Names of people, places, and organizations are specific types of the target object, referring to the names of people, places, and organizations, respectively.

[0045] A phrasal verb is the core of an action description, consisting of a main verb and its related modifiers or objects.

[0046] The event description framework is a model built on the target object and action description, used to structurally represent the semantics of an event. It includes the action executor (the subject that initiates the action), the action object (the object that receives the action), and the time when the action occurs, which is obtained through techniques such as semantic role labeling.

[0047] Evaluative words and mood markers are linguistic elements extracted from the text that reflect the author's attitude or emotional tone. Evaluative words mainly include adjectives and adverbs, while mood markers mainly include interjections and interrogative words that express tone. They are obtained through part-of-speech tagging and sentiment lexicon matching.

[0048] Adjectives and adverbs are common parts of speech for evaluative words. Interjections and interrogative words are common types of mood markers.

[0049] Positive evaluation, negative evaluation, and neutral attitude are three basic categories of the narrator's subjective inclination, which respectively indicate the author's positive, negative, or no obvious emotional inclination towards the event.

[0050] In this step, sentence segmentation technology is first applied to divide the continuous text into independent text sentences based on punctuation and grammatical rules. Then, named entity recognition and dependency parsing are performed in parallel on each sentence to extract target objects such as people's names, place names, and organization names, as well as action descriptions composed of core verbs and their dependent components.

[0051] Based on the identified target objects and action descriptions, semantic role labeling technology is further adopted to assign each semantic component in the sentence to the role it plays in an event framework, thereby constructing an event description framework with action as the core. This framework clearly identifies the action executor, the action object, and the time when the action occurs.

[0052] At the same time, part-of-speech tagging technology is used to identify adjectives and adverbs in sentences, and a pre-built sentiment dictionary is used to filter out words with sentimental tendencies as evaluative words. Through word category identification, interjections and interrogative words in sentences are located as tone markers.

[0053] Then, by analyzing the syntactic dependencies and modifiers between evaluative words, mood markers and event description frames, evaluative words and mood markers are associated with specific event frames. By combining the emotional polarity and mood type of the associated words, the subjective inclination of the narrator corresponding to each event description frame is determined through rules or classification models, and classified as positive, negative or neutral attitude.

[0054] Finally, by integrating all event description frameworks and their corresponding subjective biases, a complete text semantic understanding result containing structured event semantics and author attitude bias is generated.

[0055] For example, following the specific implementation of the previous step, the text content of the approval document regarding the construction of Factory B is first obtained, such as "Our bureau believes that the construction plan of Factory B is feasible and should be supported. Please have the relevant departments complete the approval as soon as possible before the end of 1955." Next, using sentence segmentation technology, this text is divided into two sentences. Then, in the first sentence, named entity recognition technology is used to identify the target objects: our bureau and the construction plan of Factory B. Syntactic analysis is used to identify the action descriptions: "considered feasible" and "should be supported." Finally, in the second sentence, the target objects: "relevant departments" and the action description: "complete the approval as soon as possible before the end of 1955" are identified. Subsequently, an event description framework was constructed based on this information: the first framework includes the executor (our bureau), the object of the action (the construction plan of Factory B), and the time of the action (implied to be the present); the second framework includes the relevant departments of the executor, the object of the action (approval), and the time of the action (before the end of 1955); at the same time, the evaluative words "feasible" and the tone marker "should" were extracted from the text; then, "feasible" and "should" were associated with the first event framework. Since "feasible" is a positive word and "should" indicates a suggestion, it was determined that the narrator's subjective tendency was positive evaluation; and since the second sentence has no obvious evaluative words, it was marked as a neutral attitude; finally, the two event frameworks and their corresponding subjective tendencies were integrated to form the textual semantic understanding result of the archive.

[0056] This step transforms the archival text into structured semantic information, extracting the core event elements and the narrator's subjective attitudes towards these events. This enables a deep understanding of the archival text and provides a crucial semantic foundation for subsequent collaborative analysis with other modal information, thereby comprehensively judging the authenticity and background relevance of the archives.

[0057] Step 103: Perform visual semantic understanding processing on the content of the archival image, analyze the handwriting style and seal marks in the content of the archival image, and obtain the image semantic understanding result, which includes writer features and authoritative source features.

[0058] Optionally, step 103 may specifically include: Step 1031: Locate and separate the handwriting area image and the seal area image from the content of the archive image.

[0059] Step 1032: Analyze the characters in the handwriting region image character by character, and extract the stroke shape features and connection relationship features between strokes for each character.

[0060] Step 1033: Based on the stroke shape features and connection relationship features of the characters, calculate the set of handwriting style features that represent personal writing habits.

[0061] Step 1034: Perform contour and internal pattern analysis on the seal area image to extract the outer contour shape features and graphic arrangement structure features of the seal.

[0062] Step 1035: Match the outer contour shape features and graphic arrangement structure features of the seal with the pre-stored authoritative seal template features. When the match is successful, it is determined to be the authoritative source feature corresponding to the seal imprint.

[0063] Step 1036: The handwriting style feature set is used as the writer feature, and the identified authoritative source feature is used as the authoritative source feature, together forming the image semantic understanding result.

[0064] In this step, the image semantic understanding result refers to the structured data obtained by analyzing the archival image, which contains the writer's identity characteristics and the authority information of the seal. It is used to visually represent the individual characteristics of the writer and the official source attributes of the archive.

[0065] Writer characteristics refer to the set of visual features that can characterize the personal habits of the writer of archival text. They are used to identify or distinguish different writers and are calculated by analyzing the stroke shape features and connection relationships between strokes in the handwriting area.

[0066] Authoritative provenance features refer to the characteristics corresponding to the seals and imprints that can prove that the archives came from an authoritative institution or individual, and are used to verify the official status and credibility of the archives' provenance.

[0067] Handwriting region images refer to sub-images that are segmented from the overall image of an archive and contain only handwritten or printed text. They are used specifically to analyze handwriting styles and are obtained by locating and segmenting dense text areas through image processing techniques.

[0068] A seal area image refers to a sub-image that is segmented from the overall image of the archive and contains only the seal pattern. It is used specifically to analyze the authenticity and origin of the seal. It is obtained by locating and segmenting the red or specific shaped seal area through image processing technology.

[0069] Stroke shape features refer to the features that describe the geometric shape of a single character stroke, such as the straightness, thickness, and beginning and ending shapes of the strokes. They are used to quantify writing details and are obtained by extracting the outline and analyzing the shape of the characters in the handwriting area image.

[0070] Connectivity features refer to the characteristics that describe the relative positions and connection methods between adjacent strokes in a character, such as the intersection angle of strokes and the shape of connection points. They are used to capture the continuity of writing and are obtained by analyzing the spatial topological relationships between strokes.

[0071] The handwriting style feature set is a set of high-level features obtained by comprehensively calculating the stroke shape features and connection relationship features of multiple characters, and is used to represent an individual's unique writing habits as a whole.

[0072] The outer contour shape feature refers to the features that describe the geometric shape of the overall outer boundary of the seal, such as a circle, square or special shape. It is used to initially screen the seal type and is obtained by edge detection and contour extraction on the seal area image.

[0073] Graphic arrangement structure features refer to the characteristics that describe the layout, relative position, and combination of elements such as text, patterns, and lines inside a seal. They are used to precisely match the content of the seal and are obtained by performing internal region segmentation and structural analysis on the seal area image.

[0074] Pre-stored authoritative seal template features refer to feature data extracted and stored in advance from real, standard images of authoritative seals, which are used as a benchmark for comparison and recognition.

[0075] In this step, target detection and image segmentation techniques are first applied to locate and separate the continuous text region and the stamp region of specific color and shape in the image, so as to obtain the handwriting region image and the stamp region image respectively.

[0076] For handwriting region images, optical character recognition technology is used to obtain the text sequence and the precise position of each character. Then, contour analysis technology is used to extract the stroke skeleton for each character image block, and the direction, curvature, and thickness variation of the strokes are quantified to obtain the stroke shape features. At the same time, the connection points, intersection angles and relative spatial relationships between strokes are analyzed to obtain connection relationship features. Subsequently, the above-mentioned low-level features of the entire handwriting are aggregated and summarized through feature statistics algorithms to generate a set of handwriting style features that represent individual writing habits.

[0077] For the image of the seal area, the geometric parameters of its outer boundary are first extracted as the outer contour shape features by the edge detection algorithm. Then, the image segmentation technology is used to distinguish the internal text, patterns and lines, analyze its spatial layout and arrangement rules, extract the graphic arrangement structure features, and perform similarity matching between these two types of features and the pre-stored authoritative seal template feature library. When the matching degree exceeds the preset threshold, the institution or identity corresponding to the seal is determined, that is, the authoritative source feature.

[0078] Finally, the handwriting style feature set is integrated as the writer feature, and together with the successfully matched authoritative source feature, they constitute the image semantic understanding result.

[0079] For example, following the specific implementation of the previous step, firstly, a high-resolution image of the approval document regarding the construction of Factory B is loaded; secondly, densely written areas are detected and segmented from the high-resolution image as handwriting region images, and a red circular area at the signature is detected as a seal region image; then, character recognition is performed on the handwriting region images, and the stroke direction, curvature, and other details of each character are analyzed to extract stroke shape features; simultaneously, the connection points and intersection angles between strokes are analyzed to extract connection relationship features; then, these features of all characters are summarized, and a stable [characteristic] is derived through statistical algorithms. A set of handwriting style features was formed by analyzing a fixed writing pattern. At the same time, the image of the seal area was analyzed, and its circular outer contour was extracted as the outer contour shape feature. The layout of the internal circular text and pentagram pattern was analyzed to obtain the graphic arrangement structure feature. Then, these seal features were matched with a pre-stored authoritative seal template library. It was found that they highly matched the official template features of the Industry Bureau of City A, thus determining that the authoritative source feature of the seal was the Industry Bureau of City A. Finally, the set of handwriting style features was used as the writer feature and combined with the matched authoritative source feature to output the image semantic understanding result of the document.

[0080] This step transforms visual image content into structured, comparable key semantic information, providing an indispensable dimension of visual evidence for subsequent cross-verification with textual content and carrier information to verify the recorder's identity, confirm the responsible agency, and comprehensively assess the authenticity of the archives.

[0081] Step 104: Perform material and trace analysis on the physical information of the carrier of the target archive to obtain the carrier's era characteristics and preservation status characteristics.

[0082] Optionally, step 104 may specifically include: Step 1041: Obtain a carrier material sample and a carrier surface image from the carrier physical information of the target file. The carrier material sample includes carrier color features, texture features and fiber structure features.

[0083] Step 1042: Analyze the carrier material sample. Based on the color change characteristics and fiber structure characteristics of the carrier material sample, and in conjunction with the pre-stored historical period material characteristic database, infer the production age range of the carrier material sample to obtain the first age inference result.

[0084] Step 1043: Analyze the carrier surface image and identify creases and stains from the carrier surface image.

[0085] Step 1044: Analyze the folding history and stress conditions of the target file based on the direction and depth characteristics of the fold marks.

[0086] Step 1045: Based on the color and shape diffusion characteristics of the stain, infer the source of the stain's composition and the environment in which it formed.

[0087] Step 1046: Based on the first age inference results, and combined with the stress analysis of crease marks and the inference of the formation environment of stain marks, determine the carrier age characteristics and preservation status characteristics of the target archive.

[0088] In this step, the carrier era characteristics refer to the characteristics of the production era or period range to which the inferred archival carrier materials belong, which are used to help determine the background of the formation time of the archives.

[0089] Preservation status characteristics refer to the features that describe the current physical condition and damage of the archival medium, and are used to assess the preservation integrity and historical traces of the archives.

[0090] A carrier material sample refers to a data sample about the carrier material itself extracted from the carrier's physical information. It includes specific attributes such as carrier color characteristics, texture characteristics, and fiber structure characteristics. It is used for material analysis and is obtained through detection techniques such as spectroscopy and microscopic imaging.

[0091] A carrier surface image refers to a high-resolution digital image obtained from the physical information of the carrier that records the macroscopic morphology of the carrier surface. It is used to observe surface traces and is obtained through high-resolution scanning or photography.

[0092] Carrier color characteristics refer to data describing the color attributes of carrier materials, such as hue, brightness, and saturation.

[0093] Texture characteristics refer to data describing the tactile feel and texture of a carrier material's surface, such as roughness or smoothness.

[0094] Fiber structure characteristics refer to data describing the distribution, morphology, and arrangement of fibers within a carrier material.

[0095] Color change characteristics specifically refer to the color changes of the carrier material due to aging over time.

[0096] The pre-stored historical period material feature database refers to a dataset that is established in advance and contains standard features of common carrier materials from different historical periods, which is used as a reference for age comparison.

[0097] The first chronology inference refers to the preliminary conclusion about the production period of the material obtained by comparing the characteristics of the carrier material sample with the material characteristic database of historical periods.

[0098] Crease marks and stain marks are two types of physical marks identified from images of the surface of a carrier. Crease marks are linear marks caused by folding of paper or other carriers, while stain marks are non-native material marks that adhere to the surface of the carrier.

[0099] The direction feature describes the direction in which the crease extends, while the depth feature describes the degree of indentation of the crease.

[0100] The folding history and stress conditions are inferred from the analysis of fold marks, indicating the number of times the archive was folded, the manner in which it was folded, and the external forces it was subjected to.

[0101] Color characteristics and shape diffusion characteristics describe the visual attributes of stains and marks. Color characteristics refer to the color that appears, while shape diffusion characteristics refer to the shape of the stains and marks as they spread and penetrate the substrate.

[0102] The source of the ingredients and the formation environment are inferred from the characteristics of the stain traces and the external conditions under which they were formed, such as water stains, oil stains, or formation in a humid environment.

[0103] In this step, the carrier material sample data and carrier surface image are first loaded and parsed from the carrier physical information. Then, the carrier material sample is quantified by spectral analysis to determine the chromaticity and lightness parameters of the carrier. The color difference value with the standard white is calculated to assess the degree of aging and obtain the color change characteristics. By analyzing the microstructure of fibers, parameters such as fiber morphology, length, and arrangement density are extracted to obtain fiber structural characteristics. These characteristics are then matched with a pre-stored database of material characteristics from historical periods. Using a feature similarity measurement algorithm, the most likely production period range of the material is inferred, generating the first period inference result. For the carrier surface image, image segmentation and abnormal region detection technology are used to identify linear creases and blocky stains. For creases, orientation features are extracted through orientation field analysis and depth features are estimated through image grayscale gradient analysis. The two are combined to infer the folding history and stress distribution of the archive. For stains, color space analysis is used to extract the main hue and chromaticity distribution as color features, and morphological analysis is used to quantify edge ambiguity and penetration morphology as shape diffusion features. Then, the source of its composition and formation environment are inferred by comparing with the stain knowledge base.

[0104] Finally, by combining the results of the first dating inference, the physical history revealed by the crease analysis, and the preservation conditions reflected by the stain analysis, a joint assessment was conducted to determine the production era and physical preservation status of the carrier material, namely, the carrier's era characteristics and preservation status characteristics.

[0105] For example, following the specific implementation of the previous step, the physical information of the carrier regarding the approval of the construction of Factory B was first read; then, the carrier material sample data recording the material properties and high-definition carrier surface images were loaded. By analyzing the material samples, the pale yellow color change characteristics of the paper were quantified, and its fiber structure characteristics of bamboo pulp and wood pulp were observed; then, these characteristics were compared with a pre-stored historical period material characteristic database, and it was found that they highly matched the characteristics of commonly used office paper in the 1950s, thus inferring that the first period inference result was that the material was produced in the 1950s; Simultaneously, the surface image of the carrier was analyzed, automatically identifying two parallel longitudinal linear creases and an irregular light yellow stain. Analyzing the direction and depth of the creases, it was inferred that the document had been folded longitudinally for preservation, with uniform force applied during the folding. Further analysis of the color characteristics and the diffused, hazy shape of the stain suggested that its composition might be related to water, and its formation environment might have been damp. Finally, combining the results of the first dating inference, the analysis of the creases, and the analysis of the stain, it was determined that the carrier of the document was paper from the 1950s, and its preservation condition was characterized by being folded for preservation and accompanied by dampness marks resembling water stains.

[0106] This step enables us to make a comprehensive judgment about the era and preservation condition of the archival medium, providing key evidence from the physical dimension for subsequent steps to assess the overall authenticity of the archives, verify the timeline of their formation, and understand their preservation context.

[0107] Step 105: Based on the formation time and responsible person information in the metadata of the target archive, cross-verify and semantically associate the subjective inclination of the recorder, the characteristics of the writer, and the era characteristics of the carrier to generate a multimodal semantic collaborative representation of the target archive. The multimodal semantic collaborative representation is used to characterize the authenticity of the archive content, the formation background, and the relationship between them.

[0108] Optionally, step 105 may specifically include: Step 1051: Extract the formation time information and responsible person information from the metadata of the target file.

[0109] Step 1052: Compare the formation time information with the action occurrence time contained in the event description framework in the text semantic understanding result. When the action occurrence time is within the time range identified by the formation time information, it is recorded as a time consistency verification result.

[0110] Step 1053: Verify the identity of the person in charge by comparing the information with the writer features and authoritative source features in the image semantic understanding results. When the information with the person in charge matches the identity of the writer identified by the writer features or the identity of the seal to which it belongs, it is recorded as the identity consistency verification result.

[0111] Step 1054: Compare the formation time information with the carrier age characteristics to determine whether the material production age range identified by the carrier age characteristics includes the formation time information. If it includes the formation time information, record it as a material age consistency verification result.

[0112] Step 1055: Based on the time consistency verification results, identity consistency verification results, and material era consistency verification results, the semantic association record is generated by associating the attitude inclination reflected by the narrator's subjective inclination, the personal writing habits reflected by the writer's characteristics, and the historical background reflected by the era characteristics of the carrier.

[0113] Optionally, step 1055 may specifically include the following steps: based on the time consistency verification results, determine the degree of consistency between the occurrence time of the event described in the text semantic understanding results and the formation time information in the archival metadata, and infer the temporal relationship between the described event and the formation of the archive based on the degree of consistency, as a recording timing feature; based on the identity consistency verification results, determine the matching situation between the responsible person information and the writer characteristics or authoritative source characteristics, and infer the recorder's position in the event by combining the recorder's subjective tendency in the text semantic understanding results, as a recording position feature; based on the material era consistency verification results, determine the logical relationship between the production era of the physical material of the carrier and the formation time of the archive, and construct the physical background constraint of the formation of the archive by combining the historical period information reflected by the carrier era characteristics, as a background authenticity constraint feature; combine the recording timing feature, the recording position feature and the background authenticity constraint feature to generate a semantic association record, which includes an assessment conclusion on the authenticity of the archive content, a qualitative description of the recorder's position, and a contextual description of the background of the event.

[0114] Step 1056: Integrate the semantic association records to generate a multimodal semantic collaborative representation of the target file.

[0115] In this step, metadata refers to information recorded and stored during the record management process that describes the basic attributes of the records and is used to identify the administrative and background information of the records.

[0116] The creation time refers to the date or time period during which the document was created or officially created, as recorded in the document management system. It is used to provide the official document creation time point and is obtained by parsing the date field in the metadata.

[0117] The responsible party information refers to the name of the organization or individual responsible for the content of the document, as recorded in the document management system. It is used to identify the legal source of the document and is obtained by parsing the responsible party field in the metadata.

[0118] Multimodal semantic collaborative representation refers to the unified and structured semantic expression generated by deeply cross-verifying and semantically associating heterogeneous information from multiple sources such as text, images, and carriers. It is used to comprehensively and collaboratively characterize the authenticity, background, and relationships of archival content.

[0119] Content authenticity refers to the degree to which the events recorded in the archives conform to objective facts, as well as whether the archives themselves are genuine.

[0120] The context of an archive's formation refers to the historical period, social environment, and specific circumstances in which it was created.

[0121] The connection refers to the inherent relationship between this file and other files in terms of events, people, time, or background.

[0122] The time of action occurrence is the point in time when the event occurred, inferred from the text, as contained in the event description framework in the text semantic understanding result.

[0123] The result of time consistency verification is a conclusion drawn by comparing the time information formed with the time when the action occurred, regarding whether the two are logically consistent.

[0124] Identity consistency verification results are conclusions about whether the identities match by verifying the information of the person in charge with the writer features or authoritative source features in the image semantic understanding results.

[0125] The consistency of material era verification results are derived by comparing the formation time information with the era characteristics of the carrier material to determine whether the era of the archival carrier material and the formation time of the archive are logically reasonable.

[0126] Semantic association records are descriptive records generated based on the above verification results, after associating the deep semantics of the narrator's subjective inclination, the writer's characteristics, and the era characteristics of the medium, and are used to carry the results of association analysis.

[0127] The degree of consistency of time information is a quantitative or qualitative measure that describes the closeness or overlap between the time of an action and the time of its formation.

[0128] The timing characteristic of the record is a feature inferred from the analysis of temporal consistency regarding the chronological or synchronous relationship between the content of the record and the record itself.

[0129] The characteristics of a narrative stance are inferred from identity consistency analysis and the subjective bias of the text, which are characteristics of the role or stance that the narrator may play in the events being narrated.

[0130] Background authenticity constraints are physical constraints on the historical context of the archives, derived from the analysis of material age consistency and the historical background of the carrier.

[0131] The assessment conclusion is a judgmental description of the degree of authenticity of the archival content in the semantically related records.

[0132] Qualitative description is a non-quantitative explanation of the narrator's position attributes in semantically related records.

[0133] Contextual descriptions are narrative descriptions of the background environment in which an event occurs within semantically associated records.

[0134] In this step, the formation time and responsible party information are first extracted from the archive metadata. Then, a time matching algorithm is used to compare the formation time with the action times of each event in the text semantic understanding results to verify whether they are within the formation time interval, thereby generating a time consistency verification result. Simultaneously, entity matching technology is used to compare the information of the person responsible with the semantic understanding results of the image: on the one hand, the writer's features are matched with a pre-stored handwriting identity database to identify the writer; on the other hand, the information of the person responsible is directly compared with the authoritative source features in the image results. If either match is successful, an identity consistency verification result is generated. In addition, through time inclusion judgment, it is verified whether the formation time is within the age range identified by the carrier's era characteristics to generate a material era consistency verification result.

[0135] Based on the above three types of verification results, a rule-based semantic association reasoning engine is activated. The engine first infers the temporal relationship between the recorded events and the formation of the archives based on the degree of consistency of time information in the temporal consistency verification results, and generates the recording timing features; Secondly, by combining the identity consistency verification results and the narrator's subjective inclination in the text semantic understanding results, the narrator's role and position in the event are inferred through the preset identity-position mapping rules, and the narration position characteristics are generated. At the same time, based on the material era consistency verification results and the historical background information contained in the carrier era characteristics, the background constraint rules are applied to construct the physical background constraint of the archive formation and generate the background authenticity constraint characteristics. Finally, the generated time-of-description features, position-of-description features, and background authenticity constraint features are integrated and filled into a structured semantic template to form a semantically related record containing authenticity assessment conclusions, position qualitative descriptions, and background context descriptions. This record is then transformed into a multimodal semantic collaborative representation of the target file through data encapsulation technology.

[0136] For example, following the specific implementation of the previous step, the process begins with processing the approval document regarding the construction of Factory B. First, the formation time information 1955-03-15 and the responsible party information, City A Industrial Bureau, are extracted from the metadata of the document. Second, the formation time 1955-03-15 is compared with the action occurrence time before the end of 1955 in the text semantic understanding results. Since the former falls within the latter's time range, the record time consistency verification result is consistent. At the same time, the responsible party information City A Industrial Bureau is checked against the authoritative source feature City A Industrial Bureau in the image semantic understanding results. The two match perfectly, and the record identity consistency verification result is a seal institution match. Next, the formation time 1955 is compared with the carrier era feature of the 1950s, confirming that the former is contained in the latter, and the record material era consistency verification result is reasonable. Then, based on these three corroborating results, semantic association reasoning was performed. Due to the high degree of temporal consistency, it was inferred that the timing of the record was almost synchronous with the formation of the archive. Because the responsible party and the seal-issuing institution matched and the subjective tone of the text was positive, it was inferred that the record's stance was that the unit that formed the archive, as the competent authority, held a supportive position. Because the material's era was reasonable, it was inferred that the background authenticity constraint was that the archive was formed in the 1950s and the carrier conformed to the characteristics of the era. Subsequently, these three features were combined to generate a semantic association record, which concluded that the content of the archive was highly authentic, the recorder, as the competent authority, supported the event, and the background was the approval process during the industrialization construction period of the 1950s. Finally, this semantic association record was integrated to generate the final multimodal semantic collaborative representation of the archive, providing core comprehensive semantic basis for its subsequent intelligent classification and association retrieval.

[0137] This step not only verifies the consistency among multimodal information, but more importantly, it generates deep semantics about the timing of the narration, the narrator's position, and the constraints of the background of its formation through reasoning. This enables a comprehensive assessment of the authenticity of the archival content and a clear depiction of its background and connections.

[0138] Step 106: Based on the archive formation background and association relationships represented by the multimodal semantic collaborative representation, construct a semantic network between archives, and realize archive classification and retrieval operations based on the semantic network.

[0139] Optionally, step 106 may specifically include: Step 1061: Define each target file as a file node, and use the multimodal semantic collaborative representation of the target file as the node attribute of the corresponding file node. The node attribute includes the degree of authenticity of the file content, the recorder's position, and the background context of its formation.

[0140] Step 1062: Calculate the similarity between the node attributes of any two file nodes among multiple file nodes.

[0141] Step 1063: For any two archive nodes, when the similarity meets the preset association conditions, establish a semantic association edge between the two archive nodes. The semantic association edge is used to characterize the semantic association between the two archives in terms of their formation background, the recorder's position, or the authenticity of their content.

[0142] Step 1064: Combine all file nodes and all semantically related edges to construct a semantic network between target files.

[0143] Step 1065: In the semantic network, the archive nodes are classified according to their node attributes to form a semantic-based archive classification.

[0144] Step 1066: When a retrieval request is received, locate the file node that matches the retrieval request in the semantic network, and find other target file nodes that have semantic association with the target file node based on the semantic association edges, and return them as retrieval results.

[0145] In this step, an archive node refers to an abstract entity in the constructed network structure that represents a specific archive and is used to uniquely identify and carry relationships within the semantic network.

[0146] Node attributes refer to the set of deep semantic features attached to a file node, used to describe the file represented by that node, and are used to quantitatively or qualitatively describe various attributes of the file.

[0147] The degree of authenticity of archival content is a component of node attributes, referring to the quantitative or hierarchical assessment of the authenticity and credibility of archival content.

[0148] The narrator's stance is a component of the node attributes, referring to a qualitative description of the attitude or role held by the narrator of the archival text.

[0149] The background context is a component of node attributes, referring to descriptive text that describes the historical period and specific environment in which the archive was created.

[0150] Similarity refers to the degree of semantic closeness or relevance of the node attributes of any two archive nodes, calculated by a specific algorithm. It is used to measure the strength of potential associations between archives and is calculated by comparing the attribute features of the two nodes.

[0151] Preset association conditions refer to a pre-defined numerical threshold or rule used to determine whether the similarity between two file nodes is large enough to consider that they have a semantic relationship. It is the standard for deciding whether to establish a connection edge.

[0152] The semantic network between target files refers to a network graph structure composed of all target file nodes and the semantic association edges connecting these nodes. It is used to visualize and store the complex semantic relationships between files and is constructed by combining all nodes and edges.

[0153] Semantic association is an abstract relationship represented by the edge connecting two archive nodes in the semantic network between target archives. It means that the two archives have an inherent and comprehensible connection in one or more aspects such as the background of their formation, the perspective of the recorder, or the authenticity of the content.

[0154] Semantic-based archival classification refers to the automated clustering or grouping of archival nodes based on the semantic similarity of their node attributes, rather than relying on traditional keywords or fixed themes. This is achieved by analyzing the semantic network structure or the similarity of node attributes.

[0155] In this step, firstly, graph database technology is used to create nodes for each document, and its multimodal semantic collaborative representation is used as the attribute vector of the node, including dimensions such as the degree of authenticity, the recorder's position and the background context. Secondly, feature engineering methods are used to vectorize the attributes of all nodes, and cosine similarity and other measurement algorithms are used to calculate the similarity between any two nodes, generating a quantified similarity matrix. Based on a preset similarity threshold, the matrix is ​​filtered, and undirected edges are created in the graph structure for node pairs that exceed the threshold. These edges are defined as semantically related edges. All nodes and edges are integrated to construct an interconnected archive semantic network. A community detection algorithm is then applied to analyze the network topology and node attribute similarity, dividing the nodes into semantically similar groups to form an automatic archive classification based on deep semantics.

[0156] During the retrieval phase, the system locates nodes in the semantic network that directly match the retrieval request by query parsing. Then, starting from these nodes, the system executes a graph traversal algorithm to retrieve all nodes that are first-degree or multi-degree related along the semantic association edges. This returns an extended result set that includes directly matching nodes and their semantically related nodes, thereby achieving intelligent retrieval based on semantic association.

[0157] For example, following the specific implementation of the previous step, assume that the approval document regarding the construction of Factory B has been processed, referred to as document A, and its multimodal semantic collaborative representation has been generated. There is also another processed document regarding the procurement of production equipment for Factory B, referred to as document B. First, document A and document B are created as two document nodes, and their respective multimodal semantic collaborative representations, such as the representation of document A, which includes high authenticity, the supportive stance of the competent authority, and the background of the 1950s, are attached to the node as node attributes. Second, the attribute data of these two nodes are extracted, converted into numerical feature vectors, and the cosine similarity between the two is calculated, resulting in a high similarity value of 0.85. Next, 0.85 is compared with the preset association threshold of 0.7. Since it exceeds the threshold, a semantic association edge is established between the two nodes representing archive A and archive B, indicating that the two are semantically related. Then, these two nodes and the edge between them are combined with all other processed archive nodes and edges to construct a complete semantic network. At the same time, the community detection algorithm is run to analyze this network, automatically grouping archive A, archive B, and several other archive nodes related to factory B that are similar in background and stance into a group, forming a semantic category called "Archives Related to the Construction of Factory B in the 1950s". Finally, when a user searches for the approval document for the construction of factory B, the node of archive A is located in the semantic network, and based on the semantic association edge, the directly connected archive node B, as well as another factory site survey archive node associated with archive B, are automatically found, and these three archives are returned as search results, realizing extended retrieval based on deep semantic association.

[0158] This step successfully constructs a network structure that reveals the complex semantic relationships between archives by transforming the archive representation rich in deep semantics into network nodes and related edges. Based on this network, it is possible not only to achieve automatic and accurate classification based on multi-dimensional semantic features such as archive authenticity, stance, and background, but also to support intelligent retrieval that breaks through the limitations of traditional keyword matching.

[0159] Figure 2 This application provides a schematic diagram of the structure of an intelligent archival classification and retrieval system based on multimodal semantic understanding, as shown below. Figure 2 As shown, the system includes: The acquisition module 21 is used to acquire multimodal data of the target archive, the multimodal data including the archive text content, archive image content and the carrier physical information of the target archive; The first processing module 22 is used to perform deep semantic understanding processing on the archival text content to obtain text semantic understanding results, which include the recorded events and the subjective tendencies of the recorder. The second processing module 23 is used to perform visual semantic understanding processing on the content of the archival image, analyze the handwriting style and seal marks in the content of the archival image, and obtain the image semantic understanding result, which includes writer features and authoritative source features. Analysis module 24 is used to perform material and trace analysis on the physical information of the carrier of the target archive to obtain the carrier's age characteristics and preservation status characteristics; The generation module 25 is used to generate a multimodal semantic collaborative representation of the target archive by cross-verifying and semantically associating the recorder's subjective tendencies, the writer's characteristics, and the era characteristics of the carrier based on the formation time and responsible person information in the metadata of the target archive. The multimodal semantic collaborative representation is used to characterize the authenticity of the archive content, the formation background, and the related relationships. The construction module 26 is used to construct a semantic network between archives based on the archive formation background and association relationships represented by the multimodal semantic collaborative representation, and to realize archive classification and retrieval operations based on the semantic network.

[0160] Figure 2 The aforementioned intelligent classification and retrieval system for archives based on multimodal semantic understanding can perform... Figure 1 The implementation principle and technical effects of the intelligent classification and retrieval method for archives based on multimodal semantic understanding described in the illustrated embodiment will not be repeated here. The specific methods by which each module and unit of the intelligent classification and retrieval system for archives based on multimodal semantic understanding in the above embodiments have been described in detail in the embodiments related to this method, and will not be elaborated upon here.

[0161] In one possible design, Figure 2 The illustrated embodiment of an intelligent archive classification and retrieval system based on multimodal semantic understanding can be implemented as a computing device, such as... Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32; The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are invoked and executed by the processing component 32.

[0162] The processing component 32 is used for the above Figure 1 The embodiment describes an intelligent classification and retrieval method for archives based on multimodal semantic understanding.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for intelligent classification and retrieval of archives based on multimodal semantic understanding, characterized in that, include: Acquire multimodal data of the target archive, including the archive text content, archive image content, and physical information of the target archive's carrier; Deep semantic understanding processing is performed on the archival text content to obtain text semantic understanding results, which include the recorded events and the subjective bias of the narrator; Visual semantic understanding processing is performed on the content of archival images. The handwriting style and seal marks in the archival image content are analyzed to obtain the image semantic understanding result, which includes writer characteristics and authoritative source characteristics. Material and trace analysis is performed on the physical information of the carrier of the target archive to obtain the characteristics of the carrier's era and preservation status; Based on the formation time and responsible person information in the metadata of the target archive, the subjective inclination of the recorder, the characteristics of the writer, and the era characteristics of the carrier are cross-verified and semantically associated to generate a multimodal semantic collaborative representation of the target archive. The multimodal semantic collaborative representation is used to characterize the authenticity of the archive content, the formation background, and the related relationships. Based on the archival formation background and relationships represented by the multimodal semantic collaborative representation, a semantic network is constructed among the archives, and the classification and retrieval operations of the archives are realized based on the semantic network.

2. The method according to claim 1, characterized in that, Deep semantic understanding processing is performed on the archival text content to obtain text semantic understanding results. These results include the described events and the narrator's subjective bias, including: The archival text content is divided into sentences to obtain multiple text sentences, and the target objects and action descriptions appearing in the text sentences are identified. The target objects include names of people, places, and organizations, and the action descriptions include verb phrases. Based on the target object and action description identified in the text sentence, an event description framework with action as the core is constructed. The event description framework includes the action executor, the action object, and the time when the action occurs. Extract evaluative words and mood markers from text sentences, wherein the evaluative words include adjectives and adverbs, and the mood markers include interjections and interrogative words; The event description frames are associated with evaluative words and tone markers to determine the narrator's subjective inclination corresponding to each event description frame. The narrator's subjective inclination includes positive evaluation, negative evaluation, and neutral attitude. By integrating all event description frameworks and corresponding narrators' subjective biases, a textual semantic understanding result is formed.

3. The method according to claim 1, characterized in that, Visual semantic understanding processing is performed on the content of archival images. The handwriting style and seal marks within the archival images are analyzed to obtain image semantic understanding results. These results include writer characteristics and authoritative source characteristics, including: Locate and separate the handwriting area image and the seal area image from the content of the archival image; The text in the handwriting region image is analyzed character by character to extract the stroke shape features of each character and the connection relationship features between strokes; Based on the stroke shape features and connection relationship features of characters, a set of handwriting style features representing individual writing habits is calculated; Contour and internal pattern analysis is performed on the image of the seal area to extract the outer contour shape features and graphic arrangement structure features of the seal; The outer contour shape features and graphic arrangement structure features of the seal are matched with the pre-stored authoritative seal template features. When the match is successful, it is determined to be the authoritative source feature corresponding to the seal imprint. The handwriting style feature set is used as the writer feature, and the identified authoritative source feature is used as the authoritative source feature, together forming the image semantic understanding result.

4. The method according to claim 1, characterized in that, Material and trace analysis is performed on the physical information of the target archive's carrier to obtain the carrier's age characteristics and preservation status characteristics, including: Obtain carrier material samples and carrier surface images from the carrier physical information of the target file. The carrier material samples include carrier color characteristics, texture characteristics and fiber structure characteristics. The carrier material sample is analyzed. Based on the color change characteristics and fiber structure characteristics of the carrier material sample, and combined with the pre-stored historical material characteristic database, the production age range of the carrier material sample is inferred to obtain the first age inference result. Analyze the carrier surface image to identify creases and stains. Based on the direction and depth characteristics of the creases, analyze the folding history and stress conditions of the target file; Based on the color and shape characteristics of the stain, infer the source of the stain's composition and the environment in which it formed; Based on the first-year inference results, and combined with the stress analysis of crease marks and the inference of the formation environment of stain marks, the carrier age characteristics and preservation status characteristics of the target archives are determined.

5. The method according to claim 1, characterized in that, Based on the formation time and responsible person information in the metadata of the target archive, the subjective inclination of the recorder, the characteristics of the writer, and the era characteristics of the carrier are cross-verified and semantically correlated to generate a multimodal semantic collaborative representation of the target archive, including: Extract the creation time information and responsible party information from the metadata of the target file; The formation time information is compared with the action occurrence time contained in the event description framework in the text semantic understanding result. When the action occurrence time is within the time range identified by the formation time information, it is recorded as a time consistency verification result. The responsible person information is compared with the writer features and authoritative source features in the image semantic understanding results. When the responsible person information matches the writer identity identified by the writer features or the seal to which the seal belongs identified by the authoritative source features, it is recorded as the identity consistency verification result. The formation time information is compared with the carrier age characteristics to determine whether the material production age range identified by the carrier age characteristics includes the formation time information. If it includes the formation time information, it is recorded as the material age consistency verification result. Based on the results of time consistency verification, identity consistency verification, and material era consistency verification, we correlate the attitude tendencies reflected by the narrator's subjective inclination, the personal writing habits reflected by the writer's characteristics, and the historical background reflected by the era characteristics of the carrier to generate semantically related records. By integrating the semantic association records, a multimodal semantic collaborative representation of the target file is generated.

6. The method according to claim 5, characterized in that, Based on the results of consistency verification in terms of time, identity, and material era, this study correlates the attitudes reflected in the narrator's subjective inclinations, the personal writing habits reflected in the writer's characteristics, and the historical background reflected in the era of the medium to generate semantically related records, including: Based on the time consistency verification results, the degree of consistency between the occurrence time of the events described in the text semantic understanding results and the formation time information in the archive metadata is judged, and the temporal relationship between the described events and the formation of the archive is inferred based on the degree of consistency, which serves as a characteristic of the recording timing. Based on the identity consistency verification results, the matching between the responsible person's information and the writer's characteristics or authoritative source characteristics is judged. Combined with the narrator's subjective tendency in the text semantic understanding results, the narrator's position in the event is inferred as the narration position feature. Based on the consistency verification results of the material era, the logical relationship between the production age of the physical material of the carrier and the time of the formation of the archive is determined. Combined with the historical period information reflected by the characteristics of the carrier era, the physical background constraint of the formation of the archive is constructed as a background authenticity constraint feature. By combining the characteristics of the timing of the record, the characteristics of the recorder's position, and the characteristics of the background authenticity constraint, a semantic association record is generated. The semantic association record includes the assessment conclusion of the authenticity of the archive content, the qualitative description of the recorder's position, and the contextual description of the background of the event.

7. The method according to claim 1, characterized in that, Based on the archival formation background and relationships represented by multimodal semantic collaborative representation, a semantic network is constructed among the archives, and the classification and retrieval operations of the archives are implemented based on the semantic network, including: Each target file is defined as a file node, and the multimodal semantic collaborative representation of the target file is used as the node attribute of the corresponding file node. The node attribute includes the degree of authenticity of the file content, the recorder's position, and the background context of its formation. Calculate the similarity between the node attributes of any two archive nodes among multiple archive nodes; For any two archive nodes, when the similarity meets the preset association conditions, a semantic association edge is established between the two archive nodes. The semantic association edge is used to characterize the semantic association between the two archives in terms of their formation background, the recorder's position, or the authenticity of the content. Combine all file nodes and all semantically related edges to construct a semantic network between target files; In the semantic network, archive nodes are classified according to their node attributes to form a semantic-based archive classification. When a retrieval request is received, the file node that matches the retrieval request is located in the semantic network, and other target file nodes that have semantic association with the target file node are found based on the semantic association edges and returned as retrieval results.

8. An intelligent classification and retrieval system for archives based on multimodal semantic understanding, characterized in that, include: The acquisition module is used to acquire multimodal data of the target archive, including the archive text content, archive image content, and physical information of the target archive's carrier. The first processing module is used to perform deep semantic understanding processing on the archival text content to obtain text semantic understanding results, which include the recorded events and the subjective tendencies of the recorder. The second processing module is used to perform visual semantic understanding processing on the content of the archival image, analyze the handwriting style and seal marks in the content of the archival image, and obtain the image semantic understanding result, which includes writer features and authoritative source features. The analysis module is used to perform material and trace analysis on the physical information of the carrier of the target archive to obtain the carrier's age characteristics and preservation status characteristics; The generation module is used to cross-verify and semantically associate the subjective inclination of the narrator, the characteristics of the writer, and the era characteristics of the carrier based on the formation time and responsible person information in the metadata of the target archive, and generate a multimodal semantic collaborative representation of the target archive. The multimodal semantic collaborative representation is used to characterize the authenticity of the archive content, the formation background, and the related relationships. The construction module is used to build a semantic network between archives based on the archive formation background and association relationships represented by the multimodal semantic collaborative representation, and to realize archive classification and retrieval operations based on the semantic network.

9. A computing device, characterized in that, It includes a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are invoked and executed by the processing component to implement the intelligent classification and retrieval method for archives based on multimodal semantic understanding as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that, The system contains a computer program that, when executed by a computer, implements a method for intelligent classification and retrieval of archives based on multimodal semantic understanding as described in any one of claims 1 to 7.

Images