Archive resource intelligent classification and association indexing method based on multi-modal ai analysis

Abstract

The invention relates to the technical field of archive management, and provides an archive resource intelligent classification and association indexing method based on multi-modal ai analysis, which comprises the following steps: preprocessing collected archive data to form a multi-modal data set comprising text data, image data and audio data; extracting feature vectors of the text data, the image data and the audio data, and fusing the feature vectors into a unified archive representation vector through a cross-modal attention fusion mechanism; constructing a hierarchical classification model, and performing intelligent classification on the archives in combination with a domain knowledge graph; the method comprises the following steps: constructing an associated index of an archive from four dimensions of entities, semantics, time and space and events, establishing a feedback mechanism, dynamically updating and optimizing the associated index, and providing multi-modal retrieval and visual display based on a classification system and the associated index. According to the intelligent classification and association indexing method for the archive resources, the problems that in an existing archive classification and indexing method, the multi-modal processing capacity is insufficient, the association mining depth is insufficient, and the efficiency is low are solved.

Description

Technical Field

[0001] This invention relates to the field of archival management technology, and more specifically, to an intelligent classification and association indexing method for archival resources based on multimodal AI analysis. Background Technology

[0002] The archives preserve a vast amount of historical archives, including various types such as document archives, photographic archives, audio-visual archives, and digital copies of physical archives. These archives are characterized by a large time span, diverse media formats, and complex content relationships.

[0003] Traditional classification methods mainly rely on manual indexing or automated classification based on single text features. They cannot handle the semantic information of non-text archives such as images and audio. When faced with millions or even tens of millions of archive resources, manual classification and indexing are time-consuming and labor-intensive, and consistency is difficult to guarantee. Traditional indexing methods are mostly based on manual association of archival bibliographical items, which makes it difficult to discover potential semantic relationships between archives of different types and periods. They lack a deep understanding of the archival content and cannot achieve accurate content-based retrieval and intelligent recommendation.

[0004] While some studies have attempted to apply AI technology to archival management, they are mostly limited to a single modality (such as processing only text or only images), failing to achieve unified representation and correlation analysis of multimodal archives. Furthermore, they have technical shortcomings in areas such as handwriting recognition, restoration of old images, and semantic ambiguity processing unique to historical archives. Summary of the Invention

[0005] The problem solved by this invention is the lack of multimodal processing capabilities, insufficient depth of association mining, and low efficiency in existing document classification and indexing methods.

[0006] To address the above problems, this invention provides a method for intelligent classification and association indexing of archival resources based on multimodal AI analysis, the method comprising: The collected archival data is preprocessed to form a multimodal dataset that includes text data, image data, and audio data; Feature vectors from text data, image data, and audio data are extracted and fused into a unified archival representation vector through a cross-modal attention fusion mechanism; Construct a hierarchical classification model and combine it with a domain knowledge graph to intelligently classify archives; The system constructs an associated index of archives from four dimensions: entity, semantics, spatiotemporal, and event. It establishes a feedback mechanism to dynamically update and optimize the associated index. Based on the classification system and the associated index, it provides multimodal retrieval and visualization.

[0007] Preferably, the preprocessing of the collected archival data specifically includes: It performs word segmentation, entity recognition, and simplified / traditional character conversion on text data; tilt correction, noise removal, and OCR recognition on image data; and noise reduction and speech-to-text conversion on audio data.

[0008] Preferably, the step of extracting feature vectors from text data, image data, and audio data, and fusing them into a unified archival representation vector through a cross-modal attention fusion mechanism, specifically includes: A pre-trained language model is used to extract text semantic vectors, an improved CNN network is used to extract visual feature vectors, and MFCC is used to extract acoustic features. These are combined with the semantic features of the text transcription results to form a comprehensive audio feature vector. We design a cross-modal attention fusion mechanism that calculates the correlation weights between features of different modalities through self-attention and cross-attention, and fuses text, image, and audio features into a unified archive representation vector.

[0009] Preferably, the construction of the hierarchical classification model, combined with a domain knowledge graph, for intelligent classification of archives specifically includes: Construct a hierarchical classification model that combines archival classification standards with user-defined classification dimensions; A semi-supervised learning method is adopted, which initializes the classification model with a small amount of manually labeled data and optimizes the model on unlabeled data through comparative learning. A domain knowledge graph is introduced as an external constraint to correct the classification results.

[0010] Preferably, the associated index of the archives is constructed from four dimensions: entity, semantics, spatiotemporal, and event, specifically including: Entity Association: Based on the entity information in the archive representation vector, an entity-archive association network is constructed to realize the association of the same entity in different archives; Semantic association: By calculating the cosine similarity of the archive representation vectors, we can mine sets of archives with similar content semantics; Spatiotemporal correlation: Extract timestamps and spatial information from archives, construct a spatiotemporal index, and realize archive correlation based on timeline and geographic space; Event association: By combining domain knowledge graphs, we can identify historical events involved in archives, construct an event-archive association graph, and aggregate related archives of the same event.

[0011] Preferably, a feedback mechanism is established to dynamically update and optimize the related indexes, specifically including: Establish a feedback mechanism to collect feedback from archival management personnel on the classification results and correlations, which will be used for continuous optimization of the model; Regularly perform incremental processing on newly added files, update the classification system and related indexes, and maintain the timeliness of the indexes; A graph neural network is used to dynamically optimize the association index network, thereby improving the accuracy and completeness of the association relationships.

[0012] Preferably, based on a classification system and associated indexes, it provides multimodal retrieval and visualization, specifically including: Based on the constructed classification system and associated indexes, a multimodal retrieval entry point is provided; Using knowledge graph visualization technology, the relationships between files are displayed intuitively, and the analysis of related paths and hierarchical expansion are supported.

[0013] Preferably, it also includes intelligent judgment of classification errors during the classification process, the specific process of which is as follows: Calculate the entropy value and maximum probability value of the category probability distribution output by the classification model, and label samples with high uncertainty or low confidence. By cross-validating multiple heterogeneous classification models, classification results without consensus are identified. By comparing with the error case database, identify current samples that are similar in features to historical error samples; The high uncertainty samples are those with a probability distribution entropy value ≥ 0.8, and the low confidence samples are those with a maximum probability value ≤ 0.6.

[0014] Preferably, the method also includes handling classification errors, the specific process of which is as follows: Error samples with different risk levels are subject to tiered review: low-risk samples are reviewed with AI assistance, while high-risk samples are subject to mandatory manual review. Isolate confirmed erroneous samples and exclude them from the model training dataset; The model is updated using a weighted fine-tuning approach, and specialized training sets are generated for similar errors. The weighted fine-tuning method increases the loss weight of the corrected sample to a preset value.

[0015] Preferably, classification errors are avoided by dynamically adjusting the confidence threshold. The specific process is as follows: The confidence threshold is dynamically adjusted based on the type of archive and the historical error rate. Enhance the model's ability to handle weak feature samples through data augmentation and adversarial training; A knowledge verification module and a taboo rule base are added to the classification output layer to intercept classification results with logical errors.

[0016] This invention provides an intelligent classification and association indexing method for archival resources based on multimodal AI analysis. It has multimodal fusion capabilities, breaks through the limitations of single-modal processing, and realizes unified representation of multiple types of archives such as text, images, and audio. It is especially suitable for the characteristics of multiple media coexisting in historical archives. This invention provides an intelligent classification and association indexing method for archival resources based on multimodal AI analysis. It can combine domain knowledge graphs and semi-supervised learning to solve the problems of limited labeled data and complex semantics in historical archives, thereby improving classification accuracy. This invention provides an intelligent classification and association indexing method for archival resources based on multimodal AI analysis. It can construct association indexes from multiple dimensions such as entities, semantics, spatiotemporal, and events, discover potential associations that are difficult to identify by traditional methods, and improve the utilization value of archival resources. This invention provides a method for intelligent classification and association indexing of archival resources based on multimodal AI analysis, which can significantly reduce manual intervention, improve processing efficiency, and meet the rapid processing needs of provincial archives for massive archives. This invention provides an intelligent classification and association indexing method for archival resources based on multimodal AI analysis. Through feedback mechanisms and incremental update strategies, the system can continuously learn and optimize to adapt to the ever-growing archival resources and changing user needs. Attached Figure Description

[0017] Figure 1 This is a flowchart of the intelligent classification and association indexing method for archival resources based on multimodal AI analysis according to the present invention; Figure 2 This is a flowchart of the encryption process in an embodiment of the present invention; Figure 3 This is a block diagram of an intelligent classification and association indexing system for archival resources based on multimodal AI analysis, as described in an embodiment of the present invention. Detailed Implementation

[0018] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0019] like Figure 1 As shown, this embodiment of the invention provides a method for intelligent classification and association indexing of archival resources based on multimodal AI analysis. The method includes the following steps: Step S101: Preprocess the collected archival data to form a multimodal dataset including text data, image data, and audio data; The process of collecting archival data is as follows: Text-based archives: Paper archives are scanned using a high-definition scanner (resolution 300-600dpi) to generate TIFF format images; for digitized text archives, their structured bibliographical items (such as file number, generating unit, and generating time) and unstructured full-text content are collected.

[0020] Image-based archives: For photos, drawings, seals, and other archives, use a professional scanner to scan at a resolution of 600dpi and save them in lossless TIFF format; for physical archives, take 360° photographs to generate multi-angle JPG images (resolution ≥4000×3000 pixels).

[0021] Audio archives: Oral history recordings, meeting recordings, etc., are converted to WAV format using professional audio acquisition equipment (sampling rate 44.1kHz, bit depth 16bit); audio metadata (such as recording time, speakers information, storage location, etc.) is recorded simultaneously.

[0022] Continue to refer to Figure 1 Step S102: Extract feature vectors from text data, image data, and audio data, and fuse them into a unified archive representation vector through a cross-modal attention fusion mechanism; Continue to refer to Figure 1 Step S103: Construct a hierarchical classification model and combine it with a domain knowledge graph to intelligently classify archives; Continue to refer to Figure 1 Step S104: Construct an association index for archives from four dimensions: entity, semantics, spatiotemporal, and event; establish a feedback mechanism to dynamically update and optimize the association index; and provide multimodal retrieval and visualization based on the classification system and association index.

[0023] In one embodiment, the preprocessing of the collected archival data specifically includes: It performs word segmentation, entity recognition, and simplified / traditional character conversion on text data; tilt correction, noise removal, and OCR recognition on image data; and noise reduction and speech-to-text conversion on audio data.

[0024] Specifically, it could be: Text data preprocessing: Word segmentation: A bidirectional maximum matching algorithm combined with an archive domain dictionary is used for word segmentation; Entity Recognition: A domain-specific entity recognition model based on the BERT-CRF model was trained. The entity types it can recognize include: person names, place names, organization names (including historical names), event names, and time. The entity recognition F1 score is ≥95%. Traditional / Simplified Chinese Conversion: A hybrid conversion model based on rules and statistics is used to convert traditional Chinese characters in archives to simplified Chinese characters with an accuracy rate of ≥99%; a mapping table is established for variant characters to be uniformly replaced.

[0025] Image data preprocessing: Tilt correction: Hough transform is used to detect the edges of text lines, the tilt angle is calculated, and correction is performed through affine transformation. The correction error is ≤0.5°.

[0026] Noise Removal: Adaptive filtering algorithms are used for different noise types: bilateral filtering is used for Gaussian noise; median filtering is used for salt-and-pepper noise.

[0027] Contrast Enhancement: Faded files are enhanced using the Retinex algorithm, with brightness adjusted and contrast enhancement controlled to avoid distortion.

[0028] Special area processing: YOLOv7 is used to train the seal detection model to locate the seal area; Canny edge detection is used to extract the signature outline.

[0029] Audio data preprocessing: Noise reduction processing: Spectral subtraction is used to first calculate the noise spectrum of the first 0.5 seconds of silence segment, and then noise suppression is performed on the speech segment, improving the signal-to-noise ratio by ≥10dB.

[0030] Speech-to-text: Using a Wenet-based archival domain speech recognition model, the recognition accuracy is ≥92% (for clear recordings) and ≥85% (for recordings with high noise).

[0031] In one embodiment, the step of extracting feature vectors from text data, image data, and audio data, and fusing them into a unified archival representation vector through a cross-modal attention fusion mechanism, specifically includes: Pre-trained language models (such as BERT's historical archive domain fine-tuning model) are used to extract text semantic vectors, improved CNN networks (such as adding a seal recognition attention mechanism to ResNet) are used to extract visual feature vectors, MFCC is used to extract acoustic features, and combined with the semantic features of the text transcription results, an audio comprehensive feature vector is formed. This design employs a cross-modal attention fusion mechanism. By calculating the association weights between features from different modalities through self-attention and cross-attention, text, image, and audio features are fused into a unified archival representation vector. Specifically, the mechanism works as follows: First, the text modality is used as the query, while the image and audio modalities are used as the key and value, respectively, for cross-attention calculation. This yields text-guided image fusion features and text-guided audio fusion features. Then, the original text features are concatenated with these two fused features and input into a self-attention layer, ultimately outputting a unified archival representation vector.

[0032] In one embodiment, the construction of a hierarchical classification model (year, institution, issue, retention period), combined with a domain knowledge graph, for intelligent classification of archives specifically includes: Construct a hierarchical classification model that combines archival classification standards with user-defined classification dimensions; A semi-supervised learning method is adopted, which initializes the classification model with a small amount of manually labeled data and optimizes the model on unlabeled data through comparative learning. Introducing a domain knowledge graph as an external constraint (including historical events, institutional evolution, and interpersonal relationships) to correct the classification results.

[0033] In one embodiment, an association index for archives is constructed from four dimensions: entity, semantics, spatiotemporal, and event, specifically including: Entity Association: Based on the entity information in the archive representation vector, an entity-archive association network is constructed to realize the association of the same entity in different archives; Semantic association: By calculating the cosine similarity of the archive representation vectors, we can mine sets of archives with similar content semantics; Spatiotemporal correlation: Extract timestamps and spatial information from archives, construct a spatiotemporal index, and realize archive correlation based on timeline and geographic space; Event Association: By combining domain knowledge graphs, historical events involved in archives are identified, and an event-archive association graph is constructed to aggregate archives related to the same event. Specific methods for event identification can include: first, using named entity recognition to extract entities from the archives; then, querying the knowledge graph to obtain the events these entities participated in and associating these events with the archives; or using a pre-trained event extraction model to analyze the archive text content, extract event trigger words and arguments, and then link them with events in the knowledge graph.

[0034] Furthermore, a feedback mechanism is established to dynamically update and optimize the related indexes, specifically including: Establish a feedback mechanism to collect feedback from archival management personnel on the classification results and correlations, which will be used for continuous optimization of the model; Regularly perform incremental processing on newly added files, update the classification system and related indexes, and maintain the timeliness of the indexes; A graph neural network is used to dynamically optimize the association index network, thereby improving the accuracy and completeness of the association relationships.

[0035] Furthermore, based on the classification system and associated indexes, multimodal retrieval and visualization are provided, specifically including: Based on the constructed classification system and associated indexes, a multimodal retrieval entry point is provided; Using knowledge graph visualization technology, the relationships between files are displayed intuitively, and the analysis of related paths and hierarchical expansion are supported.

[0036] In one embodiment, the method further includes intelligently determining classification errors during the classification process, as detailed below: Calculate the entropy value and maximum probability value of the category probability distribution output by the classification model, and label samples with high uncertainty or low confidence. By cross-validating multiple heterogeneous classification models, classification results without consensus are identified. By comparing with the error case database, identify current samples that are similar in features to historical error samples; The high uncertainty samples are those with a probability distribution entropy value ≥ 0.8, and the low confidence samples are those with a maximum probability value ≤ 0.6.

[0037] Furthermore, it also includes handling classification errors, the specific process of which is as follows: Error samples with different risk levels are subject to tiered review: low-risk samples are reviewed with AI assistance, while high-risk samples are subject to mandatory manual review. Isolate confirmed erroneous samples and exclude them from the model training dataset; The model is updated using a weighted fine-tuning approach, and specialized training sets are generated for similar errors. The weighted fine-tuning method increases the loss weight of the corrected sample to a preset value, which is dynamically set according to the difficulty of correcting the erroneous sample and the historical error rate (preferably 3 times).

[0038] Furthermore, classification errors are avoided by dynamically adjusting the confidence threshold. The specific process is as follows: The confidence threshold is dynamically adjusted based on the type of archive and the historical error rate. Enhance the model's ability to handle weak feature samples through data augmentation and adversarial training; A knowledge verification module and a taboo rule base are added to the classification output layer to intercept classification results with logical errors. The taboo rule base rules can be of the type that files in a certain category could not have appeared before a specific time period, or that certain combinations of entities must belong to a certain major category; the knowledge verification method can be to verify whether the key entities in the classification results match the typical entities under that category.

[0039] Please see Figure 2 In one embodiment, to ensure the security of archival resources, the archival resources are encrypted during the intelligent classification and pre-association indexing process. The specific process is as follows: Based on encrypted archives in historical archives, feature vectors of encrypted archives are extracted according to their encryption levels, and feature vectors of encrypted archives within the same level are categorized. Key features are extracted from the feature vectors of encrypted files of the same level. The key features are features whose number of identical features reaches a preset first threshold, and features whose number of a certain type of features whose similarity reaches a preset second threshold reaches a preset second threshold. When extracting features from the collected archival data, feature similarity is compared in descending order of encryption level. Specifically: First, the characteristics of the archive data are compared with the key features of the feature vector of the highest level of encrypted archives. Archive data with similarity greater than or equal to the preset similarity threshold are encrypted at the same level. The features of the archive data with a similarity less than a preset similarity threshold are compared with the key features of the feature vector of the second-highest encrypted archive. The archive data with a similarity greater than or equal to the preset similarity threshold are encrypted at the same level. The features of the archive data with a similarity lower than the preset similarity threshold are compared with the key features of the feature vector of the third-highest level encrypted archive. This process continues until the archive data is successfully encrypted or all levels of encrypted archives have been compared. If the similarity between the features of the archive data and the key features of the feature vector of the lowest level encrypted archive is less than the preset similarity threshold, then the archive data is not encrypted.

[0040] The first quantity threshold, the second quantity threshold, and the similarity threshold can all be obtained through historical data verification. Feature vector extraction from encrypted files can be performed using a hardware-level trusted execution environment, such as Intel SGX or ARM TrustZone.

[0041] This encryption method not only enables accurate and rapid encryption during the intelligent classification of archival resources, but also allows for the intelligent setting of encryption levels for each type of archival data, thereby improving the security of archival resources.

[0042] In practical applications, the hardware environment upon which this invention's method relies can be: a server (CPU: Intel Xeon Gold 6248, GPU: NVIDIA A100, memory: 256GB), a storage system (capacity ≥ 100TB), a scanner, and an audio acquisition device; the software environment includes: a Linux operating system, a TensorFlow / PyTorch deep learning framework, and an Elasticsearch search engine. A specific example is shown below: Step 1: Multimodal Data Acquisition and Preprocessing of Archival Resources The historical archives of a provincial archive from 1949 to 2000 were digitized, including: Documents: Scanned as JPG images (300dpi), then converted into text using OCR (using a CRNN-based historical document recognition model); Photo archive: Scanned in TIFF format, retaining the original resolution; Oral history archives: converting tape recordings into WAV format and then into text using a speech recognition model (such as Wenet).

[0043] Preprocessing example: Perform conversion of variant Chinese characters on the OCR results (e.g., unify "後" to "后"), perform skew correction on the image (detect the text line direction using the Hough transform), and perform noise reduction on the audio (using spectral subtraction).

[0044] Step 2: Multi-modal feature fusion representation Text feature extraction: Use the BERT model fine-tuned on the historical archives corpus of this province to output a 768-dimensional text vector; Image feature extraction: Adopt an improved ResNet-50, add a seal detection branch to the network (a seal detector trained using YOLOv5), and extract 2048-dimensional image features; Fusion mechanism: Adopt a cross-modal Transformer, where the self-attention module learns the feature associations within the same modality, and the cross-attention module learns the association weights between different modalities, and finally outputs a 512-dimensional fused feature vector.

[0045] Step 3: Intelligent classification based on fused features Classification system: Adopt a three-level classification of year - institution - problem, where the problem dimension includes 12 major categories such as politics, economy, culture, etc.; Semi-supervised training: Use 10,000 manually labeled data to initialize the classification model (adopting a multi-layer perceptron), and train it on 100,000 unlabeled data through a contrastive loss function; Knowledge constraint: Introduce the knowledge graph of historical events in this province (including 5,000 event nodes), and correct classification errors through entity linking.

[0046] Step 4: Construction of multi-dimensional association index Entity association: Extract the person entities in the archives through named entity recognition, and construct a person - archive bipartite graph, such as associating all documents, photos, and oral archives related to Zhang San; Semantic association: Calculate the cosine similarity of the archive fusion vectors, and regard archives with a similarity ≥ 0.85 as semantically related; Spatio-temporal association: Extract the time information and space information in the archives (such as the provincial capital city A), and construct a spatio-temporal index tree; Event association: Identify the events involved in the archives based on the knowledge graph, and aggregate the relevant archives to form an event archive set.

[0047] Step 5: Dynamic update and optimization of the association index Feedback mechanism: The archive administrator marks the incorrect classification results and association relationships through the Web interface, and summarizes them once a week for model fine-tuning; Incremental update: Process the newly added archives monthly, update the classification model through incremental training, and update the association index using a graph incremental algorithm; GNN optimization: GraphSAGE is used to learn the representation of the association network, optimize the association weights between entities, and improve the association accuracy.

[0048] Step 6: Intelligent Search and Visual Display Search function: Supports searching by inputting text (such as agricultural policies in the 1950s) or uploading images (such as photos of historical scenes); Visualization: Neo4j combined with D3.js is used to visualize the relationship graph. Users can click on a file node to view the files that are directly and indirectly related to it.

[0049] like Figure 3 As shown in the figure, this embodiment of the invention also provides an intelligent classification and association indexing system for archival resources based on multimodal AI analysis, the system comprising: The preprocessing module is used to preprocess the collected archival data to form a multimodal dataset including text data, image data, and audio data; The feature extraction module is used to extract feature vectors from text data, image data, and audio data, and fuse them into a unified archival representation vector through a cross-modal attention fusion mechanism; The intelligent classification module is used to build a hierarchical classification model and combine it with a domain knowledge graph to intelligently classify archives. The association index module is used to construct an association index for archives from four dimensions: entity, semantics, spatiotemporal, and event. It establishes a feedback mechanism to dynamically update and optimize the association index. Based on the classification system and association index, it provides multimodal retrieval and visualization. This invention also provides a storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when it is run.

[0050] Specifically, in this embodiment, the storage medium can be configured to store a computer program for performing the following steps: Step S1: Preprocess the collected archival data to form a multimodal dataset including text data, image data, and audio data; Step S2: Extract feature vectors from text data, image data, and audio data, and fuse them into a unified archive representation vector through a cross-modal attention fusion mechanism; Step S3: Construct a hierarchical classification model and combine it with a domain knowledge graph to intelligently classify the archives; Step S4: Construct an association index for the archives from four dimensions: entity, semantics, spatiotemporal, and event. Establish a feedback mechanism to dynamically update and optimize the association index. Based on the classification system and association index, provide multimodal retrieval and visualization.

[0051] This invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0052] Specifically, the aforementioned electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the aforementioned processor, and the input / output device is connected to the aforementioned processor.

[0053] Specifically, in this embodiment, the processor can be configured to perform the following steps via a computer program: Step S1: Preprocess the collected archival data to form a multimodal dataset including text data, image data, and audio data; Step S2: Extract feature vectors from text data, image data, and audio data, and fuse them into a unified archive representation vector through a cross-modal attention fusion mechanism; Step S3: Construct a hierarchical classification model and combine it with a domain knowledge graph to intelligently classify the archives; Step S4: Construct an association index for the archives from four dimensions: entity, semantics, spatiotemporal, and event. Establish a feedback mechanism to dynamically update and optimize the association index. Based on the classification system and association index, provide multimodal retrieval and visualization.

[0054] If the above functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0055] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0056] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0057] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0058] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for intelligent classification and association indexing of archival resources based on multimodal AI analysis, characterized in that, The method includes: The collected archival data is preprocessed to form a multimodal dataset that includes text data, image data, and audio data; Feature vectors from text data, image data, and audio data are extracted and fused into a unified archival representation vector through a cross-modal attention fusion mechanism; Construct a hierarchical classification model and combine it with a domain knowledge graph to intelligently classify archives; The system constructs an associated index of archives from four dimensions: entity, semantics, spatiotemporal, and event. It establishes a feedback mechanism to dynamically update and optimize the associated index. Based on the classification system and the associated index, it provides multimodal retrieval and visualization.

2. The method for intelligent classification and association indexing of archival resources based on multimodal AI analysis according to claim 1, characterized in that, The preprocessing of the collected archival data specifically includes: It performs word segmentation, entity recognition, and simplified / traditional character conversion on text data; tilt correction, noise removal, and OCR recognition on image data; and noise reduction and speech-to-text conversion on audio data.

3. The method for intelligent classification and association indexing of archival resources based on multimodal AI analysis according to claim 1, characterized in that, The extraction of feature vectors from text data, image data, and audio data, and their fusion into a unified archival representation vector through a cross-modal attention fusion mechanism, specifically includes: A pre-trained language model is used to extract text semantic vectors, an improved CNN network is used to extract visual feature vectors, and MFCC is used to extract acoustic features. These are combined with the semantic features of the text transcription results to form a comprehensive audio feature vector. We design a cross-modal attention fusion mechanism that calculates the correlation weights between features of different modalities through self-attention and cross-attention, and fuses text, image, and audio features into a unified archive representation vector.

4. The method for intelligent classification and association indexing of archival resources based on multimodal AI analysis according to claim 1, characterized in that, The construction of a hierarchical classification model, combined with a domain knowledge graph, for intelligent classification of archives specifically includes: Construct a hierarchical classification model that combines archival classification standards with user-defined classification dimensions; A semi-supervised learning method is adopted, which initializes the classification model with a small amount of manually labeled data and optimizes the model on unlabeled data through comparative learning. A domain knowledge graph is introduced as an external constraint to correct the classification results.

5. The method for intelligent classification and association indexing of archival resources based on multimodal AI analysis according to claim 1, characterized in that, The archive's relational index is constructed from four dimensions: entity, semantics, spatiotemporal, and event. Specifically, it includes: Entity Association: Based on the entity information in the archive representation vector, an entity-archive association network is constructed to realize the association of the same entity in different archives; Semantic association: By calculating the cosine similarity of the archive representation vectors, we can mine sets of archives with similar content semantics; Spatiotemporal correlation: Extract timestamps and spatial information from archives, construct a spatiotemporal index, and realize archive correlation based on timeline and geographic space; Event association: By combining domain knowledge graphs, we can identify historical events involved in archives, construct an event-archive association graph, and aggregate related archives of the same event.

6. The method for intelligent classification and association indexing of archival resources based on multimodal AI analysis according to claim 5, characterized in that, Establish a feedback mechanism to dynamically update and optimize the related indexes, specifically including: Establish a feedback mechanism to collect feedback from archival management personnel on the classification results and correlations, which will be used for continuous optimization of the model; Regularly perform incremental processing on newly added files, update the classification system and related indexes, and maintain the timeliness of the indexes; A graph neural network is used to dynamically optimize the association index network, thereby improving the accuracy and completeness of the association relationships.

7. The method for intelligent classification and association indexing of archival resources based on multimodal AI analysis according to claim 6, characterized in that, Based on a classification system and related indexes, it provides multimodal retrieval and visualization, specifically including: Based on the constructed classification system and associated indexes, a multimodal retrieval entry point is provided; Using knowledge graph visualization technology, the relationships between files are displayed intuitively, and the analysis of related paths and hierarchical expansion are supported.

8. The method for intelligent classification and association indexing of archival resources based on multimodal AI analysis according to claim 1, characterized in that, It also includes intelligent detection of classification errors during the classification process, the specific process of which is as follows: Calculate the entropy value and maximum probability value of the category probability distribution output by the classification model, and label samples with high uncertainty or low confidence. By cross-validating multiple heterogeneous classification models, classification results without consensus are identified. By comparing with the error case library, identify current samples that have similar characteristics to historical error samples.

9. The method for intelligent classification and association indexing of archival resources based on multimodal AI analysis according to claim 8, characterized in that, It also includes handling classification errors, the specific process of which is as follows: Error samples with different risk levels are subject to tiered review: low-risk samples are reviewed with AI assistance, while high-risk samples are subject to mandatory manual review. Isolate confirmed erroneous samples and exclude them from the model training dataset; The model is updated using a weighted fine-tuning approach, and specialized training sets are generated for similar errors. The weighted fine-tuning method increases the loss weight of the corrected sample to a preset value.

10. The method for intelligent classification and association indexing of archival resources based on multimodal AI analysis according to claim 9, characterized in that, Classification errors are avoided by dynamically adjusting the confidence threshold. The specific process is as follows: The confidence threshold is dynamically adjusted based on the type of archive and the historical error rate. Enhance the model's ability to handle weak feature samples through data augmentation and adversarial training; A knowledge verification module and a taboo rule base are added to the classification output layer to intercept classification results with logical errors.

Images