A multimodal intelligent AI classification system and method for archive organization

Abstract

The present invention belongs to the field of data processing and discloses a multimodal intelligent AI classification system and method for archive organization. The system and method include introducing a cross-modal consistency loss function and a structured constraint loss in a deployed dual-channel generator, and performing mutual generation scoring on original multimodal data and evolved multimodal data based on a modal quality assessor. The system performs cyclic optimization through a conflict identification and resolution mechanism, extracting features based on the enhanced multimodal data to obtain a quality score triplet. The system uses a PPO algorithm for dynamic routing decisions and introduces a closed-loop optimization for quality assessment. According to the decision action execution path rule, a multimodal feature matrix is ​​constructed based on comprehensive features. A multimodal classifier is designed based on an Attention-Transformer, a modal weight adaptive adjustment module is introduced, and a classification result verification mechanism is added to form a closed-loop optimization. The system realizes intelligent AI classification of multimodal archives.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and more specifically, to a multimodal intelligent AI classification system and method for archive organization. Background Art

[0002] Archival data typically consists of two modalities: images (e.g., scanned documents) and text (e.g., electronic documents). The quality of the data varies (e.g., blurry images, non-standard text layout). Traditional single-modality processing methods struggle to effectively utilize multimodal information and have limited capabilities for processing incomplete data (e.g., isolated modality data).

[0003] Therefore, when classifying and processing multimodal data, it is necessary to run it on different devices (such as edge devices and high-performance servers). Due to the differences in modalities, the transmission method, resource usage, and processing methods required at runtime are all different. Considering the efficiency of processing, modal routing is a feasible processing method, but how to dynamically select the processing path based on data quality and device resources while ensuring classification accuracy and efficiency is a complex problem. In addition, when classifying modal data, content recognition must be performed first, which involves the conversion of data content or format. Different modalities have different conversion methods, and there may be cross-modal inconsistencies (such as mismatches between images and text descriptions) or low quality (such as unreasonable pseudo-text generation). Therefore, ensuring conversion quality is also a key challenge.

[0004] In view of this, the present invention proposes a multimodal intelligent AI classification system and method for archive organization to solve the above problems. Summary of the Invention

[0005] In order to overcome the above-mentioned defects of the prior art and to achieve the above-mentioned purpose, the present invention provides the following technical solutions: a multimodal intelligent AI classification system for archive organization, comprising a data acquisition and transmission module: a scanned copy of a paper archive is obtained by a scanner or a camera, and combined with an electronic document to form original multimodal data;

[0006] Perception layer module: This module receives raw multimodal data and introduces a cross-modal consistency loss function and a structured constraint loss into the deployed dual-channel generator to generate associated image features or pseudo-text descriptions to form evolving multimodal data.

[0007] Based on the modal quality assessor, the original multimodal data and the evolved multimodal data are scored for mutuality; through the conflict identification and resolution mechanism, the enhanced multimodal data is output;

[0008] Feature extraction and optimization module: Extracts features from enhanced multimodal data to obtain quality score triples; uses the PPO algorithm for dynamic routing decisions and introduces closed-loop optimization for quality assessment to optimize and update the quality score triples and output decision actions; executes path rules based on decision actions and outputs comprehensive features;

[0009] Classification and verification module: Construct a multimodal feature matrix based on comprehensive features, and design a multimodal classifier based on Attention-Transformer. Introduce a modal weight adaptive adjustment module, set the multimodal feature matrix as input, and output the classification result; add a classification result verification mechanism to form a closed-loop optimization; obtain the final archive classification result.

[0010] Preferably, the method for acquiring the enhanced multimodal data includes:

[0011] Scan paper files are obtained through scanners or cameras, combined with electronic documents to form original multimodal data, which is then transmitted to the perception layer through the API interface;

[0012] A dual-channel generator is deployed in the perception layer. It learns the cross-modal mapping relationship between images and text through adversarial training. A cross-modal consistency loss function is introduced into the adversarial training, along with a structured constraint loss. Differentiated quality thresholds are pre-set for different modal file types. The generator generates associated image features based on text, or pseudo-text descriptions based on images, to acquire evolving multimodal data.

[0013] Design a modality quality evaluator to score the co-generacy of original multimodal data and evolved multimodal data. The multi-dimensional evaluation indicators of the modality quality evaluator include image quality or text quality and cross-modal consistency.

[0014] Introducing conflict identification and resolution mechanisms to output optimal evolving multimodal data;

[0015] The output is enhanced multimodal data, including original multimodal data, evolved multimodal data and symbiosis scores.

[0016] Preferably, the dual-channel generator includes:

[0017] For image modality data, we deploy Better-DeblurGAN-v3 to automatically inpaint images and extract local key regions. This includes adding multi-scale convolution kernels to the DeblurGAN-v3 encoder to capture image interference at different scales, and adding an RPN region proposal network to automatically locate and enhance key regions.

[0018] For text data, the dynamic blocking strategy of the VS-ADSP document structure parser is used to identify non-standard typesetting, and the SB-GNN graph neural network is combined to reconstruct the document logical topology and convert non-standard typesetting documents into standard versions; among them, the VS-ADSP document structure parser includes the introduction of document segmentation evolution based on visual-semantic combination in ADSP, and the SB-GNN graph neural network includes adding visual features and semantic features to the node features of GNN.

[0019] For isolated modal data, the conditional variational autoencoder (CVAE) is introduced into the cross-modal generative adversarial network (Cross-GAN), and variational constraints are added to the generator of Cross-GAN. Specifically, for extremely isolated modal data, visual question answering enhancement (VQA-E) is introduced, and the semantic description of the image is generated through the pre-trained VQA-E model.

[0020] Preferably, the method of introducing a conflict identification and resolution mechanism includes:

[0021] A threshold for cross-modal consistency is predefined. If the cross-modal consistency value output by the modality quality evaluator is less than the threshold, it is marked as a conflict.

[0022] For conflicting original multimodal data and evolved multimodal data, a dual-channel generator is used to generate multiple possible evolved multimodal data, and then the modal quality evaluator is used to re-evaluate the cross-modal consistency until the conflict is resolved.

[0023] Preferably, the method for obtaining the comprehensive features includes:

[0024] Perform preliminary modal feature extraction on the enhanced multimodal data to obtain the quality score triple (s_img, s_txt, s_consist);

[0025] The PPO algorithm is used for dynamic routing decisions. The input is set as a quality score triplet (s_img, s_txt, s_consist). A closed-loop optimization of quality assessment is introduced. If the quality score of the modal data is lower than the preset threshold, it is fed back to the dual-channel generator to trigger data regeneration. Modal feature extraction is performed again based on the regenerated modal data, and the quality score triplet is updated. After the closed-loop optimization of quality assessment is completed, the final quality score triplet (s_img, s_txt, s_consist) is obtained.

[0026] Use the final quality score triplet to import into the PPO algorithm and output the final decision action;

[0027] Execute path rules according to decision actions and output comprehensive features.

[0028] Preferably, the method of performing preliminary extraction of modal features on the enhanced multimodal data to obtain a quality score triplet (s_img, s_txt, s_consist) includes:

[0029] Step Q1: Perform resolution detection on the image data in the evolving multimodal data, select an appropriate feature extraction model based on the resolution, and extract the visual feature vector f_img based on the features including layout, seal position, signature area confidence, and initial OCR recognition results.

[0030] Perform rule extraction on text data in evolving multimodal data, extract key fields, and use them as auxiliary input for DistilBERT. Features include keyword frequency, named entities, and paragraph topic distribution, and extract the semantic feature vector f_txt.

[0031] Step Q2: Calculate the OCR recognition accuracy based on the edit distance and semantic rationality, introduce multi-scale analysis to calculate the blur and SSIM values ​​in the Laplacian variance and SSIM structure integrity, use YOLOv8 to detect the seal area, take the highest confidence score as the seal detection confidence, and output the image quality score s_img through the calculation formula; the calculation formula is , is the regularization coefficient, is the variance of each indicator or other regularization term;

[0032] The semantic coherence score is calculated based on DistilBERT's NextSentencePrediction, and the document layout is analyzed using SB-GNN to evaluate the structural completeness. The text quality score s_txt is output through the calculation formula. The calculation formula is: ;

[0033] Step Q3: Calculate the similarity between the image OCR result and the electronic text title, use the CLIP model to calculate the image-text matching degree between the image and the text, and output the cross-modal consistency score s_consist. The calculation formula is: s_consist= Similarity+ CLIP image and text matching degree;

[0034] in, 、 、 、 、 、 、 and is the weight coefficient, which matches the preset weight based on the archive metadata;

[0035] Step Q4: Output the quality score triplet (s_img, s_txt, s_consist).

[0036] Preferably, the method for making dynamic routing decisions using the PPO algorithm includes:

[0037] Define the state space S = [s_img, s_txt, s_consist, CPU load, remaining power, network bandwidth, storage capacity], and define the action space A = {pure image path, pure text path, multimodal fusion path};

[0038] Define the reward function , using the initial quality score as the initial reward for reinforcement learning;

[0039] Dynamically adjust hyperparameters for edge devices and high-performance servers, introduce a long-term reward mechanism, and use discounted returns to calculate long-term performance;

[0040] Design routing rules as soft constraints, and select paths based on probability distribution, including:

[0041] If s_img>0.9 and s_consist>0.8, the pure image path is selected with a probability of 90% and the multimodal fusion path is selected with a probability of 10%;

[0042] If s_txt>0.9 and s_consist>0.8, the plain text path is selected with a probability of 90% and the multimodal fusion path is selected with a probability of 10%;

[0043] If s_img < 0.4, the probability of switching to the text path is 80%, and the probability of selecting the multimodal fusion path is 20%;

[0044] If s_txt < 0.4, the probability of switching to the image path is 80%, and the probability of selecting the multimodal fusion path is 20%;

[0045] If 0.4≤s_img, s_txt≤0.9 or s_consist<0.6, the multimodal fusion path is selected with a probability of 70% and the single-modal path is selected with a probability of 30%;

[0046] The output is the decision action a∈A.

[0047] Preferably, the execution method of the path rule includes:

[0048] For the pure image path, deep CNN is used to extract visual features and combined with the initial OCR recognition results to generate text features;

[0049] For text modalities, we use a deep NLP model to extract semantic features and combine them with the output of DistilBERT to generate structured features.

[0050] For the multimodal fusion path, a cross-modal SP-Transformer model is used to fuse the structured and textual features of the original multimodal data and the evolved multimodal data to generate a generative modality embedding vector;

[0051] The cross-modal SP-Transformer model is based on the two-stream Transformer framework, introduces MT-ARNet to process fuzzy features, and adds NSR to intervene in conflicting data in the original multimodal data and evolved multimodal data;

[0052] The text features, structural features and embedding vectors are recorded as comprehensive features.

[0053] Preferably, the method for obtaining the final archive classification result includes:

[0054] Step C1: construct a multimodal feature matrix based on the comprehensive features, wherein the multimodal feature matrix forms a unified feature representation by normalizing and aligning the dimensions of different modal features;

[0055] Step C2: Design a multimodal classifier based on the Attention-Transformer multi-layer perception fusion network. Introduce a modality weight adaptive adjustment module into the Transformer attention mechanism to dynamically adjust the weights of image, text, and structured features according to the modality importance of different file types.

[0056] Step C3: For the archive classification task, a predefined archive type label set is used, and a supervised learning method is used to pre-train the multimodal classifier. The training objective is to minimize the classification cross entropy loss, and a regularization term is introduced at the same time.

[0057] Step C4: Input the multimodal feature matrix into the trained multimodal classifier and output the classification probability distribution of the archive; select the archive type label with the highest confidence as the final classification result based on the probability distribution;

[0058] Step C5: Design a classification result verification mechanism. For archival data whose classification confidence is lower than the preset threshold, trigger the manual review process and feed the manual review results back to the dual-channel generator to form a closed-loop optimization.

[0059] Step C6: Output the final archive classification results, including the archive type label, classification confidence and corresponding multimodal feature matrix, for subsequent archive management and retrieval.

[0060] The present invention also discloses a multimodal intelligent AI classification method for archive organization, comprising:

[0061] Step S1: Obtain a scanned copy of a paper file through a scanner or camera, combine it with the electronic document to form original multimodal data, and transmit it to the perception layer through the API interface;

[0062] Step S2: In the perception layer, a dual-channel generator is used to generate multimodal data through adversarial training. A cross-modal consistency loss function and a structured constraint loss are introduced to generate associated image features or pseudo-text descriptions based on a preset differential quality threshold.

[0063] Step S3: Design a modality quality evaluator to score the co-generacy of the original multimodal data and the generated multimodal data based on image quality, text quality, and cross-modal consistency indicators; through a conflict identification and resolution mechanism, mark data below the cross-modal consistency threshold, generate multiple possible multimodal data, re-evaluate them, and output enhanced multimodal data;

[0064] Step S4: Perform modal feature extraction on the enhanced multimodal data to obtain the quality score triple (s_img, s_txt, s_consist); use the PPO algorithm to make dynamic routing decisions, set the state space, action space and reward function, and introduce closed-loop optimization. If the quality score is lower than the threshold, data regeneration is triggered, the quality score triple is updated, and the decision action is output;

[0065] Step S5: Execute path rules according to the decision action, including pure image path, pure text path and multimodal fusion path, use deep CNN, NLP model or cross-modal SP-Transformer model to extract features and output comprehensive features;

[0066] Step S6: Construct a multimodal feature matrix based on the comprehensive features, and form a unified feature representation through normalization and dimension alignment; design a multimodal classifier based on Attention-Transformer, introduce a modal weight adaptive adjustment module, and use supervised learning pre-training;

[0067] Step S7: Input the multimodal feature matrix into the classifier, output the file classification probability distribution, and select the highest confidence label as the classification result; for file data with confidence below the threshold, trigger manual review and feedback optimization;

[0068] Step S8: Output the final archive classification results, including type labels, classification confidence, and multimodal feature matrix.

[0069] The technical effects and advantages of the multimodal intelligent AI classification system for archive organization of the present invention are as follows:

[0070] 1. Through multimodal data enhancement, quality assessment, dynamic routing and closed-loop optimization, it can effectively handle common problems in archival data such as noise, ambiguity, non-standard formats, and significantly improve classification accuracy and robustness.

[0071] 2. The dynamic routing decision and modal weight adaptive adjustment module of the PPO algorithm enable the system to dynamically adjust the processing strategy according to data quality, device resources and task requirements, which not only improves computing efficiency but also enhances the system's adaptability.

[0072] 3. Through multi-level closed-loop optimization (data enhancement, feature extraction, and classification verification), a high degree of intelligence and automation is achieved, reducing the need for human intervention. The introduction of a classification result verification mechanism ensures the accuracy of low-confidence data, while continuous model optimization through human feedback balances efficiency and accuracy.

[0073] 4. Through technologies such as cross-modal consistency loss, cross-modal generative adversarial networks (Cross-GAN), and the CLIP model, we achieve deep fusion and complementarity of multimodal data such as images and text. This cross-modal capability significantly improves the comprehensiveness and accuracy of archive classification.

[0074] In summary, it can significantly improve the efficiency of archive management, support the digitization of old archives, and provide strong support for intelligent retrieval and cross-domain applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 This is a schematic diagram of the structure of a multimodal intelligent AI classification system for file organization according to the present invention;

[0076] Figure 2 Schematic diagram of the steps of a multimodal intelligent AI classification method for file organization according to the present invention. DETAILED DESCRIPTION

[0077] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0078] Example 1

[0079] See also Figure 1 and Figure 2 As shown, the multimodal intelligent AI classification system for file organization described in this embodiment includes:

[0080] Archival data typically includes multiple modalities such as paper scans and electronic documents, with large variations in data format and quality. Unified processing and enhancement of these heterogeneous data is a challenge. The semantic association between images and text is difficult to accurately model, especially in archival data, where images may be blurry or damaged, and text may be in non-standard formatting. Generating high-quality cross-modal data is a challenge. Designing a mechanism that comprehensively evaluates the quality of multimodal data ensures that the generated data meets high-quality standards in terms of image, text, and cross-modal consistency. During the generation of multimodal data, inconsistencies between image and text descriptions may occur. Identifying and resolving these conflicts is a key issue.

[0081] To address the above issues, this design uses adversarial training to learn the cross-modal mapping relationship between images and text, introduces a cross-modal consistency loss function and a structured constraint loss, and sets differentiated quality thresholds for different modalities to address the complexity of cross-modal mapping. Multi-dimensional evaluation indicators (image quality, text quality, and cross-modal consistency) are designed to provide an objective quality scoring mechanism. Through conflict identification and resolution, the output of enhanced multimodal data is ensured to be optimal, solving the problem of processing conflicting data. Specific contents include:

[0082] Scanners or cameras are used to obtain scanned paper documents (image data, such as JPG and PNG), which are combined with electronic documents (text data, such as PDF and Word) to form raw multimodal data, which is then transmitted to the perception layer through the API interface.

[0083] The perception layer deploys a dual-channel generator, which learns the cross-modal mapping relationship between images and text through adversarial training to ensure the authenticity and consistency of the generated data. A cross-modal consistency loss function is introduced in the adversarial training to enforce semantic alignment between the generated text and the original image (e.g., calculating image-text similarity through the CLIP model). For special archives such as engineering drawings, a structured constraint loss is added (e.g., the generated CAD description text must conform to the SVG vector primitive syntax). Differentiated quality thresholds are pre-set for different archive types under different modalities (e.g., medical archives require a NER accuracy of ≥90%, engineering drawings require a SSIM of ≥0.85). The generator generates associated image features from text (e.g., generating abstract symbols from "circuit diagrams") or pseudo-text descriptions from images (e.g., using a VQA model to explain the content of a drawing). For example, for a document consisting only of scanned drawings, CVAE can generate multiple possible pseudo-text descriptions (e.g., "circuit diagram, 2020 design" or "architectural drawing, floor plan") and select the optimal description through a discriminator. Acquire and generate multimodal data; (For example, for a scanned copy of an ambiguous contract signature page, the generator not only completes the text but also infers the signatory's position (such as "Technical Director") based on the context and generates a corresponding virtual signature image.)

[0084] The dual-channel generator specifically includes:

[0085] Scanned documents may be blurry or damaged, making automatic repair and extraction of key areas (such as seals and signatures) challenging. Text in archives may contain non-standard layouts (such as handwritten or out of order), making parsing and reconstructing the document's logical structure challenging. In some cases, there may only be single-modal data (such as only images or only text), requiring the generation of high-quality pseudo-modal data (such as generating text descriptions based on images).

[0086] To address the above issues, this design uses an improved Better-DeblurGAN-v3, which captures image interference at different scales through multi-scale convolution kernels and automatically locates key areas in combination with the RPN region proposal network to address image blur and key information extraction. Through the VS-ADSP document structure parser (introducing visual-semantic joint segmentation) and the SB-GNN graph neural network (fusing visual and semantic features), logical reconstruction of non-standard typesetting documents is achieved, addressing the issue of text non-standardization. The conditional variational autoencoder (CVAE) is introduced into Cross-GAN to add variational constraints, and combined with visual question answering enhancement (VQA-E) to generate semantic descriptions, addressing the challenge of generating pseudo-modal data from single-modal data. Specific contents include:

[0087] For image modality data, Better-DeblurGAN-v3 (a network further optimized based on the improved lightweight scan quality enhancement network of U-Net++) is deployed to automatically repair images (for example, repair blur, tilt, and shadow interference in images) and extract local key areas (such as signatures and seals). Among them, Better-DeblurGAN-v3 includes: adding multi-scale convolution kernels (such as 3x3, 5x5, and 7x7) to the encoder of DeblurGAN-v3 (a network improved based on the lightweight scan quality enhancement network of U-Net++) to capture image interference of different scales (such as large shadows and small folds), and adding the RPN Region Proposal Network (Region Proposal Network) combined with the Faster R-CNN structure to automatically locate and enhance key areas;

[0088] When adding the RPN region proposal network, it is necessary to train the RPN end-to-end, taking into account the joint optimization of target existence judgment and position regression. The specific process includes:

[0089] The loss function of RPN can be defined as: ;

[0090] in, is the classification loss (determining whether the area is a key area), The model predicts The probability that an anchor box is a key area (foreground); is the true label (1 for positive samples, 0 for negative samples), and the binary cross entropy loss is used. , is the number of anchor boxes in a mini-batch (usually 256);

[0091] is the regression loss (locating the region boundary), The model predicts The boundary offset of the positive sample anchor box, is the actual boundary offset (calculated by the GT box and the anchor box), Using smooth L1 loss, ,and , is the number of positive sample anchor boxes (usually 128);

[0092] is the balance coefficient, which is used to adjust the weights of classification and regression tasks, usually set to =10 (the regression loss amplitude is small and needs to be amplified).

[0093] In image modality processing, by improving DeblurGAN-v3, adding multi-scale convolution kernels and RPN region proposal networks, the system can more accurately repair blurred images and extract key area features, significantly improving the availability of image data.

[0094] For text data, the VS-ADSP document structure parser uses a dynamic segmentation strategy to identify non-standard layouts (such as mixed columns and nested tables). Combined with the SB-GNN graph neural network, it reconstructs the document's logical topology and converts non-standard layout documents into standard formats. The VS-ADSP document structure parser incorporates the evolution of document segmentation based on vision-semantic joint segmentation (VSJS) within ADSP. Specifically, ADSP combines image layout information (such as visual segmentation lines in scanned documents) with text semantic information (such as semantic boundaries of titles and body text) to improve the parsing accuracy of complex layouts (such as mixed columns and nested tables). For example, in a multi-column medical report, VS-ADSP can accurately separate the columns "Diagnosis Results" and "Laboratory Data." The SB-GNN graph neural network incorporates visual features (such as geometric information of the layout bounding box) and semantic features (such as word embeddings extracted by BERT) into the GNN node features.

[0095] The node update formula of GNN can be defined as: ; By assigning independent weights to self-loops and neighbors and , or share weights, while retaining the node's own characteristics and neighbor characteristics, enhancing the model's expressiveness. Representation node In the The feature vector of the layer, 、 Representation node In the The feature vector of the layer, represents the normalization coefficient, which is usually related to the node degree or edge weight, Representation node The set of neighbor nodes of Represents a nonlinear activation function (such as ReLU, Sigmoid).

[0096] In text modality processing, by introducing visual-semantic joint document segmentation and graph neural networks, the system can effectively parse non-standard typesetting documents and reconstruct logical topology, greatly improving the structured processing capabilities of text data.

[0097] For isolated modal data (such as only scanned drawings), the conditional variational autoencoder CVAE (Conditional Variational Autoencoder) is introduced into the cross-modal generative adversarial network Cross-GAN, and variational constraints are added to the Cross-GAN generator to ensure the diversity and authenticity of the generated data. Specifically, for extremely isolated modal data (such as historical photos without any text labels), visual question answering enhancement VQA-E (Visual Question Answering Enhancement) is introduced to generate semantic descriptions of the images through pre-trained VQA-E models (such as CLIP-ViT).

[0098] The loss function after Cross-GAN introduces the conditional variational autoencoder CVAE can be optimized as follows: ; represents the KL divergence term of the variational constraint, is the balance coefficient, represents a generator, represents the discriminator.

[0099] Through the dual-channel generator of the perception layer module, the system introduces a cross-modal consistency loss function and a structured constraint loss in adversarial training, effectively generating correlated image features or pseudo-text descriptions. This approach significantly improves the quality and consistency of multimodal data and addresses the issues of information loss or incompleteness in traditional single-modal processing.

[0100] For isolated modal data, by introducing conditional variational autoencoder (CVAE) and visual question answering enhancement (VQA-E), the system can generate high-quality cross-modal descriptions to make up for the shortcomings of single modal data.

[0101] Design a modality quality evaluator (based on a deep learning classifier) ​​to score the mutuality of original and generated multimodal data; for example, evaluate the accuracy of text restoration by Better-DeblurGAN-v3 or the authenticity of generated images. The scoring results are used to assign weights for subsequent modality fusion. The multi-dimensional evaluation metrics of the modality quality evaluator include image quality (e.g., PSNR (peak signal-to-noise ratio), SSIM (structural similarity), seal detection confidence) or text quality (e.g., BERT semantic coherence score, named entity recognition (NER) accuracy), and cross-modal consistency (e.g., image-text matching calculated using the CLIP model).

[0102] In archival data, there may be semantic deviations between images and texts, making it difficult to accurately determine whether the image and text descriptions are consistent. When conflicts are found, it is difficult to ensure the quality of the final output data.

[0103] To address these issues, this design accurately identifies conflicting data by pre-defining cross-modal consistency thresholds and combining them with the scoring of a modal quality assessor. A dual-channel generator is used to generate multiple possible evolutionary data, which are then iteratively evaluated by the modal quality assessor until the conflict is resolved, effectively addressing the issue of conflict resolution. Specific features include:

[0104] Introducing a conflict identification and resolution mechanism to output the optimal generated multimodal data (i.e., closest to the original multimodal data);

[0105] Methods for introducing conflict identification and resolution mechanisms include:

[0106] A threshold for cross-modal consistency is predefined (e.g., dynamically adjusted based on archival metadata or historical data). If the cross-modal consistency value output by the modality quality evaluator is less than the threshold, it is marked as a conflict.

[0107] For conflicting original multimodal data and generated multimodal data, a dual-channel generator is used to generate multiple possible generated multimodal data, and then a modality quality evaluator is used to re-evaluate the cross-modal consistency until the conflict is resolved.

[0108] The output is enhanced multimodal data (image + text), including original multimodal data, generated multimodal data and co-genesis score.

[0109] The introduction of a modality quality assessor evaluates the compatibility of raw and evolved data using multi-dimensional metrics (such as image quality, text quality, and cross-modal consistency). This, combined with conflict identification and resolution mechanisms, forms a closed-loop optimization loop. This mechanism ensures the reliability and robustness of output data, making it particularly suitable for processing ambiguous, damaged, or non-standard data commonly found in archives.

[0110] Archival images may have problems such as low resolution, blur, and unclear seal areas. Archival texts may have problems such as semantic incoherence and chaotic layout. It is difficult to accurately measure the semantic consistency between images and texts for multimodal data. In view of this, this design combines resolution detection, multi-scale fuzziness analysis, SSIM structural integrity, seal detection confidence and other indicators to calculate image quality scores through a weighted formula to solve the diversity problem of image quality assessment. The semantic coherence analysis of DistilBERT and the layout structure evaluation of SB-GNN are combined with a weighted formula to calculate the text quality score to solve the complexity problem of text quality assessment. Using the similarity between OCR and text titles and the image-text matching degree of the CLIP model, a cross-modal consistency score is calculated through a weighted formula to solve the accuracy problem of consistency assessment.

[0111] Perform preliminary modal feature extraction on the enhanced multimodal data to obtain the quality score triple (s_img, s_txt, s_consist);

[0112] Although lightweight models (such as MobileNetV3 and DistilBERT) are used to extract features, the computational overhead of feature extraction can still be high for large-scale archival data. This is especially true in scenarios with high real-time requirements. The efficiency of feature extraction directly affects the latency of subsequent quality assessment and routing decisions, potentially degrading overall system performance. Therefore, adaptive improvements are being made to the lightweight model. Specific improvements include:

[0113] Step Q1: Perform resolution detection on the image data (images inpainted using Better-DeblurGAN-v3) in the generated multimodal data. Select an appropriate feature extraction model based on the resolution. For example, for low-resolution images, use a lighter-weight model (such as MobileNetV3-Small, dynamically selecting between Small and Large versions), while for high-resolution or complex images, use a more powerful model (such as EfficientNet-B0). Features include layout, seal position, signature area confidence, and preliminary OCR recognition results. Extract the visual feature vector f_img to improve feature extraction accuracy.

[0114] For the text data in the generated multimodal data (the standard version text after VS-ADSP and SB-GNN parsing), rule extraction is performed to extract key fields (such as dates and names) and use them as auxiliary inputs for DistilBERT to reduce dependence on DistilBERT and thus improve efficiency. Combining the rule extraction results with DistilBERT, the features include keyword frequency (such as "confidentiality" and "contract"), named entities (such as names and dates), and paragraph topic distribution, and extracting the semantic feature vector f_txt; to improve the structured degree of feature extraction.

[0115] Inaccurate quality assessment may lead to incorrect routing decisions, such as misjudging low-quality modality data as high-quality, thereby selecting the wrong processing path.

[0116] Step Q2: Calculate the OCR recognition accuracy based on the edit distance and semantic rationality. Introduce multi-scale analysis to calculate the blur and SSIM values ​​in the Laplacian variance and SSIM structure integrity. Detect the degree of blur / tilt through the calculation formula to improve the robustness to complex background images. Use YOLOv8 to detect the seal area. Take the highest confidence score as the seal detection confidence. The larger the value, the clearer the seal and the position complies with the document specifications. The output range is [0,1]. Output the image quality score s_img through the calculation formula. The calculation formula is , is the regularization coefficient, is the variance of each indicator or other regularization term;

[0117] When calculating OCR accuracy based on edit distance and semantic plausibility, if true, correct text exists, it can be used directly. If true labels are unavailable, in addition to using BERT to calculate semantic perplexity (PPL), a pre-trained semantic similarity model (such as SimCSE) can be considered to further improve the accuracy of semantic plausibility assessment. OCR accuracy is mapped to the [0, 1] range.

[0118] Multi-scale analysis is introduced in the Laplacian variance to more comprehensively evaluate the clarity of the image. The closer the Blur value is to 1, the clearer the image (high variance), and it automatically falls into the [0, 1] interval. SSIM structural integrity is used to measure the layout structure similarity between an image and a standard template. Multi-scale analysis is introduced to better capture the details and structural information of the image, and the SSIM value is linearly mapped to [0, 1].

[0119] The semantic coherence score is calculated based on DistilBERT's NextSentencePrediction, and SB-GNN is used to analyze the document layout and evaluate the structural completeness (such as the missing rate of required fields). The text quality score s_txt is output through the calculation formula; the calculation formula is ;

[0120] For example, scenario 1: Blurry scan of medical report

[0121] Input: Blurry X-ray report (OCR recognition error, seal is clearer).

[0122] Indicator calculation:

[0123] OCR = 0.65 (“pneumonia” was misidentified as “lung award”);

[0124] Blur=0.3, SSIM=0.4 (format distortion);

[0125] =0.9;

[0126] Weight selection: Medical file preset weights (0.4, 0.2, 0.4);

[0127] Comprehensive score: s_img=0.686;

[0128] Scenario 2: High-definition engineering drawings

[0129] Input: Clear scanned CAD drawings (no stamps, accurate OCR).

[0130] Indicator calculation:

[0131] OCR=0.95;

[0132] Blur=0.9, SSIM=0.85 (fits the template);

[0133] =0.0;

[0134] Weight selection: Engineering drawing preset weights (0.1, 0.6, 0.3);

[0135] Comprehensive score: s_img==0.626.

[0136] Step Q3: Calculate the similarity (cosine similarity) between the image OCR result and the electronic text title, and use the CLIP model to calculate the image-text matching degree as a supplement to the consistency score. Output the cross-modal consistency score s_consist, the calculation formula is: s_consist= Similarity+ CLIP image and text matching degree;

[0137] in, 、 、 、 、 、 、 and is a weight coefficient that matches the preset weight based on archive metadata (such as file name suffix, upload tag); the weight can also be dynamically adjusted based on gradient descent or reinforcement learning.

[0138] Step Q4: Output the quality score triplet (s_img, s_txt, s_consist).

[0139] The PPO algorithm is used for dynamic routing decisions to improve training stability. The input is set to a quality score triplet (s_img, s_txt, s_consist), and a closed-loop optimization of the quality assessment is introduced. This includes: if the quality score of the modal data falls below a preset threshold, feedback is sent to the dual-channel generator, triggering data regeneration. Modal feature extraction is then performed again based on the regenerated modal data, and the quality score triplet is updated. After the closed-loop optimization of the quality assessment is completed, the final quality score triplet (s_img, s_txt, s_consist) is obtained.

[0140] Through closed-loop optimization of quality assessment, the system can regenerate and extract features from low-quality data, continuously update the quality score triples, and ensure that the output comprehensive features have high confidence and consistency.

[0141] If the dynamic modality routing scheme detects that the quality of a modality data is low (such as s_img < 0.4 or s_txt < 0.4), it will feedback to the original scheme to trigger further image restoration (such as calling Better-DeblurGAN-v3) or text generation (such as calling Cross-GAN and CVAE).

[0142] Use the final quality score triplet to import into the PPO algorithm and output the final decision action;

[0143] Execute path rules according to decision actions and output comprehensive features.

[0144] In multimodal data processing, the quality of different modalities varies significantly. How can we dynamically select the optimal processing path (e.g., pure image, pure text, or multimodal fusion) based on data quality? To address this issue, this design uses modal feature extraction to extract a score triplet of image quality, text quality, and cross-modal consistency, providing a basis for subsequent decision-making. Using the PPO algorithm from reinforcement learning, the processing path is dynamically selected based on the quality score triplet, solving the path selection optimization problem. A closed-loop feedback mechanism for quality assessment triggers regeneration when data quality falls below a threshold, iteratively optimizing the feature extraction process to ensure high final feature quality.

[0145] Methods for using the PPO algorithm for dynamic routing decisions include:

[0146] Define the state space S = [s_img, s_txt, s_consist, CPU load, remaining power, network bandwidth, storage capacity]. Use the LSTM model to predict the system resource status (such as CPU load and network bandwidth) in the future. Define the action space A = {pure image path, pure text path, multimodal fusion path}.

[0147] The quality score of the initial output (such as SSIM, NER accuracy, CLIP image-text matching) is used as part of the reinforcement learning reward function to define the reward function Use initial quality scores as initial rewards for reinforcement learning; accelerate training convergence.

[0148] For edge devices and high-performance servers, we dynamically adjust hyperparameters, introduce a long-term reward mechanism, and use discounted returns to calculate long-term performance. For example, for edge devices: =0.7, =0.5, =0.5, =0.5; High performance server: =1, =0.3, =0.2, =0.2.

[0149] Design routing rules as soft constraints and select paths based on probability distribution, including:

[0150] If s_img>0.9 and s_consist>0.8, the pure image path is selected with a probability of 90% and the multimodal fusion path is selected with a probability of 10%;

[0151] If s_txt>0.9 and s_consist>0.8, the plain text path is selected with a probability of 90% and the multimodal fusion path is selected with a probability of 10%;

[0152] If s_img < 0.4, the probability of switching to the text path is 80%, and the probability of selecting the multimodal fusion path is 20%;

[0153] If s_txt < 0.4, the probability of switching to the image path is 80%, and the probability of selecting the multimodal fusion path is 20%.

[0154] If 0.4≤s_img, s_txt≤0.9 or s_consist<0.6, the multimodal fusion path is selected with a probability of 70% and the single-modal path is selected with a probability of 30%;

[0155] You can use pre-trained rule-based routing strategies (such as threshold judgment) as the initial strategy to accelerate convergence. Record the basis for each routing decision (such as quality score, system resource status, and action probability distribution) and generate a decision log for subsequent analysis and optimization.

[0156] The output is the decision action a∈A.

[0157] The feature extraction and optimization module utilizes the Proximal Policy Optimization (PPO) algorithm to dynamically select a processing path (e.g., an image-only path, a text-only path, or a multimodal fusion path) based on quality score triplets. This dynamic routing mechanism not only improves computational efficiency but also adaptively adjusts policies based on data quality and device resources (e.g., CPU load, network bandwidth), making it particularly suitable for deployment on edge devices and in complex environments.

[0158] Path rule execution methods include:

[0159] For the pure image path, deep CNN (such as ResNet) is used to extract visual features and combine the initial OCR recognition results to generate text features;

[0160] For text modalities, we use deep NLP models (such as BERT) to extract semantic features and combine them with the output of DistilBERT to generate structured features.

[0161] For the multimodal fusion path, a cross-modal SP-Transformer model (such as ViLT) is used to fuse the structured and textual features of the original multimodal data and the generated multimodal data to generate the generated modality embedding vector;

[0162] The cross-modal SP-Transformer model is based on the two-stream Transformer framework, introduces MT-ARNet to process fuzzy features, and adds NSR to intervene in the original multimodal data and generate conflicting data in the multimodal data;

[0163] The text features, structural features and embedding vectors are recorded as comprehensive features;

[0164] The method for obtaining the final file classification result includes:

[0165] Step C1: construct a multimodal feature matrix based on the comprehensive features, wherein the multimodal feature matrix forms a unified feature representation by normalizing and aligning the dimensions of different modal features;

[0166] Step C2: Design a multimodal classifier based on the Attention-Transformer multi-layer perception fusion network. Introduce a modality weight adaptive adjustment module into the Transformer attention mechanism to dynamically adjust the weights of image, text, and structured features according to the modality importance of different file types.

[0167] Step C3: For the archive classification task, a predefined set of archive type labels is defined, for example, administrative archives, technical archives, including but not limited to financial archives, and historical archives. A supervised learning method is used to pre-train a multimodal classifier, where the training data includes the enhanced multimodal data and its corresponding archive type labels. The training objective is to minimize the classification cross entropy loss, while introducing a regularization term to prevent overfitting.

[0168] Step C4: Input the multimodal feature matrix into the trained multimodal classifier and output the classification probability distribution of the archive; select the archive type label with the highest confidence as the final classification result based on the probability distribution;

[0169] Step C5: Design a classification result verification mechanism. For archival data with classification confidence below the preset threshold, trigger the manual review process and feed the manual review results back to the training dataset of the dual-channel generator to form a closed-loop optimization. Use weighted loss functions or incremental learning to clarify the quality control of feedback data and the model update strategy.

[0170] Step C6: Output the final archive classification results, including the archive type label, classification confidence and corresponding multimodal feature matrix, for subsequent archive management and retrieval.

[0171] Classification results come with interpretable reports, such as "Because the title contains 'technical cooperation' and an official seal was detected, it takes priority over keywords in the text." Interactive rule debugging provides real-time adjustment capabilities. Users can drag highlighted areas in the heat map or enter natural language to modify rules and instantly view the adjusted classification results. Rule adjustments and classification results are recorded in blockchain logs, supporting historical traceability. For example, users can query the classification history of a file to see if it has changed due to rule adjustments.

[0172] The classification and verification module uses an Attention-Transformer-based multimodal classifier and introduces an adaptive modal weight adjustment module, which dynamically adjusts the weights of different modal features based on the characteristics of the archive type. This design significantly improves classification accuracy and generalization, making it particularly suitable for processing archival data with large variations in modal importance.

[0173] The classification result verification mechanism forms a closed-loop optimization through manual review feedback and model retraining, which can continuously improve the performance of the classifier and reduce the misclassification rate, especially when processing complex or low-confidence files.

[0174] Example 2

[0175] See also Figure 2 As shown, for the parts not described in detail in this embodiment, please refer to the description of Example 1. A multimodal intelligent AI classification method for file organization is provided, including:

[0176] Step S1: Obtain a scanned copy of a paper file through a scanner or camera, combine it with the electronic document to form original multimodal data, and transmit it to the perception layer through the API interface;

[0177] Step S2: In the perception layer, a dual-channel generator is used to generate multimodal data through adversarial training. A cross-modal consistency loss function and a structured constraint loss are introduced to generate associated image features or pseudo-text descriptions based on a preset differential quality threshold.

[0178] Step S3: Design a modality quality evaluator to score the co-generacy of the original multimodal data and the generated multimodal data based on image quality, text quality, and cross-modal consistency indicators; through a conflict identification and resolution mechanism, mark data below the cross-modal consistency threshold, generate multiple possible multimodal data, re-evaluate them, and output enhanced multimodal data;

[0179] Step S4: Perform modal feature extraction on the enhanced multimodal data to obtain the quality score triple (s_img, s_txt, s_consist); use the PPO algorithm to make dynamic routing decisions, set the state space, action space and reward function, and introduce closed-loop optimization. If the quality score is lower than the threshold, data regeneration is triggered, the quality score triple is updated, and the decision action is output;

[0180] Step S5: Execute path rules according to the decision action, including pure image path, pure text path and multimodal fusion path, use deep CNN, NLP model or cross-modal SP-Transformer model to extract features and output comprehensive features;

[0181] Step S6: Construct a multimodal feature matrix based on the comprehensive features, and form a unified feature representation through normalization and dimension alignment; design a multimodal classifier based on Attention-Transformer, introduce a modal weight adaptive adjustment module, and use supervised learning pre-training;

[0182] Step S7: Input the multimodal feature matrix into the classifier, output the file classification probability distribution, and select the highest confidence label as the classification result; for file data with confidence below the threshold, trigger manual review and feedback optimization;

[0183] Step S8: Output the final archive classification results, including type labels, classification confidence, and multimodal feature matrix.

[0184] Example 3

[0185] This embodiment discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the operation mode of the multimodal intelligent AI classification system for archive organization provided above is realized.

[0186] Since the electronic device introduced in this embodiment is an electronic device used to implement a multimodal intelligent AI classification system for file organization in the embodiment of this application, based on the multimodal intelligent AI classification system for file organization introduced in the embodiment of this application, those skilled in the art can understand the specific implementation of the electronic device of this embodiment and its various variations, so how the electronic device implements the method in the embodiment of this application will not be described in detail here. As long as those skilled in the art implement the electronic device used in the multimodal intelligent AI classification system for file organization in the embodiment of this application, it falls within the scope of protection of this application.

[0187] The above formulas are all dimensionless and numerical calculations. The formula is a formula that is closest to the actual situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0188] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the principles of the present invention are within the scope of protection of the present invention. It should be noted that for users of ordinary skill in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal intelligent AI classification system for archive organization, characterized by: include: Data collection and transmission module: obtains scanned copies of paper archives through scanners or cameras, and combines them with electronic documents to form original multimodal data; Perception layer module: This module receives raw multimodal data and introduces a cross-modal consistency loss function and a structured constraint loss into the deployed dual-channel generator to generate associated image features or pseudo-text descriptions to form evolving multimodal data. Based on the modal quality assessor, the original multimodal data and the evolved multimodal data are scored for mutuality; through the conflict identification and resolution mechanism, the enhanced multimodal data is output; Feature extraction and optimization module: Extracts features based on enhanced multimodal data to obtain quality score triples; uses the PPO algorithm for dynamic routing decisions, introduces closed-loop optimization for quality assessment, optimizes and updates the quality score triples, and outputs decision actions; Execute path rules according to decision actions and output comprehensive features; Path rule execution methods include: For the pure image path, deep CNN is used to extract visual features and combined with the initial OCR recognition results to generate text features; For text modalities, we use a deep NLP model to extract semantic features and combine them with the output of DistilBERT to generate structured features. For the multimodal fusion path, a cross-modal SP-Transformer model is used to fuse the structured and textual features of the original multimodal data and the evolved multimodal data to generate a modality embedding vector; The cross-modal SP-Transformer model is based on the two-stream Transformer framework, introduces MT-ARNet to process fuzzy features, and adds NSR to intervene in conflicting data in the original multimodal data and evolved multimodal data; The text features, structural features and embedding vectors are recorded as comprehensive features; Classification and verification module: Construct a multimodal feature matrix based on comprehensive features, and design a multimodal classifier based on Attention-Transformer. Introduce a modal weight adaptive adjustment module, set the multimodal feature matrix as input, and output the classification result; add a classification result verification mechanism to form a closed-loop optimization; obtain the final archive classification result.

2. The multimodal intelligent AI classification system for file organization according to claim 1, characterized in that: The method for obtaining the enhanced multimodal data includes: Scan paper files are obtained through scanners or cameras, combined with electronic documents to form original multimodal data, which is then transmitted to the perception layer through the API interface; A dual-channel generator is deployed in the perception layer. It learns the cross-modal mapping relationship between images and text through adversarial training. A cross-modal consistency loss function is introduced into the adversarial training, along with a structured constraint loss. Differentiated quality thresholds are pre-set for different modal file types. The generator generates associated image features based on text, or pseudo-text descriptions based on images, to acquire evolving multimodal data. Design a modality quality evaluator to score the co-generacy of original multimodal data and evolved multimodal data. The multi-dimensional evaluation indicators of the modality quality evaluator include image quality or text quality and cross-modal consistency. Introducing conflict identification and resolution mechanisms to output optimal evolving multimodal data; Output enhanced multimodal data, including original multimodal data, evolved multimodal data, and symbiosis scores.

3. The multimodal intelligent AI classification system for file organization according to claim 2, characterized in that: The dual-channel generator comprises: For image modality data, we deploy Better-DeblurGAN-v3 to automatically inpaint images and extract local key regions. This includes adding multi-scale convolution kernels to the DeblurGAN-v3 encoder to capture image interference at different scales, and adding an RPN region proposal network to automatically locate and enhance key regions. For text data, the VS-ADSP document structure parser's dynamic segmentation strategy is used to identify non-standard layouts, and the SB-GNN graph neural network is combined to reconstruct the document's logical topology and convert non-standard layout documents into standard versions. The VS-ADSP document structure parser includes the introduction of document segmentation evolution based on visual-semantic combination in ADSP, and the SB-GNN graph neural network includes the addition of visual and semantic features to the node features of the GNN. For isolated modal data, the conditional variational autoencoder (CVAE) is introduced into the cross-modal generative adversarial network (Cross-GAN), and variational constraints are added to the generator of Cross-GAN. Specifically, for extremely isolated modal data, visual question answering enhancement (VQA-E) is introduced, and the semantic description of the image is generated through the pre-trained VQA-E model.

4. The multimodal intelligent AI classification system for file organization according to claim 3, characterized in that: The method of introducing the conflict identification and resolution mechanism includes: A threshold for cross-modal consistency is predefined. If the cross-modal consistency value output by the modality quality evaluator is less than the threshold, it is marked as a conflict. For conflicting original multimodal data and evolved multimodal data, a dual-channel generator is used to generate multiple possible evolved multimodal data, and then the modal quality evaluator is used to re-evaluate the cross-modal consistency until the conflict is resolved.

5. The multimodal intelligent AI classification system for file organization according to claim 4, characterized in that: The method for obtaining the comprehensive features includes: Perform preliminary modal feature extraction on the enhanced multimodal data to obtain the quality score triple (s_img, s_txt, s_consist); The PPO algorithm is used for dynamic routing decisions. The input is set as a quality score triplet (s_img, s_txt, s_consist). A closed-loop optimization of quality assessment is introduced. If the quality score of the modal data is lower than the preset threshold, it is fed back to the dual-channel generator to trigger data regeneration. Modal feature extraction is performed again based on the regenerated modal data, and the quality score triplet is updated. After the closed-loop optimization of quality assessment is completed, the final quality score triplet (s_img, s_txt, s_consist) is obtained. Use the final quality score triplet to import into the PPO algorithm and output the final decision action; Execute path rules according to decision actions and output comprehensive features.

6. The multimodal intelligent AI classification system for file organization according to claim 5, characterized in that: The method of performing preliminary modal feature extraction on the enhanced multimodal data to obtain a quality score triplet (s_img, s_txt, s_consist) includes: Step Q1: Perform resolution detection on the image data in the evolving multimodal data, select an appropriate feature extraction model based on the resolution, and extract the visual feature vector f_img based on the features including layout, seal position, signature area confidence, and initial OCR recognition results. Perform rule extraction on text data in evolving multimodal data, extract key fields, and use them as auxiliary input for DistilBERT. Features include keyword frequency, named entities, and paragraph topic distribution, and extract the semantic feature vector f_txt. Step Q2: Calculate the OCR recognition accuracy based on the edit distance and semantic rationality, introduce multi-scale analysis to calculate the blur and SSIM values ​​in the Laplacian variance and SSIM structure integrity, use YOLOv8 to detect the seal area, take the highest confidence score as the seal detection confidence, and output the image quality score s_img through the calculation formula; the calculation formula is , is the regularization coefficient, is the variance of each indicator or other regularization term; Calculate semantic coherence scores based on DistilBERT's NextSentencePrediction and use SB-GNN to analyze document layout and evaluate structural completeness; Output the text quality score s_txt through the calculation formula; the calculation formula is ; Step Q3: Calculate the similarity between the image OCR result and the electronic text title, use the CLIP model to calculate the image-text matching degree between the image and the text, and output the cross-modal consistency score s_consist. The calculation formula is: ×Similarity+ ×CLIP image and text matching degree; in, 、 、 、 、 、 、 and is the weight coefficient, which matches the preset weight based on the archive metadata; Step Q4: Output the quality score triplet (s_img, s_txt, s_consist).

7. The multimodal intelligent AI classification system for file organization according to claim 6, characterized in that: The method for making dynamic routing decisions using the PPO algorithm includes: Define the state space S = [s_img, s_txt, s_consist, CPU load, remaining power, network bandwidth, storage capacity], and define the action space A = {pure image path, pure text path, multimodal fusion path}; Define the reward function , using the initial quality score as the initial reward for reinforcement learning; Dynamically adjust hyperparameters for edge devices and high-performance servers, introduce a long-term reward mechanism, and use discounted returns to calculate long-term performance; Design routing rules as soft constraints, and select paths based on probability distribution, including: If s_img>0.9 and s_consist>0.8, the pure image path is selected with a probability of 90% and the multimodal fusion path is selected with a probability of 10%; If s_txt>0.9 and s_consist>0.8, the plain text path is selected with a probability of 90% and the multimodal fusion path is selected with a probability of 10%; If s_img < 0.4, the probability of switching to the text path is 80%, and the probability of selecting the multimodal fusion path is 20%; If s_txt < 0.4, the probability of switching to the image path is 80%, and the probability of selecting the multimodal fusion path is 20%; If 0.4≤s_img, s_txt≤0.9 or s_consist<0.6, the multimodal fusion path is selected with a probability of 70% and the single-modal path is selected with a probability of 30%; The output is the decision action a∈A; in, 、 、 and is the weight coefficient.

8. The multimodal intelligent AI classification system for file organization according to claim 7, characterized in that: The method for obtaining the final file classification result includes: Step C1: construct a multimodal feature matrix based on the comprehensive features, wherein the multimodal feature matrix forms a unified feature representation by normalizing and aligning the dimensions of different modal features; Step C2: Design a multimodal classifier based on the Attention-Transformer multi-layer perception fusion network. Introduce a modality weight adaptive adjustment module into the Transformer attention mechanism to dynamically adjust the weights of image, text, and structured features according to the modality importance of different file types. Step C3: For the archive classification task, a predefined archive type label set is used, and a supervised learning method is used to pre-train the multimodal classifier. The training objective is to minimize the classification cross entropy loss, and a regularization term is introduced at the same time. Step C4: Input the multimodal feature matrix into the trained multimodal classifier and output the classification probability distribution of the archive; select the archive type label with the highest confidence as the final classification result based on the probability distribution; Step C5: Design a classification result verification mechanism. For archival data whose classification confidence is lower than the preset threshold, trigger the manual review process and feed the manual review results back to the dual-channel generator, forming a closed-loop optimization. Step C6: Output the final archive classification results, including the archive type label, classification confidence and corresponding multimodal feature matrix, for subsequent archive management and retrieval.

9. A multimodal intelligent AI classification method for file organization, applied to the multimodal intelligent AI classification system for file organization according to any one of claims 1 to 8, characterized in that: The multimodal intelligent AI classification method for file organization includes: Step S1: Obtain a scanned copy of a paper file through a scanner or camera, combine it with the electronic document to form original multimodal data, and transmit it to the perception layer through the API interface; Step S2: In the perception layer, a dual-channel generator is used to generate multimodal data through adversarial training. A cross-modal consistency loss function and a structured constraint loss are introduced to generate associated image features or pseudo-text descriptions based on a preset differential quality threshold. Step S3: Design a modality quality evaluator to score the co-generacy of the original multimodal data and the generated multimodal data based on image quality, text quality, and cross-modal consistency indicators; through a conflict identification and resolution mechanism, mark data below the cross-modal consistency threshold, generate multiple possible multimodal data, re-evaluate them, and output enhanced multimodal data; Step S4: Perform modal feature extraction on the enhanced multimodal data to obtain the quality score triple (s_img, s_txt, s_consist); use the PPO algorithm to make dynamic routing decisions, set the state space, action space and reward function, and introduce closed-loop optimization. If the quality score is lower than the threshold, data regeneration is triggered, the quality score triple is updated, and the decision action is output; Step S5: Execute path rules according to the decision action, including pure image path, pure text path and multimodal fusion path, use deep CNN, NLP model or cross-modal SP-Transformer model to extract features and output comprehensive features; Step S6: Construct a multimodal feature matrix based on the comprehensive features, and form a unified feature representation through normalization and dimension alignment; design a multimodal classifier based on Attention-Transformer, introduce a modal weight adaptive adjustment module, and use supervised learning pre-training; Step S7: Input the multimodal feature matrix into the classifier, output the file classification probability distribution, and select the highest confidence label as the classification result; for file data with confidence below the threshold, trigger manual review and feedback optimization; Step S8: Output the final archive classification results, including type labels, classification confidence, and multimodal feature matrix.

Images