A multi-modal semi-supervised learning method and system for sparse-labeled visual rich documents

Abstract

The application discloses a kind of multi-modal semi-supervised learning methods and systems for sparse annotation visual rich document, belong to artificial intelligence, document intelligent understanding and semi-supervised learning technical field, including cross-modal alignment pre-training, context-aware semi-supervised learning, class discovery and calibration and open world balanced identification, visual representation of visual rich document image is extracted using visual encoder, text representation of document is extracted using text encoder;Visual guided disambiguation text representation is generated by prior knowledge guide module, before cross-modal interaction, the contrast loss of two directions of text to vision, vision to text is calculated simultaneously.The application constructs the complete technical chain of document understanding scheme covering full supervision alignment→closed space semi-supervised→class semi-supervised→open world discovery, four steps progressive extension, scene adaptability is strong, different annotation conditions, different open degree of document processing demand can be flexibly combined and adapted.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, intelligent document understanding, and semi-supervised learning, and more specifically, to a multimodal semi-supervised learning method and system for sparsely labeled visual rich documents. Background Technology

[0002] Visually rich documents include documents such as invoices, receipts, insurance policies, medical bills, and contracts that contain multimodal information such as text, layout, and visual styles. Document Key Information Extraction (DKIE) is the core task of automatically extracting key entity information from these documents and is the foundation for achieving automated document processing.

[0003] Existing technologies face four progressively different technical challenges in the field of intelligent document understanding: First, multimodal feature alignment is insufficient in fully supervised scenarios. Existing pre-trained methods such as the LayoutLM series ignore the semantic ambiguity of similar texts when performing visual-text cross-modal fusion, and fail to address the misalignment of representation spaces before cross-modal interaction, resulting in insufficient quality of multimodal joint representations and limiting the performance ceiling of downstream tasks.

[0004] Second, the annotation cost of semi-supervised learning in a closed label space is high. In real-world business scenarios, collecting enough labeled samples for each entity category is extremely costly. Existing pseudo-label semi-supervised learning methods do not consider the contextual dependencies of the DKIE task. The semantics of the same text may be completely different under different document layouts. At the same time, they cannot cope with the statistical characteristics of DKIE data, which have high intra-class variance and low inter-class variance, resulting in large pseudo-label noise and limited semi-supervised effects.

[0005] Third, the ability to handle unknown categories in open label spaces is lacking. In real-world scenarios, unlabeled documents inevitably contain new categories that have never appeared in the label set, and unknown categories are usually a minority of categories that have fine-grained similarities with known categories. Existing semi-supervised methods suffer severe performance degradation under this type of missing-class semi-supervised setting and cannot effectively identify unknown categories.

[0006] Fourth, the competition between old and new categories in open-world generalized category discovery. In generalized category discovery tasks that utilize labeled old category information while simultaneously identifying old and new categories in unlabeled data, parameterized classifiers exhibit systematic overconfidence in the new category. This leads to a trade-off between the accuracy of the old and new categories during training, and existing methods cannot simultaneously guarantee a balanced recognition accuracy between the old and new categories.

[0007] In view of this, the present invention is proposed to solve the above-mentioned technical problems. Summary of the Invention

[0008] The purpose of this invention is to provide a multimodal semi-supervised learning method and system for sparsely labeled visual rich documents, in order to solve the technical problems of existing visual rich document intelligent understanding technology, such as insufficient semantic alignment of multimodal features in fully supervised scenarios, large pseudo-label noise in semi-supervised scenarios with closed label spaces, failure of unknown category recognition in semi-supervised scenarios with missing classes, imbalance of recognition accuracy between new and old categories in open-world scenarios, and lack of a systematic solution covering the entire application chain.

[0009] To achieve the above objectives, the present invention provides the following technical solution: The first objective of this invention is to provide a multimodal semi-supervised learning method for sparsely annotated visual rich documents, comprising the following steps: S1. Cross-modal alignment pre-training: Visual encoders are used to extract visual representations of rich document images, and text encoders are used to extract text representations of documents. A visually guided deambiguous text representation is generated through a prior knowledge-guided module. Before cross-modal interaction, the contrast loss in both the text-to-visual and visual-to-text directions is calculated simultaneously. The cross-alignment loss is obtained by weighted fusion of mixing coefficients to achieve bidirectional modal semantic space alignment. The visual-text similarity calculation results are reused, and the similarity is hierarchically sorted along the visual-to-text and text-to-visual dimensions, respectively. High-similarity negative samples in each level are selected to form an effective negative sample subset to drive joint representation contrastive learning. S2. Context-aware semi-supervised learning: In scenarios with a closed label space and limited labeled samples, a first similarity matrix between labeled data prediction and pseudo-labels, and a second similarity matrix between labeled data prediction and strongly enhanced prediction of unlabeled data are constructed. The two similarity matrices are fused with statistically distributed parameterized random mixing coefficients, and guided pseudo-labels are generated after graph Laplacian normalization. Intra-class feature alignment is achieved through momentum queues grouped by category and exponential decay weights, and inter-class feature separation is achieved through nonlinear projection and adaptive contrastive loss. S3. Missing Class Detection and Correction: In open-label scenarios where unlabeled data contains unknown categories, global category features and local batch features are fused to calculate adaptive debiased prototypes to correct batch sampling bias. A Gaussian mixture model is used to model the distance distribution from unlabeled samples to each prototype, estimating the category frequency distribution of the unlabeled data. Based on the learning gap between known and unknown classes, a differential confidence threshold is dynamically set, and the gradient contribution of minority class pseudo-labels is strengthened through posterior probability weights. Unlabeled features are mapped to binary hash codes, and erroneous connections between fine-grained categories are severed using an adaptive Hamming distance threshold. The largest connected component is retained in the category subgraph to generate a high-quality affinity graph, driving multi-granularity graph contrastive learning. S4. Open World Equilibrium Recognition: In generalized category discovery scenarios that simultaneously include old and new categories, an independent debiasing head is set up to generate auxiliary predictions; a feature momentum memory M and a prediction probability momentum memory P are maintained; nearest neighbor samples are retrieved based on the feature representation of the debiasing head, and the mean of the nearest neighbor predictions is used as the calibration target; semantic calibration of the main classifier is achieved through KL divergence constraints; the prediction results of the main classifier and the debiasing classifier are fused with statistically distributed parameterized mixing coefficients; a self-similarity matrix is ​​constructed based on the fused predictions and a contrastive loss is applied; cross-entropy alignment constraints are applied to predictions in the debiasing classifier with confidence levels higher than a preset threshold to balance the recognition accuracy of the old and new categories.

[0010] Furthermore, the processing procedure of the prior knowledge guidance module is as follows: The initial visual features are output by the visual encoder, and then two pooled features are generated by global average pooling and max pooling. After passing through a fully connected layer and activation function, the features are multiplied element-wise with the initial visual features to obtain a weighted visual representation. Calculate the difference attention weights between visual regions and perform weighted optimization on the weighted visual representation; The similarity matrix between the optimized visual representation and the text representation is calculated. Attention weights are generated using temperature-controlled Softmax and applied to the text representation to obtain a deambiguous text representation.

[0011] Furthermore, the formula for calculating the cross-alignment loss is: =α +(1-α) ; in, For contrast loss in the text-to-visual direction; α represents the contrast loss from visual to text direction; α is the mixing coefficient, with an optimal value of 0.4.

[0012] Furthermore, the two-dimensional hierarchical sorting and filtering process is as follows: along the two dimensions of visual to text and text to visual, all sample pairs are divided into multiple similarity interval layers according to their similarity values. Within each layer, samples are filtered in descending order, and finally, the Top-L' high similarity negative samples are obtained. The optimal value of L' is 32. The total loss for cross-modal alignment pre-training is: =ψ + +ω ; in, For masked language modeling loss, For cross-alignment loss, The contrastive learning loss is represented jointly, with ψ and ω as balance coefficients.

[0013] Furthermore, the random mixing coefficients follow a Gamma distribution, i.e., α ~ Gamma(β, β), where the optimal value of β is 0.6; The total loss of context-aware semi-supervised learning is: = + +λ ; in, For supervised loss of labeled samples, For pseudo-label loss of unlabeled samples, The loss is a bi-branch adaptive alignment loss, where λ is the balance coefficient, with an optimal value of 0.2.

[0014] Furthermore, the adaptive debiasing prototype is calculated as follows: ; in, For the j-th local batch feature of the k-th class, Local batch features and global category features Cosine similarity weight; The formula for calculating the adaptive Hamming distance threshold is: ; Where K is the hash code length and s is the average cosine similarity of samples within the batch.

[0015] Furthermore, the total loss from missing class detection and calibration is: ; in, For supervised loss of labeled samples, For unlabeled pseudo-label loss with minority class weights, For multi-granularity graph contrast learning loss, This is the balance coefficient.

[0016] Furthermore, the update methods for the feature momentum memory M and the prediction probability momentum memory P are as follows: ; ; in, For batch characteristics, For batch prediction probability, , The momentum coefficient has an optimal value of 0.99. The mixing coefficients follow a Gamma distribution, i.e., α ~ Gamma(β, β), where the optimal value of β is 0.5.

[0017] Furthermore, the total loss for open-world equilibrium identification is: ; in, For semantic calibration loss, To eliminate biased contrast loss, To align constraint loss, , The optimal value for the confidence threshold η of the alignment constraint loss is 0.95, representing the balance coefficient.

[0018] The second objective of this invention is to provide a multimodal semi-supervised learning system for sparsely labeled visual rich documents, employing any of the aforementioned multimodal semi-supervised learning methods for sparsely labeled visual rich documents, including: The multimodal feature extraction module is used to extract visual features from visual rich documents through a visual encoder and extract text features from documents through a text encoder. The cross-modal alignment pre-training module is used to perform cross-modal alignment pre-training and output the aligned multimodal joint representation; The context-aware semi-supervised learning module is used to perform context-aware semi-supervised learning, generate high-quality pseudo-labels in a closed label space and complete semi-supervised training. The missing class detection and calibration module is used to perform missing class detection and calibration, realizing the discovery of unknown categories and feature calibration in the open label space; The open-world equilibrium recognition module is used to perform open-world equilibrium recognition, balancing the recognition accuracy of new and old categories in generalized category discovery scenarios; The joint optimization engine is used to perform weighted combination and end-to-end optimization scheduling of the loss functions of the multimodal feature extraction module, cross-modal alignment pre-training module, context-aware semi-supervised learning module, missing class detection and calibration module, and open-world equilibrium recognition module.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention constructs a document understanding scheme that covers a complete technology chain, including fully supervised alignment, semi-supervised closed space, semi-supervised missing class discovery, and open-world discovery. The four steps are progressively extended and highly adaptable to different scenarios, and can be flexibly combined to meet document processing needs with different annotation conditions and different degrees of openness.

[0020] 2. It shares underlying components such as Gamma distribution parameterized mixing strategy, contrastive learning framework, and adaptive threshold mechanism, with unified technical logic, low engineering implementation complexity, and forms a technical evolution path from heuristic to statistical drive.

[0021] 3. Technical performance improvements at each stage: The cross-modal alignment stage achieves bidirectional dynamic semantic alignment before cross-modal interaction, effectively eliminating textual semantic ambiguity. On the DocVQA benchmark, the base model's ANLS index is improved by 5.61% compared to the suboptimal method. In the semi-supervised stage, the contextual dependencies of DKIE are incorporated into the pseudo-label generation, adapting to the statistical characteristics of document data. Under the 5% annotation setting of the FUNSD dataset, the F1 score is improved by 2.11%-2.53% compared with the suboptimal method. The missing-class semi-supervised stage systematically solves the missing-class semi-supervised DKIE problem, achieving a 14.58% improvement over the suboptimal method on the FUNSD full-class F1 score; The open-world phase specifically addresses the competition between new and old classes in generalized class discovery, achieving an 8.3% improvement in overall accuracy across all classes on the CUB fine-grained dataset compared to the suboptimal method, thus simultaneously improving the accuracy of both new and old classes.

[0022] 4. This invention can be widely applied to scenarios such as automated processing of financial bills, extraction of medical record information, extraction of key clauses of legal contracts, and verification of government form information, significantly reducing annotation costs and improving the intelligence level and scenario adaptability of document processing. Attached Figure Description

[0023] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention, but do not constitute an undue limitation of the invention. Obviously, the drawings described below are merely some embodiments, and those skilled in the art can obtain other drawings based on these drawings without creative effort. In the drawings: Figure 1 This is an architecture diagram of a multimodal semi-supervised learning system for sparsely annotated visual rich documents provided in an embodiment of this application. Figure 2 A DCMAI framework diagram provided for embodiments of this application; Figure 3 A PLBR framework diagram provided for embodiments of this application; Figure 4 SRE framework diagram provided for embodiments of this application; Figure 5 A DeGCD framework diagram provided for embodiments of this application; Figure 6 A diagram showing the relationship between the shared core components of DCMAI, PLBR, SRE, and DeGCD provided for embodiments of this application. Detailed Implementation

[0024] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0025] Example 1

[0026] See Figures 1 to 6 As shown, a multimodal semi-supervised learning method for sparsely labeled visual rich documents includes the following steps: S1. Cross-Modal Alignment Pre-training (DCMAI): Visual encoders are used to extract visual representations of rich document images, and text encoders are used to extract text representations of documents. A prior knowledge-guided module generates visually guided deambiguous text representations. Before cross-modal interaction, contrastive losses in both text-to-visual and visual-to-text directions are calculated simultaneously. Cross-alignment loss is obtained through weighted fusion of mixing coefficients to achieve bidirectional modal semantic space alignment. The visual-text similarity calculation results are reused, and similarity is hierarchically sorted along both visual-to-text and text-to-visual dimensions. High-similarity negative samples within each level are selected to form an effective negative sample subset, driving joint representation contrastive learning. S1 is used in scenarios with full supervision and sufficient labeled data, outputting high-quality aligned multimodal features to provide a feature foundation for subsequent semi-supervised stages. It includes: (1) Multimodal feature extraction: Visual encoders (such as Vision Transformer, ViT) are used to extract the visual representation of the rich document image, and text encoders (such as BERT) are used to extract the text representation of the document, while incorporating the layout and position information of the document.

[0027] (2) Prior Knowledge Guidance Deambiguity: Through the Prior Knowledge Guidance (PKG) module, visual features are first subjected to dual pooling (global average pooling + max pooling) and fully connected transformation to generate visual attention weights, and the initial visual features are weighted and optimized; then the visual-text similarity matrix is ​​calculated, and text attention weights are generated through temperature-controlled Softmax to weight and correct the text representation, thereby eliminating semantic ambiguity of similar texts.

[0028] (3) Bidirectional cross-alignment: Before the interaction of the cross-modal encoder, the contrast loss in both the text-to-visual and visual-to-text directions is calculated simultaneously. The cross-alignment loss (CA Loss) is obtained by weighted fusion of the mixing coefficients determined by the Gamma distribution, so that the visual and text representations can complete fine-grained semantic space alignment before the interaction.

[0029] (4) Hierarchical interactive sampling and joint comparison: The visual-text similarity matrix obtained in the cross-alignment loss calculation process is reused and hierarchically sorted along the two dimensions of visual→text and text→visual respectively. High similarity negative samples are selected to drive joint representation comparison learning with extremely low additional computational cost and improve the discriminativeness of multimodal joint representation.

[0030] S1 is pre-trained on a large-scale unlabeled invoice corpus to obtain a multimodal feature extractor with visual-text depth alignment.

[0031] (1) Multimodal feature extraction

[0032] Input an invoice image and extract initial visual features using the ViT visual encoder. ∈ Where d is the feature dimension, The number of visual tokens; the invoice text content is recognized by the OCR engine, and the initial text features are extracted by the BERT text encoder. ∈ ,in The number of text tokens is used; at the same time, the coordinate information of the text box is encoded as a position embedding and integrated into the text features.

[0033] (2) Prior Knowledge Guidance Module (PKG)

[0034] First, the initial visual features are subjected to dual-channel pooling: =AvgPool( ); =MaxPool( ); The two pooled features are concatenated and then passed through a fully connected layer with a sigmoid activation function to generate visual attention weights. These weights are then multiplied element-wise with the initial visual features to obtain a preliminary weighted visual representation. ; in It is the Sigmoid activation function. For element-wise multiplication, [;] represents feature concatenation.

[0035] Furthermore, differential attention weights between image regions are calculated using two learnable linear transformation functions R and Z: for Linear transformations are performed to obtain the feature space difference matrix, and softmax normalization is used to obtain the inter-region attention weight matrix A∈ Combine the attention weight matrix with Multiplying them yields the final optimized visual representation. ; =A·X; Computational similarity matrix between visual and text representations: ; in The similarity matrix is ​​L2 norm. A temperature-controlled Softmax is applied column-by-column to the similarity matrix to obtain the visual attention weights for each text token. Weighted generation of visually guided deambiguous text representation: = · ; By using visual semantic guidance, the semantic ambiguity of similar numerical texts such as "1970" and "80202" is eliminated, and their semantic attributes in the invoice layout are clarified.

[0036] (3) Bidirectional Cross Alignment Loss (CA Loss)

[0037] Before cross-modal encoder interaction, the InfoNCE contrast loss in both directions is calculated separately: Text → Visual Contrast Loss For each text token, match the corresponding positive visual sample and calculate the InfoNCE loss; Visual → Text Contrast Loss For each visual token, match the corresponding positive text sample and calculate the InfoNCE loss.

[0038] The cross-alignment loss is obtained by weighted fusion using the mixing coefficient α: =α +(1-α) ; In this embodiment, the optimal value of α is 0.4. The bidirectional contrast loss enables fine-grained semantic space alignment between visual and text representations before they enter the cross-modal encoder.

[0039] (4) Hierarchical Interactive Sampling (HISS)

[0040] Reusing the similarity matrix T, the similarity is sorted in descending order along both the visual → text and text → visual dimensions, and the top-L' high-similarity negative samples are selected, with L' being optimally 32. The joint representation contrastive learning loss is then calculated based on the selected negative sample set. It requires almost zero additional computational cost to obtain an efficient negative sample subset, thereby improving the discriminative power of the joint representation.

[0041] (5) Total pre-training loss =ψ + +ω ; in, The loss is modeled using the standard masking language, and ψ and ω are balance coefficients. In this embodiment, ψ is set to 1.0 and ω is set to 0.5.

[0042] After this training phase is completed, the resulting cross-modal feature extractor provides a high-quality feature foundation for the subsequent semi-supervised phase.

[0043] S2. Context-Aware Semi-Supervised Learning (PLBR): In scenarios with a closed label space and limited labeled samples, a first similarity matrix between labeled data prediction and pseudo-labels, and a second similarity matrix between labeled data prediction and strongly enhanced prediction of unlabeled data are constructed. The two similarity matrices are fused with statistically distributed parameterized random mixing coefficients, and guided pseudo-labels are generated after graph Laplacian normalization. Intra-class feature alignment is achieved through momentum queues grouped by category and exponential decay weights, and inter-class feature separation is achieved through nonlinear projection and adaptive contrastive loss. S2 is designed for scenarios with closed label spaces and limited labeled samples. It performs semi-supervised training based on the alignment features output from the first stage. This includes: (1) Similarity matrix bias correction: Two types of similarity matrices are constructed. The first type is the similarity matrix between the predicted results of the labeled data and the pseudo-labels. The second category is the similarity matrix between the prediction results of labeled data and the strongly enhanced predictions of unlabeled data. The two matrices are fused using random mixing coefficients that follow a Gamma distribution. After being normalized by Graph Laplace, guiding pseudo-labels are generated. The pseudo-label bias is corrected by using reliable predictions from the labeled data.

[0044] (2) Two-branch adaptive alignment: Two branches are set up: intra-class alignment and inter-class alignment. The intra-class alignment branch maintains the momentum queue grouped by category, calculates the class-wise similarity between weak enhancement, strong enhancement prediction and queue, and applies exponential decay weight to the sorted similarity to increase the alignment weight of difficult intra-class samples and enhance the compactness of intra-class features; the inter-class alignment branch maps nonlinear representation through MLP projection head and expands the feature distance of samples of different categories with contrast loss with adaptive weight to enhance inter-class separability.

[0045] Semi-supervised fine-tuning was initiated with 5%-10% of the labeled invoice data, using a teacher-student architecture, where the teacher model was updated from the student model via an exponential moving average (EMA).

[0046] (1) Similarity Matrix Deviation Correction Module (SMBR)

[0047] The student model outputs a prediction probability matrix P∈ on the labeled data. Where C is the number of categories, The number of labeled samples; the teacher model outputs a pseudo-label matrix U∈ for unlabeled weakly augmented data. Output a prediction matrix for unlabeled, strongly augmented data. ∈ , This represents the number of unlabeled samples. Construct two similarity matrices: (Similarity matrix between labeled predictions and pseudo-labels); (Similarity matrix between labeled prediction and unlabeled strongly enhanced prediction). The two matrices are fused using random mixing coefficients α that follow a Gamma distribution: , α~Gamma(β,β); In this embodiment, the optimal value of β is 0.6, and the random mixing mechanism can improve the generalization ability of the model.

[0048] Construct a graph Laplacian matrix for the fusion matrix W and normalize it: ; Where D is the degree matrix and L is the Laplacian matrix. The normalized matrix is ​​applied to the initial pseudo-labels to obtain the corrected guiding pseudo-labels. The noise of the pseudo-labels is reduced by utilizing the reliable predictions of the labeled data, thus adapting to the contextual dependencies of the invoice text.

[0049] The normalized system matrix Multiplying the initial pseudo-label matrix U by the matrix multiplication, we achieve pseudo-label propagation and correction based on the graph structure, resulting in the guiding pseudo-label U': U´=U· ; The corrected pseudo-labels integrate the predicted priors of the labeled data with the neighborhood consistency information of the graph structure, which can effectively reduce pseudo-label noise.

[0050] (2) Bi-branch adaptive alignment mechanism (DBAA)

[0051] Intra-class aligned branches: Maintaining momentum queues grouped by category Each queue stores historical features for its corresponding category and updates them using a momentum mechanism. The similarity between weakly and strongly augmented features of the unlabeled data and the corresponding category queues is calculated separately. , After sorting the similarities in descending order, an exponentially decaying weight is applied: ; This assigns higher weights to difficult-to-align samples with large intra-class differences, thus enhancing the alignment effect of these samples. The intra-class alignment loss is the sum of the weak and strong enhancement alignment losses: ; Inter-class alignment branch: Features are mapped to a non-linear space using an MLP projector, the contrast loss between different class samples is calculated, adaptive weights are applied to increase the inter-class feature distance, and the inter-class alignment loss is denoted as... .

[0052] Total alignment loss for two branches: ; (3) Total semi-supervisory loss ; in, The cross-entropy loss for labeled samples, λ is the pseudo-label cross-entropy loss for unlabeled samples, and λ is the balance coefficient. In this embodiment, the optimal value of λ is 0.2.

[0053] S3, Missing Class Detection and Correction (SRE): In open-label scenarios where unlabeled data contains unknown categories, global category features and local batch features are fused to calculate adaptive debiased prototypes to correct batch sampling bias; the distance distribution from unlabeled samples to each prototype is modeled using a Gaussian mixture model to estimate the category frequency distribution of the unlabeled data; based on the learning gap between known and unknown classes, a differential confidence threshold is dynamically set, and the gradient contribution of minority class pseudo-labels is strengthened through posterior probability weights; unlabeled features are mapped to binary hash codes, and erroneous connections between fine-grained categories are severed through an adaptive Hamming distance threshold; the maximum connected component is retained in the category subgraph to generate a high-quality affinity graph, driving multi-granularity graph contrastive learning; S3 addresses scenarios with open label spaces and unlabeled data containing unknown categories, extending the unknown category discovery capability beyond the second-stage semi-supervised framework. This includes: (1) Progressive prototype estimation: Integrating global category features and local batch features, the adaptive debiased prototype (ADP) is calculated with cosine similarity as the adaptive weight to correct the prototype bias caused by batch sampling; the Gaussian Mixture Model (GMM) is used to model the Euclidean distance distribution from unlabeled samples to each prototype to estimate the distribution of the number of samples of known and unknown classes in the unlabeled data.

[0054] (2) Differentiation threshold and minority class enhancement: Quantify the learning gap between known and unknown classes, dynamically set a confidence threshold for unknown classes that is lower than that for known classes; construct minority class weight coefficients based on posterior probability to enhance the gradient contribution of minority class pseudo-labels and alleviate class imbalance problem.

[0055] (3) Collaborative refinement mechanism: Unlabeled features are mapped to binary hash codes through dot product operation. An adaptive Hamming distance threshold is calculated based on the average cosine similarity within the batch to remove erroneous connection edges between fine-grained similar categories. The largest connected component (LCC) is retained for each category subgraph, and small components that are mixed across categories are removed. A high-quality affinity graph is generated by combining pseudo-labels. Based on the affinity graph, category-level and sample-level multi-granularity graph comparison learning is driven to further optimize feature representation.

[0056] When new categories such as electronic invoice-specific fields appear in the unlabeled invoice data, switch to the missing category semi-supervised mode.

[0057] (1) Progressive Prototype Estimation Module (PPE)

[0058] Adaptive Debiased Prototype (ADP): Maintaining global class features (Global center of class k), local batch features are calculated for each training batch. Global and local features are fused using cosine similarity as the weight: ; ; By using adaptive weights to correct prototype bias caused by batch sampling, a more accurate category prototype can be obtained. , For the j-th local batch feature of the k-th class, Local batch features and global category features The cosine similarity weight.

[0059] GMM class frequency estimation: Calculate the Euclidean distance from all unlabeled samples to each known class prototype, use a Gaussian mixture model to model the distance distribution, fit the distance distribution components of known and unknown classes, and estimate the proportion of known and unknown class samples in the unlabeled data.

[0060] (2) Differentiation threshold and minority class enhancement

[0061] The learning gap metric U between the known and unknown classes is quantified, and a confidence threshold lower than that of the known classes is dynamically set for the unknown classes. It adapts to the characteristics of few unknown class samples and insufficient learning; it constructs minority class weight coefficients based on predicted posterior probabilities. Weighted enhancement of the gradient of minority class pseudo-labels: ; in For indicator functions, (·) represents the cross-entropy loss. To enhance operation, This is a smoothing term.

[0062] (3) Collaborative Refining Mechanism (SR)

[0063] Dynamic Threshold Hash Correction (DTHR): Unlabeled sample features are mapped to binary hash codes through a fully connected layer. The average cosine similarity s of samples within a batch is calculated, and the adaptive Hamming distance threshold is calculated using the following formula: ; Where K is the hash code length, sample pairs with a Hamming distance less than the threshold are considered to be of the same type and the connection edge is retained, while those with a Hamming distance greater than the threshold are disconnected, effectively eliminating erroneous connections between fine-grained similar categories.

[0064] Structural calibration (SC): For each category's subgraph, retain the largest connected component, remove cross-category mixed small components, and combine pseudo-label information to generate a high-quality affinity graph. .

[0065] Affinity graph-driven multi-granularity graph contrastive learning, including category-level and sample-level contrastives: ; in This is the balance coefficient.

[0066] (4) Total missing class semi-supervised loss ; in, For supervised loss of labeled samples, For unlabeled pseudo-label loss with minority class weights, For multi-granularity graph contrast learning loss, As a balance coefficient, in this embodiment Take 0.2.

[0067] S4. Open World Equalization Recognition (DeGCD): In generalized category discovery scenarios that simultaneously include old and new categories, an independent debiasing head is set up to generate auxiliary predictions; a feature momentum memory M and a prediction probability momentum memory P are maintained; nearest neighbor samples are retrieved based on the feature representation of the debiasing head, and the mean of the nearest neighbor predictions is used as the calibration target; semantic calibration of the main classifier is achieved through KL divergence constraints; the prediction results of the main classifier and the debiasing classifier are fused with statistically distributed parameterized mixing coefficients; a self-similarity matrix is ​​constructed based on the fused predictions and a contrastive loss is applied; cross-entropy alignment constraints are applied to predictions in the debiasing classifier with confidence levels higher than a preset threshold to balance the recognition accuracy of the old and new categories.

[0068] S4 addresses the generalized category discovery scenario by simultaneously processing both labeled old categories and unlabeled new categories, resolving the competition between old and new category recognition. It includes: (1) Dual momentum memory and semantic calibration: Maintain feature momentum memory and prediction probability momentum memory, and update them in batches using momentum mechanism; extract feature representations by independent debiasing head, retrieve Top-K nearest neighbor samples in feature memory, use the mean of nearest neighbor predictions as calibration target, minimize the KL divergence between the main classifier prediction and the calibration target, and debias the main classifier by constraining the semantic consistency of nearest neighbors.

[0069] (2) Gamma Mixture Debiased Contrast: Random mixing coefficients that follow the Gamma distribution are used to fuse the predicted logits of the main GCD classifier and the debiased classifier; the self-similarity matrix is ​​calculated based on the mixed prediction and contrast loss is applied to suppress the overconfidence problem of new categories.

[0070] (3) Confidence filtering alignment constraint: Cross-entropy constraint is calculated only for predictions in the bias classifier whose confidence exceeds the preset threshold, to prevent the new category from overlearning and destroying the feature distribution of the old category, and to ensure that the accuracy of the new and old categories is improved simultaneously.

[0071] It should be noted that steps S1-S4 can be deployed independently or cascaded to form a complete system, depending on the actual scenario: use only S1 when there is sufficient labeled data; combine S1 and S2 when the label space is closed and the labels are limited; combine S1-S3 when there are unknown categories; and enable S1-S4 in open-world scenarios. S1-S4 share the underlying technical foundation of multimodal feature extraction, contrastive learning framework, and adaptive threshold control.

[0072] During the open-world deployment phase, where the system is continuously running and new categories are constantly emerging, S4 is enabled to ensure balanced recognition of known old fields and newly added fields.

[0073] (1) Momentum dual memory and semantic calibration loss

[0074] Construct a feature memory M and a prediction probability memory P to store the features and prediction results of historical samples, respectively, and update them in batches using a momentum mechanism: ; ; in, For batch characteristics, For batch prediction probability, , The momentum coefficient is 0.99, and its optimal value is 0.99.

[0075] The bias-reduced feature representation of the samples is extracted using an independent bias-reducing head (MLP + auxiliary classifier). Search for the Top-K nearest neighbors in the feature memory M, and take the predicted mean of the nearest neighbors. As the calibration objective, minimize the KL divergence: ; Where m is the number of old category samples and n is the number of new category samples, the main classifier is constrained by the nearest neighbor semantic consistency to alleviate overconfidence in the new category.

[0076] (2) Gamma distribution mixed debiased contrast loss

[0077] Using random mixing coefficients α that follow a Gamma distribution, the main GCD classifier is fused to predict... With biased classifier prediction : ; We set the value to 0.5, calculate the self-similarity matrix between samples based on the fused predictions, and then apply the contrastive loss. This further curbs overconfidence in new categories.

[0078] (3) Alignment constraint loss with confidence filtering

[0079] Only for the partial classifier whose confidence exceeds the threshold The prediction calculation of cross-entropy constraints ensures the reliability of the constraints: ; In this embodiment The optimal value is 0.95, which, through the self-consistency constraint of high-confidence samples, prevents the new category from overlearning and disrupting the feature distribution of the old category.

[0080] (4) Total loss ; in, For semantic calibration loss, To eliminate biased contrast loss, To align constraint loss, , This is the balance coefficient; in this embodiment... Take 0.35, Take 0.2.

[0081] Table 1: Comparison of Core Performance Indicators at Different Technology Levels

[0082] The basic performance of the DeGCD module is validated on a generalized generalized category discovery benchmark dataset, and its debiased equilibrium learning mechanism can be transferred and adapted to open category discovery scenarios for visual rich documents.

[0083] Table 2: Comparison Table of Core Technology Components for Four-Layer Technology Sharing

[0084] The aforementioned shared components constitute the unified underlying technology support of this invention. Each layer is implemented specifically for different scenarios under a unified technical logic, forming a technical evolution path from heuristic to statistically driven.

[0085] The processing procedure of the prior knowledge-guided module is as follows: The initial visual features are output by the visual encoder, and then two pooled features are generated by global average pooling and max pooling. After passing through a fully connected layer and activation function, the features are multiplied element-wise with the initial visual features to obtain a weighted visual representation. Calculate the difference attention weights between visual regions and perform weighted optimization on the weighted visual representation; The similarity matrix between the optimized visual representation and the text representation is calculated. Attention weights are generated using temperature-controlled Softmax and applied to the text representation to obtain a deambiguous text representation.

[0086] The process of dual-dimensional hierarchical sorting and filtering is as follows: along the two dimensions of visual to text and text to visual, all sample pairs are divided into multiple similarity interval layers according to their similarity values. Within each layer, samples are filtered in descending order, and finally the Top-L' high similarity negative samples are obtained. The optimal value of L' is 32. The mixing coefficients follow a Gamma distribution, i.e., α ~ Gamma(β, β), where the optimal value of β is 0.5.

[0087] Example 2

[0088] This embodiment is applied to the extraction of medical record form information in medical institutions. The scenario is characterized by fixed entity categories (name, diagnosis, medication, test items, etc.), which belong to a closed label space. However, the labeling cost is high. Therefore, only the first stage of cross-modal alignment pre-training and the second stage of context-aware semi-supervised learning are deployed.

[0089] First, DCMAI pre-training was completed using a large-scale unlabeled medical record form corpus to obtain a multimodal feature extractor adapted to medical documents; then, PLBR semi-supervised fine-tuning was started using 5% of labeled medical record data.

[0090] To address the strong context-dependent nature of medical texts (e.g., "insulin" is a drug in the prescription area and a medical history description in the medical history area), the SMBR module uses the contextual prior of the labeled data to guide the correction of pseudo-labels, effectively distinguishing the semantics of the same text in different layout positions; the DBAA module handles the high intra-class variance problem of fields such as test items, achieving high-precision information extraction with low annotation cost.

[0091] Example 3

[0092] This embodiment is applied to a multi-industry general document understanding platform. The scenario is characterized by large differences in the category space of documents from different industries. It is necessary to simultaneously identify known categories and continuously discover new categories. Therefore, a third-stage missing-class semi-supervised and a fourth-stage open-world equilibrium recognition are deployed, and adaptation is performed based on general pre-trained multimodal features.

[0093] Missing category discovery and calibration handles the identification of known industry documents and the discovery of categories for new industry documents. Open-world balanced recognition ensures that the process of discovering new categories does not impair the recognition ability of known categories. Together, they constitute a complete solution for open-world document understanding, which can be quickly adapted to the document processing needs of multiple industries such as finance, government affairs, and law.

[0094] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0095] A multimodal semi-supervised learning system for sparsely labeled visual rich documents, employing any of the aforementioned multimodal semi-supervised learning methods for sparsely labeled visual rich documents, including: The multimodal feature extraction module is used to extract visual features from visual rich documents through a visual encoder and extract text features from documents through a text encoder. The cross-modal alignment pre-training module is used to perform cross-modal alignment pre-training and output the aligned multimodal joint representation; The context-aware semi-supervised learning module is used to perform context-aware semi-supervised learning, generate high-quality pseudo-labels in a closed label space and complete semi-supervised training. The missing class detection and calibration module is used to perform missing class detection and calibration, realizing the discovery of unknown categories and feature calibration in the open label space; The open-world equilibrium recognition module is used to perform open-world equilibrium recognition, balancing the recognition accuracy of new and old categories in generalized category discovery scenarios; The joint optimization engine is used to perform weighted combination and end-to-end optimization scheduling of the loss functions of the multimodal feature extraction module, cross-modal alignment pre-training module, context-aware semi-supervised learning module, missing class detection and calibration module, and open-world equilibrium recognition module.

[0096] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal semi-supervised learning method for sparsely labeled visual rich documents, characterized in that, Includes the following steps: S1. Cross-modal alignment pre-training: Visual encoders are used to extract visual representations of rich document images, and text encoders are used to extract text representations of documents; a visually guided deambiguous text representation is generated through a prior knowledge-guided module; before cross-modal interaction, the contrast loss in both text-to-visual and visual-to-text directions is calculated simultaneously, and the cross-alignment loss is obtained through weighted fusion of hybrid coefficients to achieve bidirectional modal semantic space alignment. The visual-text similarity calculation results are reused, and the similarity is sorted hierarchically along the two dimensions of visual to text and text to visual. High similarity negative samples in each level are selected to form an effective negative sample subset to drive joint representation contrastive learning. S2. Context-aware semi-supervised learning: In scenarios with a closed label space and limited labeled samples, a first similarity matrix between labeled data prediction and pseudo-labels, and a second similarity matrix between labeled data prediction and strongly enhanced prediction of unlabeled data are constructed. The two similarity matrices are fused with statistically distributed parameterized random mixing coefficients, and guided pseudo-labels are generated after graph Laplacian normalization. Intra-class feature alignment is achieved through momentum queues grouped by category and exponential decay weights, and inter-class feature separation is achieved through nonlinear projection and adaptive contrastive loss. S3. Missing Class Detection and Correction: In open-label scenarios where unlabeled data contains unknown categories, global category features and local batch features are fused to calculate adaptive debiased prototypes to correct batch sampling bias. A Gaussian mixture model is used to model the distance distribution from unlabeled samples to each prototype, estimating the category frequency distribution of the unlabeled data. Based on the learning gap between known and unknown classes, a differential confidence threshold is dynamically set, and the gradient contribution of minority class pseudo-labels is strengthened through posterior probability weights. Unlabeled features are mapped to binary hash codes, and erroneous connections between fine-grained categories are severed using an adaptive Hamming distance threshold. The largest connected component is retained in the category subgraph to generate a high-quality affinity graph, driving multi-granularity graph contrastive learning. S4, Open World Equilibrium Recognition: In generalized category discovery scenarios that simultaneously include old and new categories, an independent de-biasing mechanism is set to generate auxiliary predictions; Maintain the feature momentum memory M and the prediction probability momentum memory P, retrieve nearest neighbor samples based on the debiased feature representation, use the mean of the nearest neighbor predictions as the calibration target, and achieve semantic calibration of the main classifier through KL divergence constraints; fuse the prediction results of the main classifier and the debiased classifier with the mixture coefficients parameterized by statistical distribution, construct a self-similarity matrix based on the fused predictions and apply contrastive loss. Cross-entropy alignment constraints are applied to predictions with confidence levels higher than a preset threshold in the biased classifier to balance the recognition accuracy of the new and old classes.

2. The multimodal semi-supervised learning method for sparsely labeled visual rich documents according to claim 1, characterized in that, The processing procedure of the prior knowledge guidance module is as follows: The initial visual features are output by the visual encoder, and then two pooled features are generated by global average pooling and max pooling. After passing through a fully connected layer and activation function, the features are multiplied element-wise with the initial visual features to obtain a weighted visual representation. Calculate the difference attention weights between visual regions and perform weighted optimization on the weighted visual representation; The similarity matrix between the optimized visual representation and the text representation is calculated. Attention weights are generated using temperature-controlled Softmax and applied to the text representation to obtain a deambiguous text representation.

3. The multimodal semi-supervised learning method for sparsely labeled visual rich documents according to claim 1, characterized in that, The formula for calculating the cross alignment loss is: =α +（1-α） ； in, For contrast loss in the text-to-visual direction; α represents the contrast loss from visual to text direction; α is the mixing coefficient, with an optimal value of 0.

4.

4. The multimodal semi-supervised learning method for sparsely labeled visual rich documents according to claim 1, characterized in that, The process of dual-dimensional hierarchical sorting and filtering is as follows: along the two dimensions of visual to text and text to visual, all sample pairs are divided into multiple similarity interval layers according to their similarity values. Within each layer, samples are filtered in descending order, and finally, the Top-L' high-similarity negative samples are obtained. The optimal value of L' is 32. The total loss for cross-modal alignment pre-training is: =ψ + +oh ; in, For masked language modeling loss, For cross-alignment loss, The contrastive learning loss is represented jointly, with ψ and ω as balance coefficients.

5. The multimodal semi-supervised learning method for sparsely labeled visual rich documents according to claim 1, characterized in that, The random mixing coefficients follow a Gamma distribution, i.e., α ~ Gamma(β, β), where the optimal value of β is 0.6; The total loss of context-aware semi-supervised learning is: = + +λ ; in, For supervised loss of labeled samples, For pseudo-label loss of unlabeled samples, The loss is a bi-branch adaptive alignment loss, where λ is the balance coefficient, with an optimal value of 0.

2.

6. The multimodal semi-supervised learning method for sparsely labeled visual rich documents according to claim 1, characterized in that, The calculation method for the adaptive bias-reduction prototype is as follows: ； in, For the j-th local batch feature of the k-th class, Local batch features and global category features Cosine similarity weight; The formula for calculating the adaptive Hamming distance threshold is as follows: ； Where K is the hash code length and s is the average cosine similarity of samples within the batch.

7. The multimodal semi-supervised learning method for sparsely labeled visual rich documents according to claim 1, characterized in that, The total loss from the missing class detection and calibration is: ； in, For supervised loss of labeled samples, For unlabeled pseudo-label loss with minority class weights, For multi-granularity graph contrast learning loss, This is the balance coefficient.

8. The multimodal semi-supervised learning method for sparsely labeled visual rich documents according to claim 1, characterized in that, The update method for the feature momentum memory M and the prediction probabilistic momentum memory P is as follows: ； ； in, For batch characteristics, For batch prediction probability, , The momentum coefficient has an optimal value of 0.

99. The mixing coefficients follow a Gamma distribution, i.e., α ~ Gamma(β, β), where the optimal value of β is 0.

5.

9. The multimodal semi-supervised learning method for sparsely labeled visual rich documents according to claim 1, characterized in that, The total loss for the open-world equilibrium identification is: ； in, For semantic calibration loss, To eliminate biased contrast loss, To align constraint loss, , The is the balance coefficient; the optimal value of the confidence threshold η for the alignment constraint loss is 0.

95.

10. A multimodal semi-supervised learning system for sparsely labeled visual rich documents, employing the multimodal semi-supervised learning method for sparsely labeled visual rich documents as described in any one of claims 1-9, characterized in that, include: The multimodal feature extraction module is used to extract visual features from visual rich documents through a visual encoder and extract text features from documents through a text encoder. The cross-modal alignment pre-training module is used to perform cross-modal alignment pre-training and output the aligned multimodal joint representation; The context-aware semi-supervised learning module is used to perform context-aware semi-supervised learning, generate high-quality pseudo-labels in a closed label space and complete semi-supervised training. The missing class detection and calibration module is used to perform missing class detection and calibration, realizing the discovery of unknown categories and feature calibration in the open label space; The open-world equilibrium recognition module is used to perform open-world equilibrium recognition, balancing the recognition accuracy of new and old categories in generalized category discovery scenarios; The joint optimization engine is used to perform weighted combination and end-to-end optimization scheduling of the loss functions of the multimodal feature extraction module, cross-modal alignment pre-training module, context-aware semi-supervised learning module, missing class detection and calibration module, and open-world equilibrium recognition module.

Images