A large model-based high anti-magnetic passbook printing anomaly detection system

Abstract

The application discloses a kind of high magnetically resistant passbook printing anomaly detection systems based on large model, comprising: data acquisition processing module, for collecting job data and pre-processing;Structural construction module, for extracting row and column boundary interval, and executing field-level structure coding, construct training sample set;Model fine-tuning module, for based on training sample set, task fine-tuning is carried out to LLaVA, and printing abnormality discrimination model is obtained;Abnormal diagnosis module, for field index alignment and multi-modal sequence assembly to real-time printing job, and constraint decoding is carried out, and abnormal diagnosis result is generated;Result check module, for locating the field image area corresponding to abnormal field, similarity measure is carried out, and abnormal diagnosis result is revised;Feedback updating module, for collecting treatment feedback data, and printing abnormality discrimination model is incrementally updated.The application can realize the accurate positioning of abnormal field of passbook printing job, improve the accuracy of anomaly detection.

Description

Technical Field

[0001] This invention relates to the field of passbook printing anomaly detection technology, and in particular to a high antimagnetic passbook printing anomaly detection system based on a large model. Background Technology

[0002] In passbook printing at financial self-service machines and counter terminals, printing jobs typically involve the collaborative processing of multi-source information, including layout images, field identifiers, and field rules. Current technologies often employ rule-based layout verification or image-difference-based template comparison, relying on fixed coordinates, fonts, and layout parameters to locate fields and determine anomalies.

[0003] Another existing approach uses traditional visual algorithms for field segmentation and OCR recognition before performing consistency checks. This type of method typically obtains candidate regions using methods such as connected component analysis and projection segmentation, and then compares the character recognition results with the field values ​​to complete anomaly detection. Because the field location and character recognition chain is relatively long, any location drift, segmentation error, or recognition instability at any stage will be amplified, and there is a lack of unified modeling for the overall layout consistency and cross-field correlation between fields.

[0004] In recent years, there have been solutions that use large models for anomaly identification of invoices and documents, but most of them are limited to direct reasoning or simple input prompts, lacking a training sample construction mechanism for business fields and controllable alignment constraints. Model updates usually adopt full retraining or coarse-grained fine-tuning, lacking an incremental update mechanism linked to handling feedback, making it difficult to continuously adapt to new templates, new anomaly types and changes in on-site data distribution, resulting in model drift and difficulty in maintaining diagnostic consistency.

[0005] Furthermore, existing methods often employ heuristic matching or manually set order in the field alignment and anomaly localization stages, lacking a global optimal assignment solution mechanism based on the cost matrix. In the anomaly correction stage, corrections are often completed using a single threshold or simple comparison, lacking joint constraints of alignment compensation and similarity measurement, which makes it difficult to effectively converge false alarms.

[0006] Therefore, how to provide a high-coercivity passbook printing anomaly detection system based on a large model is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] One objective of this invention is to propose a high-coercivity passbook printing anomaly detection system based on a large model. This invention utilizes a multimodal visual language large model, field-level layout analysis method, optimal assignment matching algorithm, and incremental learning mechanism. It describes in detail the technical implementation process of anomaly detection, field alignment, anomaly correction, and model adaptive updating for passbook printing business. It has the advantages of high anomaly recognition stability, strong adaptability to complex layouts, and strong model continuous evolution capability.

[0008] A high-coherence magnetic passbook printing anomaly detection system based on a large model according to an embodiment of the present invention includes:

[0009] The data acquisition and processing module is used to collect the job data of high antimagnetic passbook printing and preprocess it to form a standard dataset;

[0010] The structure construction module is used to extract the row and column boundary intervals of each field in the standard dataset, use the interval tree algorithm to perform interval matching and boundary concatenation on the row and column boundary intervals, and perform field-level structure encoding on the standard dataset based on the matching and concatenation results to construct a training sample set;

[0011] The model fine-tuning module is used to fine-tune the multimodal visual language large model LLaVA based on the training sample set. It groups the model parameters and allocates different learning rates according to the learning rate scheduling rules, and performs gradient updates on the anomaly discrimination parameter group to obtain the print anomaly discrimination model.

[0012] The anomaly diagnosis module uses the Hungarian matching algorithm to perform field index alignment and multimodal sequence assembly on real-time printing jobs, and inputs the assembled multimodal inference sequence into the printing anomaly discrimination model for constraint decoding to generate anomaly diagnosis results.

[0013] The result verification module is used to locate the field image region corresponding to the abnormal field based on the field index alignment result, use the normalized cross-correlation algorithm to measure similarity, and correct the abnormal diagnosis result according to the preset threshold.

[0014] The feedback update module is used to collect and process feedback data and incrementally update the printing anomaly detection model.

[0015] Optionally, the data acquisition and processing module specifically includes:

[0016] The collected job data is merged and organized according to the job identifier. The merged job data is parsed for log fields and unstructured logs are converted into structured event fields. The images in the job data are subjected to size normalization, orientation correction and grayscale enhancement. The printing instructions in the job data are subjected to parameter decomposition and field normalization. The template records in the job data are subjected to version parsing and field alignment to form a standard dataset.

[0017] Optionally, the structural construction module specifically includes:

[0018] Based on the image records in the standard dataset, the pixel density projection distribution is calculated along the row and column directions respectively, and the row and column boundary intervals are obtained by segmenting according to the peak and valley changes of the projection curve.

[0019] The interval tree algorithm is used to perform interval insertion and interval retrieval on row and column boundary intervals. A balanced interval tree node structure is constructed based on the start and end coordinates of the intervals on the row and column boundary intervals. The maximum ending coordinate of the interval set is maintained in the node, and non-overlapping search paths are pruned based on the maximum ending coordinate.

[0020] Using row boundary intervals as query intervals, interval overlap calculations are performed on the interval tree. The retrieval space is reduced and candidate column boundary intervals are obtained through the maximum termination coordinate pruning strategy.

[0021] Calculate the interval overlap metric between the candidate column boundary interval and the corresponding row boundary interval. The metric includes the overlap length ratio and the interval coverage index.

[0022] Based on the overlap metric, interval stability determination conditions are constructed, and row and column intervals that meet the constraints are selected according to the stability determination conditions.

[0023] Based on the row and column intervals that satisfy stability constraints, a hierarchical clustering algorithm is used to aggregate the endpoints of the intervals, and multi-interval fusion calculation is performed based on the aggregation results to generate the field boundary range.

[0024] Based on the field boundary range, the image records in the standard dataset are cropped at the field level, and the field order encoding and field identifier alignment are performed according to the field arrangement relationship to construct a training sample set.

[0025] Optionally, the model fine-tuning module specifically includes:

[0026] The training sample set is divided into several training subsets according to the anomaly category and field type, and a multimodal training sequence is constructed for each training subset;

[0027] Based on the multimodal training sequence, calibration batches are selected, and forward computation and backward gradient sampling are performed on the multimodal visual language large model LLaVA. The gradient sensitivity index of each parameter on the calibration batch is statistically analyzed, and the model parameters are segmented by threshold based on the gradient sensitivity index to form a basic parameter group and an anomaly discrimination parameter group.

[0028] Configure group learning rate scheduling rules for each parameter group, use cosine annealing learning rate scheduling to update the learning rate round by round, and configure an upper limit value for the learning rate for the anomaly discrimination parameter group. The group learning rate scheduling rules include applying an upper limit pruning constraint to the learning rate of the anomaly discrimination parameter group and applying a scaling constraint to the learning rate of the other parameter groups.

[0029] The multimodal training sequences are extracted in batches and input into the multimodal visual language large model LLaVA to perform forward computation;

[0030] For each batch, resampling or weighted sampling is performed according to the distribution of anomaly categories, and the hard cases are weighted in the loss calculation;

[0031] Based on the anomaly category label, construct the Focal Loss function and perform backpropagation to calculate the gradient;

[0032] The gradient is subjected to norm clipping and outlier gradient masking, and the AdamW optimization algorithm is used to introduce a weight decay term to perform parameter updates;

[0033] Based on the group learning rate scheduling rules, batch learning rates are applied to the anomaly discrimination parameter group and the other parameter groups respectively. Parameter updates are performed on the anomaly discrimination parameter group and update magnitude constraints are applied to the other parameter groups.

[0034] After each training round, the continuation and termination of training iterations are controlled based on the trend of loss changes and the preset early stopping criteria.

[0035] After the predetermined number of iterations, the model weight state is saved to obtain the printed anomaly detection model.

[0036] Optionally, the formation of the basic parameter set and the anomaly discrimination parameter set specifically includes:

[0037] Calibration batches are extracted from the multimodal training sequence, and forward and backward propagation calculations are performed on the calibration batches in the multimodal visual language large model LLaVA to obtain the gradient sampling results of each parameter on the calibration batches.

[0038] The gradient sampling results are normalized by parameter scale, and a gradient sensitivity index is constructed based on the normalized gradient magnitude statistics.

[0039] Perform quantile threshold estimation on the gradient sensitivity index to determine the sensitivity threshold and generate threshold segmentation conditions;

[0040] The model parameters are filtered and grouped according to the threshold segmentation condition. Parameters with gradient sensitivity higher than the sensitivity threshold are classified into the anomaly discrimination parameter group, and parameters with gradient sensitivity lower than the sensitivity threshold are classified into the basic parameter group.

[0041] Repeat the gradient sensitivity index calculation and statistical analysis of the group consistency ratio under different calibration batches. Parameters with a consistency ratio lower than the preset threshold are reassigned, and the parameter grouping results are output.

[0042] Optionally, the anomaly diagnosis module specifically includes:

[0043] Adaptive threshold segmentation and morphological closing operations are performed on the layout image of the real-time printing job. The bounding rectangles of the text connected regions are extracted using the connected component labeling algorithm. Adjacent bounding rectangles are merged according to the spacing threshold to generate a set of field candidate boxes.

[0044] Field identifiers are mapped to field indices in a preset order, a comprehensive cost matrix of field indices and field candidate boxes is constructed, matching pairs with cost values ​​exceeding a threshold are gated and eliminated, and the Hungarian matching algorithm is used to solve the correspondence between field indices and field candidate boxes.

[0045] Based on the correspondence, the candidate field boxes are cropped and their size is normalized. The cropping results are then assembled into a model input sequence with the corresponding field identifiers according to the field index order.

[0046] Based on the model input sequence, a print anomaly detection model is used to perform constraint decoding. The output field positions are limited according to the field index order, and decoding results that do not meet the field position order constraints are masked to generate anomaly diagnosis results.

[0047] Optionally, the solution to the correspondence between the field index and the field candidate box specifically includes:

[0048] Perform sequential encoding on field identifiers according to a preset order, and map the sequential encoding results to field indexes;

[0049] The Mahalanobis distance algorithm is used to calculate the center offset distance between the field index and the field candidate box, and the width and height difference of the candidate box is calculated. The center offset distance and the width and height difference are normalized and then weighted to synthesize the matching cost.

[0050] A comprehensive cost matrix is ​​constructed based on the matching cost and the field candidate box set. Matrix elements with matching costs exceeding a threshold are marked as unselectable, forming a gating constraint.

[0051] Under gating constraints, Hungarian matching is performed on the comprehensive cost matrix, and row and column reductions are performed on the comprehensive cost matrix to eliminate the minimum row cost and minimum column cost, resulting in a reduction matrix containing zero cost elements.

[0052] Perform a minimum line cover operation on the zero-cost elements in the reduction matrix to calculate the minimum number of rows and columns required to cover all zero-cost elements.

[0053] When the minimum number of lines covered is less than the number of field indexes, select the minimum cost of the elements not covered by lines and perform subtraction update on the uncovered elements, and perform addition compensation update on the cross-coverage positions to update the distribution of zero-cost elements in the reduction matrix.

[0054] Repeat the minimum line coverage operation and matrix update until the minimum line coverage reaches the field index number to obtain a zero-cost structure matrix;

[0055] Assignment selection is performed based on a zero-cost structure matrix, and the correspondence between the output field index and the field candidate box is generated.

[0056] Optionally, the result verification module specifically includes:

[0057] Based on the correspondence between field indexes and field candidate boxes, the field indexes marked as abnormal in the anomaly diagnosis results are selected, and the cropping boundaries of the corresponding field candidate boxes are determined. The field image regions are then cropped from the layout image.

[0058] Based on the field image region and the standard field image corresponding to the same field index in the standard dataset, size normalization and grayscale normalization are performed, and the phase correlation algorithm is used to estimate the translation offset between the two. The field image region is then aligned based on the translation offset. The phase correlation algorithm is used to estimate the translation offset between the two images in the frequency domain based on phase information.

[0059] The similarity between the aligned field image region and the standard field image is calculated using a normalized cross-correlation algorithm to obtain the field similarity results.

[0060] Threshold gating is applied to the field similarity results, and the field indexes with similarity results below the threshold are retained as abnormal status markers to obtain the corrected abnormal diagnosis results.

[0061] Optionally, the feedback update module specifically includes:

[0062] Collect feedback data from the execution of anomaly diagnosis results, and associate the feedback data with the field index, anomaly category label, and corrected anomaly diagnosis results of the corresponding job to construct an incremental sample set;

[0063] Based on incremental samples, anomaly confidence scores are obtained by the printing anomaly discrimination model, and the anomaly confidence scores are probabilistically calibrated using a temperature scaling algorithm. Based on the calibrated confidence scores, sample confidence weights are generated and updated samples are obtained through screening.

[0064] Weighted incremental training is performed on the updated samples according to the sample confidence weights. The RMSProp optimization algorithm is used to perform gradient updates on the anomaly discrimination parameter group, and update magnitude constraints are applied to the other parameter groups.

[0065] During incremental training, replay sampling is performed on the updated samples, and historical samples are mixed with the updated samples according to a preset ratio to form training batches. Iterative training is then performed on the training batches.

[0066] After the predetermined incremental update rounds are completed, the model weight state is saved, and the incremental update of the print anomaly detection model is completed.

[0067] The beneficial effects of this invention are:

[0068] First, based on the standardization of job data and field-level structure encoding, this invention completes field index alignment through candidate box generation and global assignment matching, and outputs anomaly diagnosis results in conjunction with constraint decoding, so that the anomaly judgment process forms a consistent technical link from page segmentation, field alignment to diagnosis output.

[0069] Secondly, on the model side, this invention adopts a task fine-tuning method based on calibration batch gradient sensitivity grouping and group learning rate scheduling, which concentrates the anomaly detection capability in the iterative update of the anomaly detection parameter group, and on the inference side, combines the field alignment results to organize the model input sequence, thereby realizing the corresponding output of field-level anomaly diagnosis and field index.

[0070] Finally, this invention introduces phase correlation alignment and normalized cross-correlation similarity measurement in the anomaly correction stage, performs threshold gating correction on the anomaly fields, and continuously updates the model through a confidence-gated incremental update mechanism driven by the processing feedback data. Attached Figure Description

[0071] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0072] Fig. 1 This is a module structure diagram of a high diamagnetic passbook printing anomaly detection system based on a large model proposed in this invention;

[0073] Fig. 2 This is a schematic diagram illustrating the generation of a printing anomaly discrimination model for a high diamagnetic passbook printing anomaly detection system based on a large model proposed in this invention.

[0074] Fig. 3 This is a schematic diagram of constraint decoding anomaly diagnosis for a high diamagnetic passbook printing anomaly detection system based on a large model proposed in this invention. Detailed Implementation

[0075] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0076] refer to Figs. 1-3 A high-coercivity passbook printing anomaly detection system based on a large model includes:

[0077] The data acquisition and processing module is used to collect the job data of high antimagnetic passbook printing and preprocess it to form a standard dataset;

[0078] The structure construction module is used to extract the row and column boundary intervals of each field in the standard dataset, use the interval tree algorithm to perform interval matching and boundary concatenation on the row and column boundary intervals, and perform field-level structure encoding on the standard dataset based on the matching and concatenation results to construct a training sample set;

[0079] The model fine-tuning module is used to fine-tune the multimodal visual language large model LLaVA based on the training sample set. It groups the model parameters and allocates different learning rates according to the learning rate scheduling rules, and performs gradient updates on the anomaly discrimination parameter group to obtain the print anomaly discrimination model.

[0080] The anomaly diagnosis module uses the Hungarian matching algorithm to perform field index alignment and multimodal sequence assembly on real-time printing jobs, and inputs the assembled multimodal inference sequence into the printing anomaly discrimination model for constraint decoding to generate anomaly diagnosis results.

[0081] The result verification module is used to locate the field image region corresponding to the abnormal field based on the field index alignment result, use the normalized cross-correlation algorithm to measure similarity, and correct the abnormal diagnosis result according to the preset threshold.

[0082] The feedback update module is used to collect and process feedback data and incrementally update the printing anomaly detection model.

[0083] In this embodiment, the data acquisition and processing module specifically includes:

[0084] The collected job data is merged and organized according to the job identifier. The merged job data is parsed for log fields and unstructured logs are converted into structured event fields. The images in the job data are subjected to size normalization, orientation correction and grayscale enhancement. The printing instructions in the job data are subjected to parameter decomposition and field normalization. The template records in the job data are subjected to version parsing and field alignment to form a standard dataset.

[0085] In this embodiment, the structural construction module specifically includes:

[0086] Based on the image records in the standard dataset, the pixel density projection distribution is calculated along the row and column directions respectively, and the row and column boundary intervals are obtained by segmenting according to the peak and valley changes of the projection curve.

[0087] The interval tree algorithm is used to perform interval insertion and interval retrieval on row and column boundary intervals. A balanced interval tree node structure is constructed based on the start and end coordinates of the intervals on the row and column boundary intervals. The maximum ending coordinate of the interval set is maintained in the node, and non-overlapping search paths are pruned based on the maximum ending coordinate.

[0088] Using row boundary intervals as query intervals, interval overlap calculations are performed on the interval tree. The retrieval space is reduced and candidate column boundary intervals are obtained through the maximum termination coordinate pruning strategy.

[0089] Calculate the interval overlap metric between the candidate column boundary interval and the corresponding row boundary interval. The metric includes the overlap length ratio and the interval coverage index.

[0090] Based on the overlap metric, interval stability determination conditions are constructed, and row and column intervals that meet the constraints are selected according to the stability determination conditions.

[0091] Based on the row and column intervals that satisfy stability constraints, a hierarchical clustering algorithm is used to aggregate the endpoints of the intervals, and multi-interval fusion calculation is performed based on the aggregation results to generate the field boundary range.

[0092] Based on the field boundary range, the image records in the standard dataset are cropped at the field level, and the field order encoding and field identifier alignment are performed according to the field arrangement relationship to construct a training sample set.

[0093] In this embodiment, the model fine-tuning module specifically includes:

[0094] The training sample set is divided into several training subsets according to the anomaly category and field type, and a multimodal training sequence is constructed for each training subset;

[0095] Based on the multimodal training sequence, calibration batches are selected, and forward computation and backward gradient sampling are performed on the multimodal visual language large model LLaVA. The gradient sensitivity index of each parameter on the calibration batch is statistically analyzed, and the model parameters are segmented by threshold based on the gradient sensitivity index to form a basic parameter group and an anomaly discrimination parameter group.

[0096] Configure group learning rate scheduling rules for each parameter group, use cosine annealing learning rate scheduling to update the learning rate round by round, and configure an upper limit value for the learning rate for the anomaly discrimination parameter group. The group learning rate scheduling rules include applying an upper limit pruning constraint to the learning rate of the anomaly discrimination parameter group and applying a scaling constraint to the learning rate of the other parameter groups.

[0097] The multimodal training sequences are extracted in batches and input into the multimodal visual language large model LLaVA to perform forward computation;

[0098] For each batch, resampling or weighted sampling is performed according to the distribution of anomaly categories, and the hard cases are weighted in the loss calculation;

[0099] Based on the anomaly category label, construct the Focal Loss function and perform backpropagation to calculate the gradient;

[0100] The gradient is subjected to norm clipping and outlier gradient masking, and the AdamW optimization algorithm is used to introduce a weight decay term to perform parameter updates;

[0101] Based on the group learning rate scheduling rules, batch learning rates are applied to the anomaly discrimination parameter group and the other parameter groups respectively. Parameter updates are performed on the anomaly discrimination parameter group and update magnitude constraints are applied to the other parameter groups.

[0102] After each training round, the continuation and termination of training iterations are controlled based on the trend of loss changes and the preset early stopping criteria.

[0103] After the predetermined number of iterations, the model weight state is saved to obtain the printed anomaly detection model.

[0104] In this embodiment, the formation of the basic parameter set and the anomaly discrimination parameter set specifically includes:

[0105] Calibration batches are extracted from the multimodal training sequence, and forward and backward propagation calculations are performed on the calibration batches in the multimodal visual language large model LLaVA to obtain the gradient sampling results of each parameter on the calibration batches.

[0106] The gradient sampling results are normalized by parameter scale, and a gradient sensitivity index is constructed based on the normalized gradient magnitude statistics.

[0107] Perform quantile threshold estimation on the gradient sensitivity index to determine the sensitivity threshold and generate threshold segmentation conditions;

[0108] The model parameters are filtered and grouped according to the threshold segmentation condition. Parameters with gradient sensitivity higher than the sensitivity threshold are classified into the anomaly discrimination parameter group, and parameters with gradient sensitivity lower than the sensitivity threshold are classified into the basic parameter group.

[0109] Repeat the gradient sensitivity index calculation and statistical analysis of the group consistency ratio under different calibration batches. Parameters with a consistency ratio lower than the preset threshold are reassigned, and the parameter grouping results are output.

[0110] In this embodiment, the anomaly diagnosis module specifically includes:

[0111] Adaptive thresholding and morphological closing operations are performed on the layout image of the real-time print job. A connected component labeling algorithm is used to extract the bounding rectangles of the text's connected regions. Adjacent bounding rectangles are then merged according to a spacing threshold to generate a set of candidate field boxes. The generation of the candidate field box set includes:

[0112] The gray-level mean and gray-level variance are calculated within a local window, and a pixel-level adaptive threshold is calculated based on the local mean and local variance to perform binarization segmentation on the page image.

[0113] The width distribution of the foreground connected region is statistically analyzed for the binarization result, and the size and orientation parameters of the structuring element for the morphological closing operation are determined based on the width distribution. The closing operation is then performed on the binarization result, and the number of iterations for the closing operation is limited.

[0114] Perform connected component labeling on the closing operation result, extract the bounding rectangle of each connected component, and calculate the area of ​​the connected component, the aspect ratio of the bounding rectangle and the proportion of foreground pixels, and remove connected components that do not meet the preset constraints;

[0115] For the circumscribed rectangles, calculate the center point distance, horizontal spacing, vertical spacing, and overlap ratio, and group the circumscribed rectangles that satisfy the spacing and overlap constraints into the same group.

[0116] For each group, the smallest bounding box of the outer rectangle is taken as the merged field candidate box, resulting in a set of field candidate boxes;

[0117] Field identifiers are mapped to field indices in a preset order, a comprehensive cost matrix of field indices and field candidate boxes is constructed, matching pairs with cost values ​​exceeding a threshold are gated and eliminated, and the Hungarian matching algorithm is used to solve the correspondence between field indices and field candidate boxes.

[0118] Based on the correspondence, the candidate field boxes are cropped and their size is normalized. The cropping results are then assembled into a model input sequence with the corresponding field identifiers according to the field index order.

[0119] Based on the model input sequence, a print anomaly detection model is used to perform constraint decoding. The output field positions are limited according to the field index order, and decoding results that do not meet the field position order constraints are masked to generate anomaly diagnosis results.

[0120] In this embodiment, the solution to the correspondence between the field index and the field candidate box specifically includes:

[0121] Perform sequential encoding on field identifiers according to a preset order, and map the sequential encoding results to field indexes;

[0122] The Mahalanobis distance algorithm is used to calculate the center offset distance between the field index and the field candidate box, and the width and height difference of the candidate box is calculated. The center offset distance and the width and height difference are normalized and then weighted to synthesize the matching cost.

[0123] A comprehensive cost matrix is ​​constructed based on the matching cost and the field candidate box set. Matrix elements with matching costs exceeding a threshold are marked as unselectable to form gating constraints.

[0124] Under gating constraints, Hungarian matching is performed on the comprehensive cost matrix, and row and column reductions are performed on the comprehensive cost matrix to eliminate the minimum row cost and minimum column cost, resulting in a reduction matrix containing zero cost elements.

[0125] Perform a minimum line cover operation on the zero-cost elements in the reduction matrix to calculate the minimum number of rows and columns required to cover all zero-cost elements.

[0126] When the minimum number of lines covered is less than the number of field indexes, select the minimum cost of the elements not covered by lines and perform subtraction update on the uncovered elements, and perform addition compensation update on the cross-coverage positions to update the distribution of zero-cost elements in the reduction matrix.

[0127] Repeat the minimum line coverage operation and matrix update until the minimum line coverage reaches the field index number to obtain a zero-cost structure matrix;

[0128] Assignment selection is performed based on a zero-cost structure matrix, and the correspondence between the output field index and the field candidate box is generated.

[0129] In this embodiment, the result verification module specifically includes:

[0130] Based on the correspondence between field indexes and field candidate boxes, the field indexes marked as abnormal in the anomaly diagnosis results are selected, and the cropping boundaries of the corresponding field candidate boxes are determined. The field image regions are then cropped from the layout image.

[0131] Based on the field image region and the standard field image corresponding to the same field index in the standard dataset, size normalization and grayscale normalization are performed, and the phase correlation algorithm is used to estimate the translation offset between the two. The field image region is then aligned based on the translation offset. The phase correlation algorithm is used to estimate the translation offset between the two images in the frequency domain based on phase information.

[0132] Perform two-dimensional Fourier transforms on the field image region and the standard field image respectively to obtain the corresponding complex frequency domain spectrum matrix;

[0133] The complex frequency domain spectrum matrix of the field image region is multiplied point-by-point with the conjugate matrix of the complex frequency domain spectrum matrix of the standard field image to obtain the cross spectrum matrix, and the amplitude normalization process is performed on the cross spectrum matrix.

[0134] Perform a two-dimensional inverse Fourier transform on the cross-spectral matrix after amplitude normalization to obtain the correlation response map;

[0135] Locate the maximum response point in the relevant response map, and perform offset conversion on the coordinates of the maximum response point based on the image width and height to obtain the translation offset;

[0136] The field image region is translated based on the translation offset to complete the alignment transformation;

[0137] The similarity between the aligned field image region and the standard field image is calculated using a normalized cross-correlation algorithm to obtain the field similarity results.

[0138] Threshold gating is applied to the field similarity results, and the field indexes with similarity results below the threshold are retained as abnormal status markers to obtain the corrected abnormal diagnosis results.

[0139] In this embodiment, the feedback update module specifically includes:

[0140] Collect feedback data from the execution of anomaly diagnosis results, and associate the feedback data with the field index, anomaly category label, and corrected anomaly diagnosis results of the corresponding job to construct an incremental sample set;

[0141] Based on incremental samples, anomaly confidence scores are obtained by the printing anomaly discrimination model, and the anomaly confidence scores are probabilistically calibrated using a temperature scaling algorithm. Based on the calibrated confidence scores, sample confidence weights are generated and updated samples are obtained through screening.

[0142] Weighted incremental training is performed on the updated samples according to the sample confidence weights. The RMSProp optimization algorithm is used to perform gradient updates on the anomaly discrimination parameter group, and update magnitude constraints are applied to the other parameter groups.

[0143] During incremental training, replay sampling is performed on the updated samples, and historical samples are mixed with the updated samples according to a preset ratio to form training batches. Iterative training is then performed on the training batches.

[0144] After the predetermined incremental update rounds are completed, the model weight state is saved, and the incremental update of the print anomaly detection model is completed.

[0145] Example 1: To verify the feasibility of this invention in practice, it was applied to an anomaly detection scenario in a certain type of passbook printing business. In this scenario, when the terminal continuously processes printing jobs, the page image is affected by factors such as creases, stains, slight tilt, and template switching, causing field boundaries and relative positions to drift, easily resulting in anomalies such as field misalignment, missing characters, character adhesion and breakage, and occlusion. Existing rule-based verification methods rely on fixed coordinates and fixed layout parameters, requiring frequent maintenance after template changes; the traditional "field segmentation followed by identification and comparison" chain is long, and positioning deviations amplify identification and comparison errors, leading to both false alarms and false negatives, and frequent inconsistencies between the handling feedback and system diagnosis.

[0146] In this scenario, the invention first standardizes the job data to form a standard dataset, extracts field boundary intervals to complete field-level structural encoding, and constructs a training sample set for fine-tuning a multimodal visual language model to obtain a printing anomaly discrimination model. During online detection, adaptive threshold segmentation, morphological closing operations, and connected component labeling are performed on the page image to generate a set of field candidate boxes; field identifiers are mapped to field indices in a preset order, a comprehensive cost matrix is ​​constructed based on matching costs and gated elimination is performed, and Hungarian matching is used to solve the one-to-one correspondence between field indices and field candidate boxes; the model input sequence is assembled according to the field index order and constraint decoding is performed to output the anomaly diagnosis result.

[0147] For fields deemed abnormal, this invention crops the field image region based on the field index alignment result and performs size and grayscale normalization processing with the standard field image; a phase correlation algorithm is used to estimate the translation offset to complete the alignment transformation; a normalized cross-correlation algorithm is used to calculate the field similarity on the alignment result, and threshold gating is performed to correct the abnormal state label. The processed feedback data enters the confidence-gated incremental update mechanism, and after probability calibration to screen the update samples, weighted incremental training is performed and historical playback samples are mixed in to complete the continuous update of the model.

[0148] Of the 2400 statistical work samples, 620 were manually reviewed and confirmed to be abnormal. The rule-based validation scheme resulted in 148 missed detections and 173 false positives; the traditional character recognition link scheme resulted in 121 missed detections and 146 false positives. The initial model of this invention resulted in 62 missed detections and 71 false positives; after two rounds of incremental updates, the missed detections decreased to 41 and the false positives to 55. In the similarity gating correction, 96 abnormal markers were removed in the initial stage, and 88 were consistent; after the incremental update, 83 abnormal markers were removed, and 79 were consistent, increasing the number of consistent records from 1984 to 2146.

[0149] Table 1 Summary of Anomaly Detection Results

[0150]

[0151] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A high-diamagnetic passbook printing anomaly detection system based on a large model, characterized in that, include: The data acquisition and processing module is used to collect the job data of high antimagnetic passbook printing and preprocess it to form a standard dataset; The structure construction module is used to extract the row and column boundary intervals of each field in the standard dataset. It employs an interval tree algorithm to perform interval matching and boundary concatenation on these intervals, and then performs field-level structure encoding on the standard dataset based on the matching and concatenation results to construct a training sample set, including: Based on the image records in the standard dataset, the pixel density projection distribution is calculated along the row and column directions respectively, and the row and column boundary intervals are obtained by segmenting according to the peak and valley changes of the projection curve. The interval tree algorithm is used to perform interval insertion and interval retrieval on row and column boundary intervals. A balanced interval tree node structure is constructed based on the start and end coordinates of the intervals on the row and column boundary intervals. The maximum ending coordinate of the interval set is maintained in the node, and non-overlapping search paths are pruned based on the maximum ending coordinate. Using row boundary intervals as query intervals, interval overlap calculations are performed on the interval tree. The retrieval space is reduced and candidate column boundary intervals are obtained through the maximum termination coordinate pruning strategy. Calculate the interval overlap metric between the candidate column boundary interval and the corresponding row boundary interval. The metric includes the overlap length ratio and the interval coverage index. Based on the overlap metric, interval stability determination conditions are constructed, and row and column intervals that meet the constraints are selected according to the stability determination conditions. Based on the row and column intervals that satisfy stability constraints, a hierarchical clustering algorithm is used to aggregate the endpoints of the intervals, and multi-interval fusion calculation is performed based on the aggregation results to generate the field boundary range. Based on the field boundary range, the image records in the standard dataset are cropped at the field level, and the field order encoding and field label alignment are performed according to the field arrangement relationship to construct a training sample set; The model fine-tuning module is used to fine-tune the multimodal visual language large model LLaVA based on the training sample set. It groups the model parameters and allocates different learning rates according to the learning rate scheduling rules, and performs gradient updates on the anomaly discrimination parameter group to obtain the print anomaly discrimination model. The anomaly diagnosis module uses the Hungarian matching algorithm to perform field index alignment and multimodal sequence assembly on real-time printing jobs, and inputs the assembled multimodal inference sequence into the printing anomaly discrimination model for constraint decoding to generate anomaly diagnosis results. The result verification module is used to locate the field image region corresponding to the abnormal field based on the field index alignment result, use the normalized cross-correlation algorithm to measure similarity, and correct the abnormal diagnosis result according to the preset threshold. The feedback update module is used to collect and process feedback data and incrementally update the printing anomaly detection model.

2. The high-diamagnetic passbook printing anomaly detection system based on a large model according to claim 1, characterized in that, The data acquisition and processing module specifically includes: The collected job data is merged and organized according to the job identifier. The merged job data is parsed for log fields and unstructured logs are converted into structured event fields. The images in the job data are subjected to size normalization, orientation correction and grayscale enhancement. The printing instructions in the job data are subjected to parameter decomposition and field normalization. The template records in the job data are subjected to version parsing and field alignment to form a standard dataset.

3. The high-diamagnetic passbook printing anomaly detection system based on a large model according to claim 1, characterized in that, The model fine-tuning module specifically includes: The training sample set is divided into several training subsets according to the anomaly category and field type, and a multimodal training sequence is constructed for each training subset; Based on the multimodal training sequence, calibration batches are selected, and forward computation and backward gradient sampling are performed on the multimodal visual language large model LLaVA. The gradient sensitivity index of each parameter on the calibration batch is statistically analyzed, and the model parameters are segmented by threshold based on the gradient sensitivity index to form a basic parameter group and an anomaly discrimination parameter group. Configure group learning rate scheduling rules for each parameter group, use cosine annealing learning rate scheduling to update the learning rate round by round, and configure an upper limit value for the learning rate for the anomaly discrimination parameter group. The group learning rate scheduling rules include applying an upper limit pruning constraint to the learning rate of the anomaly discrimination parameter group and applying a scaling constraint to the learning rate of the other parameter groups. The multimodal training sequences are extracted in batches and input into the multimodal visual language large model LLaVA to perform forward computation; For each batch, resampling or weighted sampling is performed according to the distribution of anomaly categories, and the hard cases are weighted in the loss calculation; Based on the anomaly category label, construct the Focal Loss function and perform backpropagation to calculate the gradient; The gradient is subjected to norm clipping and outlier gradient masking, and the AdamW optimization algorithm is used to introduce a weight decay term to perform parameter updates; Based on the group learning rate scheduling rules, batch learning rates are applied to the anomaly discrimination parameter group and the other parameter groups respectively. Parameter updates are performed on the anomaly discrimination parameter group and update magnitude constraints are applied to the other parameter groups. After each training round, the continuation and termination of training iterations are controlled based on the trend of loss changes and the preset early stopping criteria. After the predetermined number of iterations, the model weight state is saved to obtain the printed anomaly detection model.

4. The high-diamagnetic passbook printing anomaly detection system based on a large model according to claim 3, characterized in that, The formation of the basic parameter set and the anomaly discrimination parameter set specifically includes: Calibration batches are extracted from the multimodal training sequence, and forward and backward propagation calculations are performed on the calibration batches in the multimodal visual language large model LLaVA to obtain the gradient sampling results of each parameter on the calibration batches. The gradient sampling results are normalized by parameter scale, and a gradient sensitivity index is constructed based on the normalized gradient magnitude statistics. Perform quantile threshold estimation on the gradient sensitivity index to determine the sensitivity threshold and generate threshold segmentation conditions; The model parameters are filtered and grouped according to the threshold segmentation condition. Parameters with gradient sensitivity higher than the sensitivity threshold are classified into the anomaly discrimination parameter group, and parameters with gradient sensitivity lower than the sensitivity threshold are classified into the basic parameter group. Repeat the gradient sensitivity index calculation and statistical analysis of the group consistency ratio under different calibration batches. Parameters with a consistency ratio lower than the preset threshold are reassigned, and the parameter grouping results are output.

5. The high-diamagnetic passbook printing anomaly detection system based on a large model according to claim 1, characterized in that, The anomaly diagnosis module specifically includes: Adaptive threshold segmentation and morphological closing operations are performed on the layout image of the real-time printing job. The bounding rectangles of the text connected regions are extracted using the connected component labeling algorithm. Adjacent bounding rectangles are merged according to the spacing threshold to generate a set of field candidate boxes. Field identifiers are mapped to field indices in a preset order, a comprehensive cost matrix of field indices and field candidate boxes is constructed, matching pairs with cost values ​​exceeding a threshold are gated and eliminated, and the Hungarian matching algorithm is used to solve the correspondence between field indices and field candidate boxes. Based on the correspondence, the candidate field boxes are cropped and their size is normalized. The cropping results are then assembled into a model input sequence with the corresponding field identifiers according to the field index order. Based on the model input sequence, the printed anomaly discrimination model is used to perform constraint decoding, which limits the output field position according to the field index order, and masks the decoding results that do not meet the field position order constraint, thereby generating anomaly diagnosis results.

6. The high-diamagnetic passbook printing anomaly detection system based on a large model according to claim 5, characterized in that, The specific steps for determining the correspondence between the field index and the field candidate box include: Perform sequential encoding on field identifiers according to a preset order, and map the sequential encoding results to field indexes; The Mahalanobis distance algorithm is used to calculate the center offset distance between the field index and the field candidate box, and the width and height difference of the candidate box is calculated. The center offset distance and the width and height difference are normalized and then weighted to synthesize the matching cost. A comprehensive cost matrix is ​​constructed based on the matching cost and the field candidate box set. Matrix elements with matching costs exceeding a threshold are marked as unselectable, forming a gating constraint. Under gating constraints, Hungarian matching is performed on the comprehensive cost matrix, and row and column reductions are performed on the comprehensive cost matrix to eliminate the minimum row cost and minimum column cost, resulting in a reduction matrix containing zero cost elements. Perform a minimum line cover operation on the zero-cost elements in the reduction matrix to calculate the minimum number of rows and columns required to cover all zero-cost elements. When the minimum number of lines covered is less than the number of field indexes, select the minimum cost of the elements not covered by lines and perform subtraction update on the uncovered elements, and perform addition compensation update on the cross-coverage positions to update the distribution of zero-cost elements in the reduction matrix. Repeat the minimum line coverage operation and matrix update until the minimum line coverage reaches the field index number to obtain a zero-cost structure matrix; Assignment selection is performed based on a zero-cost structure matrix, and the correspondence between the output field index and the field candidate box is generated.

7. The high-diamagnetic passbook printing anomaly detection system based on a large model according to claim 1, characterized in that, The result verification module specifically includes: Based on the correspondence between field indexes and field candidate boxes, the field indexes marked as abnormal in the anomaly diagnosis results are selected, and the cropping boundaries of the corresponding field candidate boxes are determined. The field image regions are then cropped from the layout image. Based on the field image region and the standard field image corresponding to the same field index in the standard dataset, size normalization and grayscale normalization are performed, and the phase correlation algorithm is used to estimate the translation offset between the two. The field image region is then aligned based on the translation offset. The phase correlation algorithm is used to estimate the translation offset between the two images in the frequency domain based on phase information. The similarity between the aligned field image region and the standard field image is calculated using a normalized cross-correlation algorithm to obtain the field similarity results. Threshold gating is applied to the field similarity results, and the field indexes with similarity results below the threshold are retained as abnormal status markers to obtain the corrected abnormal diagnosis results.

8. The high-diamagnetic passbook printing anomaly detection system based on a large model according to claim 1, characterized in that, The feedback update module specifically includes: Collect feedback data from the execution of anomaly diagnosis results, and associate the feedback data with the field index, anomaly category label, and corrected anomaly diagnosis results of the corresponding job to construct an incremental sample set; Based on incremental samples, anomaly confidence scores are obtained by the printing anomaly discrimination model, and the anomaly confidence scores are probabilistically calibrated using a temperature scaling algorithm. Based on the calibrated confidence scores, sample confidence weights are generated and updated samples are obtained through screening. Weighted incremental training is performed on the updated samples according to the sample confidence weights. The RMSProp optimization algorithm is used to perform gradient updates on the anomaly discrimination parameter group, and update magnitude constraints are applied to the other parameter groups. During incremental training, replay sampling is performed on the updated samples, and historical samples are mixed with the updated samples according to a preset ratio to form training batches. Iterative training is then performed on the training batches. After the predetermined incremental update rounds are completed, the model weight state is saved, and the incremental update of the print anomaly detection model is completed.

Images