Seal verification method and system based on visual language large model and medium

Abstract

The invention relates to the technical field of image processing and artificial intelligence, in particular to a seal verification method and system based on a visual language large model (VL model for short) and a medium. The method comprises the following steps: firstly, acquiring a reserved seal image and a to-be-verified seal image, and performing image registration and matching to generate a coincidence graph; then, constructing a data set containing a large number of coincidence graphs and corresponding discrimination results and natural language reason descriptions thereof; using the data set to train a visual language large model, so that the visual language large model learns a mapping relation between visual features of the seal coincidence graph and discrimination logic and text description; in the prediction stage, a to-be-verified coincident graph is input into a trained VL large model, the model can directly output judgment results including verification pass, suspicious or rejection in an end-to-end mode, and detailed judgment reasons are generated in a natural language mode. The problems that a traditional verification method is low in automation degree, depends on manual rules, lacks interpretability and the like are effectively solved, and verification accuracy, verification efficiency and user experience are improved.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of computer vision, natural language processing and artificial intelligence, and specifically relates to a method, system and medium for applying a large visual language model to the automated verification of seals in scenarios such as finance and government affairs. Background Technology

[0002] For banks, businesses, and government agencies, seal verification is a crucial security step when processing bills, contracts, and documents. Traditional seal verification methods mainly fall into two categories: one is manual visual comparison, which is inefficient, subjective, and prone to misjudgment due to fatigue; the other is automated seal verification systems based on traditional image processing, which typically extract the seal's outline, geometric features, and texture, and then calculate its similarity with pre-registered seals.

[0003] However, traditional automated methods have significant limitations: First, they heavily rely on carefully designed feature extraction rules and thresholds, making them less robust to noise, slight deformations, and uneven stamping pressure. Second, when verification fails, the system typically only provides a similarity score or a simple error code, unable to offer intuitive and explainable natural language reasons like "the seal border is broken," "the text is blurry," or "the pattern is misaligned," as human experts do. This "black box" decision-making process makes it difficult for reviewers to quickly pinpoint problems, reducing the efficiency and reliability of business processes.

[0004] In recent years, large-scale visual language models have demonstrated powerful capabilities in tasks such as image description and visual question answering, simultaneously understanding image content and generating fluent natural language. However, there is currently no solution to apply this technology to the field of seal verification to address its poor interpretability and insufficient automation. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, system and medium for seal verification based on a large visual language model that is highly accurate, interpretable and automated.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for verifying the authenticity of seals based on a large visual language model, comprising:

[0008] Step 1: Seal Matching and Overlap Image Generation: Obtain the unit's reserved seal image and the image of the seal to be verified. Accurately register the two using image processing techniques (such as feature point matching, affine transformation, etc.), and then generate an "overlap image" that clearly reflects the differences between the two through image differencing, color channel overlay, etc.

[0009] Step 2: Training Data Preparation: Construct a high-quality labeled dataset. This dataset consists of a large number of overlapping image samples. Each sample is labeled by experts with the final judgment result ("Pass", "Suspicious", "Reject") and a detailed explanation of the reason. The explanation of the reason is in natural language, for example: "Seal verification passed, the seal outline matches well, and there is no obvious difference"; "Seal verification rejected, the text in the lower right corner of the seal is missing, and the overall position is seriously offset"; "Seal verification suspicious, the seal texture is slightly blurry, manual review is recommended."

[0010] Step 3: Large-Scale Visual Language Model Training: Select a pre-trained large-scale visual language model (such as BLIP-2, GPT-4V, or their variants) as the base model. Fine-tune this model using the dataset prepared in Step 2. The training objective is to correlate the visual features of overlapping images with discriminative labels and natural language reasons, enabling the model to learn to make professional judgments and generate explanations based on image content.

[0011] Step 4, Imprint Prediction: In practical applications, the newly generated true coincidence image to be verified is input into the trained VL large model. The model will output a comprehensive natural language statement end-to-end, which includes both the judgment conclusion and a detailed description of the reasons supporting that conclusion.

[0012] In a second aspect, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect above.

[0013] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0014] The beneficial effects of this invention include:

[0015] 1. Excellent interpretability: This invention introduces the VL large model into the field of seal verification for the first time. By utilizing its powerful visual understanding and language generation capabilities, it provides clear and easy-to-understand natural language explanations for each seal verification result, which greatly facilitates business personnel in reviewing and making decisions.

[0016] 2. High accuracy and robustness: The VL large model can learn complex patterns and contextual information of seal differences from massive amounts of data. Its judgment ability far exceeds that of traditional methods based on fixed rules, and it is more intelligent and robust in handling noise, deformation and other situations.

[0017] 3. End-to-end automated process: The traditional multiple steps of "feature extraction - similarity calculation - threshold judgment" are simplified into an end-to-end process of "overlapping graph - VL large model - result and cause", which simplifies the system architecture and improves processing efficiency.

[0018] 4. Excellent user experience: The output natural language description conforms to human cognitive habits, making the automated system no longer a "black box" and improving users' trust in and experience of the automated seal verification system. Attached Figure Description

[0019] Figure 1 The flowchart shows the overall process of the seal verification method based on a large visual language model provided in this embodiment of the invention.

[0020] Figure 2 This is a schematic diagram of the generated superimposed image in the embodiment, where (a) is a reserved seal image, (b) is a seal image to be verified, and (c) is an example of the generated superimposed image.

[0021] Figure 3 This is a schematic diagram illustrating the principle of the VL large model training phase.

[0022] Figure 4 This is a schematic diagram illustrating the application scenario of the seal prediction stage. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0024] Example 1:

[0025] refer to Figure 1 This embodiment provides a method for verifying seals based on a large visual language model. The specific steps are as follows:

[0026] 1. Generation of coincident graphs: - Collect high-resolution images of the reserved seal and the seal to be verified. - Perform image preprocessing, including grayscale conversion, binarization, and noise reduction. - Use algorithms such as SIFT or ORB to extract feature points and perform matching, calculate the optimal affine transformation matrix using the RANSAC algorithm, and align the image to be verified with the pre-reserved seal image. - Perform a difference operation on the two aligned images to obtain a difference image. Then, overlay the reserved seal (represented in red), the seal to be verified (represented in green), and the difference area (represented in highlight color) onto a single image to generate the final superimposed image, as shown below. Figure 2 As shown.

[0027] 2. Dataset Construction and Model Training: - Collect tens of thousands of overlapping images covering various situations (clear, blurred, offset, incomplete, forged, etc.). - Hire domain experts to annotate each overlapping image. The annotation content is structured text, in a format such as: "<Conclusion> Seal Verification Rejected< / Conclusion><Reason> There are obvious misalignments at the edges of the seal, and there is a ghosting phenomenon in the middle text.< / Reason>". - Select the open-source QwenVL series large model as the basic VL large model. It consists of a visual encoder (ViT), Querying Transformer, and a large language model (LLM). - Input the overlapping image into the visual encoder to obtain image features, extract the visual features most relevant to the text through Q-Former, and input them into the LLM. During training, set the prompt as "Please verify the authenticity based on this overlapping seal image. The conclusion must be one of 'Seal Verification Passed', 'Seal Verification Suspicious', or 'Seal Verification Rejected', and give a detailed reason:", and use the annotated text as the training target. Minimize the loss function of the language model to fine-tune the entire model parameters, enabling it to learn to generate text that meets the requirements according to the image. The training process is shown in Figure 3 .

[0028] 3. Prediction and Deployment: - After training, deploy the model to the seal verification server. - When there is a new seal verification request, the system automatically generates an overlapping image and inputs it into the deployed VL large model. - The model will generate text similar to the following: "Seal Verification Suspicious. The seal to be verified basically matches the reserved seal in the overall outline, but there is a slight offset of about 1 mm in the lower left corner area, and there is a slight bleeding of the printing ink. It is recommended to have a final review by an artificial operator." As Figure 4 shown. - This system can be integrated into the business systems of banks or financial companies to provide efficient decision-making support for counter or back-office auditors.

[0029] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered within the protection scope of the present invention.

Claims

1. A method for verifying the authenticity of seals based on a large visual language model, characterized in that, Includes the following steps: S1: Seal matching and superimposed image generation steps: Obtain the reserved seal image and the image to be verified, perform registration and matching processing on the two images, and generate a superimposed image to display the differences between the two. S2: Training data preparation steps: Construct a training dataset, which contains multiple training samples. Each training sample includes an overlay image, a label of the discrimination result corresponding to the overlay image, and a reason text described in natural language. S3: VL Large Model Training Steps: Use the training dataset to train the visual language large model to learn the mapping relationship from overlapping images to discrimination results and cause descriptions; S4: Seal prediction step: Input the superimposed image of the seal to be verified into the trained VL large model, and the VL large model simultaneously outputs the discrimination result and the reason in natural language.

2. The method according to claim 1, characterized in that, In step S1, the method for generating the superimposed image includes, but is not limited to, image difference method, color overlay method, contour overlay method or a combination thereof. The superimposed image can intuitively show the differences between the reserved seal and the seal to be verified in terms of shape, texture, position or contour.

3. The method according to claim 1, characterized in that, The discrimination result labels include three categories: "pass", "suspicious", and "reject"; the reason text is a natural language description of the image region, difference type, or matching degree that led to the discrimination result.

4. The method according to claim 1, characterized in that, In step S3, the large visual language model used is based on The pre-trained model of the Transformer architecture is trained with a multi-task learning objective, including at least one task for classifying and judging the results and one task for generating causal text.

5. The method according to claim 4, characterized in that, In step S3, when training the VL large model, cueing engineering is used to integrate the discrimination task and the cause generation task into a unified text generation framework. The output of the model is structured natural language text, which contains both the discrimination result and the cause description.

6. The method according to claim 1, characterized in that, In step S4, the discrimination results and cause descriptions output by the VL large model are further used to generate a visualized verification report, which displays the overlap diagram, discrimination results, and cause descriptions together.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the seal verification method based on a large visual language model as described in any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the imprint authentication method based on a large visual language model as described in any one of claims 1 to 6.

Images