Tobacco file examination method and system based on visual-language large model

Abstract

The invention provides a tobacco file examination method and system based on a visual-language large model, and the method integrates multi-modal feature extraction, cross-file content association, a rule introduction mechanism based on the large model, and an evidence traceability mechanism based on the large model. The method comprises the following steps of: performing unified analysis, element extraction, signature normative detection and regular compliance examination on electronic files and paper scanning images including catalogs, attachments, photos and the like for case file materials of cases which have many cases in the tobacco industry and are not purchased in a specified tobacco enterprise; and outputting the results of whether the file archive is complete, whether the signature is correct, conclusion and evidence positioning and the like. In combination with a Prompt dynamic instruction and a rule knowledge base, flexible introduction of specifications and standards during case auditing in the industry is realized, and an automatic, interpretable and traceable file auditing task is completed.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of tobacco data review, and particularly relates to a tobacco file review method and system based on a visual-language large model. BACKGROUND

[0002] In various administrative management, license approval and operation supervision processes in the tobacco industry, file review has always been an important link to ensure business compliance and integrity. Files usually cover subject qualification materials, license application and reply documents, contracts, image scans and related supporting materials. The content is diverse, the structure is complex, and there are different format requirements between different business lines. With the continuous improvement of the informatization level of the industry, files have gradually changed from traditional paper archives to electronic documents and hybrid archives, which has improved the convenience of data storage and transmission to a certain extent, but has also brought new challenges to the review process. Reviewers not only need to check the integrity and authenticity of the files, but also need to ensure the consistency of the fields between different files, the compliance and standardization of the seals, and quickly and accurately complete the comparison and verification among materials of different sources, different formats and different clarity. This process is highly professional, relies on a large number of industry system specifications, and involves paper scans, electronic documents and image materials and other media. Manual review is often time-consuming and labor-intensive, and is easily affected by subjective factors, resulting in omissions, and is difficult to meet the efficient, traceable and standardized business needs.

[0003] To address the inefficiency and high cost of manual review, existing technical solutions mainly rely on OCR (Optical Character Recognition) technology to recognize the text of electronic documents and scans, and implement semi-automatic file review by combining process systems and developing pre-defined logic processes. The typical approach is to extract the text content of the file through OCR technology, then check the key fields such as license number, subject information, validity period and attachment list by the built-in rules or pre-defined logic processes of the system, and output the review conclusion on this basis. However, the existing solution highly depends on the recognition accuracy of OCR, and when faced with low-resolution scans, complex layouts, handwritten signatures or notched seals, the recognition results are often not stable, which affects the reliability of the review conclusion. In addition, such systems usually take fixed processes and rules as the core, and once new compliance requirements or business review dimensions appear, the corresponding function modules or rule logic often need to be redeveloped, which has a long development cycle and insufficient system flexibility, making it difficult to adapt to the changing industry regulatory policies and business needs. Especially in cases where comprehensive analysis across files and templates is required, or evidence tracing and interpretable output is required, the traditional OCR+rules system is not up to the task, and cannot guarantee the accuracy of the review while considering the scalability and maintainability, which has become a major technical bottleneck in the process of industry intelligent transformation.

[0004] In summary, the prior art has the following defects: (1) The file review system based on OCR and rule engine has deficiencies in recognition accuracy, especially in low-definition scanned documents, non-standard templates, handwritten signatures, and cross-page seal complex scenarios. The OCR recognition results are prone to deviation, directly affecting the reliability of subsequent field extraction and review conclusions.

[0005] (2) Existing solutions usually use pre-defined templates and fixed rules for review, lacking flexibility and adaptability. When the file template is updated or the review rules are added, related functions must be redeveloped or configured, resulting in high system maintenance costs, long development cycles, and difficulty in quickly responding to changes in industry policies and business needs.

[0006] (3) Traditional solutions mainly implement field extraction and rule comparison within a single file, lacking cross-file, cross-page semantic understanding and correlation analysis capabilities, making it difficult to handle subject information consistency, seal standardization, and other review tasks that require multi-file comprehensive reasoning. Manual review is time-consuming, and the degree of automation is limited.

[0007] (4) In terms of review conclusion explainability, existing systems lack evidence positioning and backtracking functions for review evidence, and cannot clearly display the source of the review conclusion and the triggering rules, limiting the transparency and credibility of the review process in compliance auditing, responsibility tracing, and manual review. SUMMARY

[0008] To solve the above problems in the prior art, the present application provides a tobacco file review method and system based on a visual-language large model. The technical problems to be solved by the present application are solved by the following technical solutions: A tobacco file review method based on a visual-language large model includes: S100, determining the review object and the review range according to the review requirements of the tobacco file; wherein the review object is all documents of the tobacco file, and the review range is the content points in the documents; S200, reading in the review object and pre-processing it to obtain a pre-processed review object; S300, introducing clear regulation rules and empirical rules of the tobacco industry into the automated review process to form a synergistic mechanism; S400, modeling the pre-processed review object to obtain a global representation of the review object, extracting key information elements from the global representation, calling the clear regulation rules and empirical rules to automatically review the key information elements to obtain a review result, and determining the compliance of the review result to obtain a final compliance determination result and the corresponding evidence source; S500, performing standardization detection on the signature and the signature in the key information element to obtain a final standardization detection result; S600, integrating the final compliance determination result and the final standardization detection result to output a compliance conclusion, an evidence position and a confidence score of each signature and signature area; S700, manually reviewing the conclusion with a confidence score lower than a confidence threshold, and adjusting part of the automatic review process according to the manual review result.

[0009] A tobacco file review system based on a visual-linguistic large model comprises: A determination module configured to determine a review object and a review range according to a review requirement of a tobacco file, wherein the review object is all documents of the tobacco file, and the review range is content points in the documents; A reading-in module configured to read in the review object and pre-process the review object to obtain a pre-processed review object; A rule module configured to introduce clear regulation rules and experience rules of the tobacco industry in an automatic review process to form a synergistic mechanism; A review module configured to model features of the pre-processed review object to obtain a global representation of the review object, extract key information elements from the global representation, call the clear regulation rules and the experience rules to automatically review the key information elements to obtain a review result, and perform compliance determination on the review result to obtain a final compliance determination result and a corresponding evidence source; A detection module configured to perform standardization detection on the signature and the signature in the key information element to obtain a final standardization detection result; An integration module configured to integrate the final compliance determination result and the final standardization detection result to output a compliance conclusion, an evidence position and a confidence score of each signature and signature area; A review module configured to manually review the conclusion with a confidence score lower than a confidence threshold, and adjust part of the automatic review process according to the manual review result.

[0010] An electronic device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus; The memory is used to store a computer program; The processor is used to execute the program stored on the memory to realize the steps of the tobacco file review method based on the visual-linguistic large model.

[0011] Advantages: 1.The application improves the automation and accuracy of file review. By introducing a locally deployed visual language large model in the tobacco industry file review, the application can handle both electronic and scanned paper files, achieving integrated review across media and documents, significantly reducing the workload of manual page-by-page comparison, and improving the automation level and accuracy of the review results.

[0012] 2.The application enhances the generality and adaptability of file review. The application uses a multi-modal element extraction and semantic association method, which can automatically adapt to files of different formats and sources. When facing template updates or business rule changes, only a prompt statement or rule knowledge base needs to be dynamically adjusted, without the need to redevelop system modules, making the system more general and expandable.

[0013] 3.The application improves the traceability and compliance of file review. The application generates corresponding text segments, image regions or table cells as evidence while outputting the review conclusion, supporting the explainability and compliance of the review process, and meeting the strict requirements of the industry in data security, responsibility tracing and regulatory audit.

[0014] 4.The application reduces system maintenance and labor costs. Through local deployment and automated processing, the application reduces the dependence on manual review, shortens the business processing cycle, and reduces the secondary development and maintenance costs caused by frequent template changes.

[0015] The application will be further described in detail below in conjunction with the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a flowchart of the file automatic review method provided by the application, which proposes a file automatic review method based on a visual-language large model; Figure 2 is a main process diagram of the file automatic review method provided by the application. DETAILED DESCRIPTION

[0017] The application will be further described in detail below in conjunction with the accompanying drawings and embodiments.

[0018] To solve the technical problems existing in the prior art, the application provides a file automatic review method and system based on a visual-language large model, which fuses multi-modal feature extraction, cross-file content association, a large model-based rule introduction mechanism, and a large model-based evidence traceability mechanism, realizes unified analysis of electronic and paper versions of files, signature standardization detection, compliance review, and result output, thereby significantly improving the automation level, accuracy, and processing efficiency of file review under the premise of ensuring privacy security and compliance, reducing the artificial burden and operation and maintenance cost, and promoting the intelligent and standardized development of file management.

[0019] The embodiment of the application provides a file automatic review method and system based on a visual-language large model. The embodiment takes a domestic Qwen2.5-VL visual language model as a base model, and uniformly analyzes, element extracts, signature standardization detects, and rule-compliant reviews electronic files and paper scanning images containing a directory, an attachment, a photo and the like, and outputs whether a file archive is complete, whether a signature and a seal are correct, a conclusion and evidence positioning and the like. In combination with a Prompt dynamic instruction and a rule knowledge base, the embodiment realizes flexible introduction of standards and norms when the tobacco industry audits the case, and completes an automatic, interpretable and traceable file review task.

[0020] In a first aspect, as Figure 1 indicated, the application provides a tobacco file review method based on a visual-language large model, which includes the following steps: S100, determining a review object and a review range according to a review requirement of a tobacco file; wherein the review object is all documents of the tobacco file, and the review range is content points in the documents; For a case that does not purchase according to a regulation, the application first needs to determine a review object and a range. First, read case information, confirm all electronic documents and paper scanning files involved in the case under a standard process, and automatically match a corresponding review template in combination with relevant regulations of a tobacco monopoly administrative punishment document and a business type. The template predefines completeness, necessity, relevance and detail compliance requirements of a case archive, ensures that a subsequent review process is orderly, a review content coverage is complete, and a processing result has traceability and standardization.

[0021] S200, reading in the review object and pre-processing the review object to obtain a pre-processed review object; After determining the case file material catalog, document relationships, and key points for detailed review, this embodiment of the application proceeds to the data access and preprocessing stage. This step aims to uniformly parse and standardize electronic documents and scanned paper files in the case file, providing high-quality structured data input for subsequent multimodal information extraction, rule matching, and compliance review. Through data access and preprocessing, it can be ensured that document data from different sources, in different formats, and with different resolutions all meet unified specifications and format requirements before entering the intelligent review model, reducing the impact of data heterogeneity on review accuracy.

[0022] S300 introduces both explicit rules and empirical rules from the tobacco industry into the automated review process to create a collaborative mechanism between the two. After confirming the case file material catalog and preprocessing the data, this embodiment of the application enters the industry rule introduction stage. The core objective of this step is to introduce existing empirical rules and explicitly stipulated rules into the automated review process based on a visual language big data model through Prompt dynamic instructions and knowledge base retrieval mechanisms, respectively. This enables the model to follow established industry norms and legal requirements when parsing case file content, ensuring the integrity, accuracy, and traceability of the review process.

[0023] S400, perform feature modeling on the preprocessed review object to obtain a global representation of the review object, extract key information elements from the global representation; call the explicit rules and empirical rules to automatically review the key information elements to obtain a review result; perform compliance determination on the review result to obtain a final compliance determination result; After completing data access, preprocessing, and the introduction of industry rules, this application enters the multimodal information extraction and intelligent compliance review stage. This stage relies on the domestically developed Qwen2.5-VL large-scale model to uniformly model and fuse features of multimodal data such as text content, page structure, and image information in the case file, automatically parsing key information such as the subject name, penalty amount, license number, penalty basis, and attachment list. While extracting elements, this application performs cross-document and cross-page consistency comparisons to ensure that fields such as the subject, amount, and basis remain consistent across different documents. Combining previously introduced empirical rules and explicit regulations, this application triggers empirical rules to judge the document structure and attachment integrity through the Prompt command, performs compliance judgments of laws and standards through knowledge base retrieval, automatically outputs each review conclusion and the corresponding evidence location, and triggers a manual review process when low-confidence or conflicting conclusions are reached, achieving a collaborative approach between automated review and manual confirmation.

[0024] S500, Perform a standardization check on the seals and signatures in the key information elements to obtain the final standardization check result; After the intelligent compliance review is completed, the application will perform special normative detection on the seal and signature elements in the case file. In this stage, the application first automatically locates all the seal, signature and embossed seal areas in the document, extracts the area image and text information, and performs quantitative analysis on the existence, integrity, clarity and cross-page continuity of the seal to confirm whether the seal is missing, damaged or discontinuous. For handwritten signatures, the application will manually review the low-confidence results based on automatic recognition to ensure the authenticity and legality of the signature. The detection results will be combined with the compliance review conclusion to form a normative judgment of the seal and signature and an evidence chain, ensuring the completeness of the administrative punishment document in terms of formal requirements.

[0025] S600, integrating the final compliance judgment result and the final normative detection result, outputting the compliance conclusion, evidence position and confidence score of each seal and signature area; After the existence, integrity, clarity, continuity and authenticity of the seal and signature are detected, the application will integrate all the detection results with the previous rule judgment results to output the compliance conclusion, evidence position and confidence score of each seal and signature area. All results will be recorded in a structured format to ensure the traceability and explainability of the review conclusion.

[0026] S700, manually reviewing the conclusions with a confidence score below the threshold value, and adjusting part of the automatic review process according to the manual review results.

[0027] This step mainly performs human-machine cooperation and continuous optimization.

[0028] As a specific embodiment of the application, combined with Figure 1 and Figure 2 , S100 includes: S110, obtaining case file material targets according to the review requirements of tobacco files, and reading and analyzing all documents in the case file according to the case file material targets, and determining all documents as review objects; After the case file materials are filed, there will be a directory file. First, read and analyze the directory file, automatically analyze all files in the case file, and compare them with the standard document list specified in the Tobacco Monopoly Administrative Punishment Document. Through comparison, the application can automatically distinguish which files belong to essential documents, such as administrative punishment decision, case investigation report, payment receipt, etc., and which files belong to non-essential documents, such as supplementary explanation, meeting records, etc.

[0029] S120, logically and business-relatedly associating the review objects to obtain associated review objects; On the basis of confirming the material catalog, the application will further analyze the logic and business association between each file in the case file, and determine which documents need to be jointly audited. For example, the penalty amount in the penalty decision needs to be consistent with the payment voucher, and the case and evidence materials in the case investigation report need to be cross-verified with the penalty basis. Through automatic document association analysis, the application can provide document-level task allocation basis for collaborative review, ensure that multiple auditors complete their respective review responsibilities, and avoid omissions and repetitions.

[0030] S130, in the content of the review object, determining the content points as the review range.

[0031] After confirming the document catalog and association relationship, the application will automatically focus on the detail compliance check of the document content. This includes automatically comparing the fine amount in the penalty decision to ensure that the value is consistent with the case facts and payment voucher; performing integrity detection on documents that must be signed or stamped to verify whether the signature or stamp exists, is clear, and the grommet stamp of cross-page documents is continuous. At the same time, in view of the problem that handwritten signatures may be difficult to identify, the application will automatically mark the signature content with low confidence, leaving a manual review link to ensure that the signature is legal and compliant. For the case and the penalty basis, the application will extract the relevant clauses to check whether they meet the requirements of industry regulations and management systems to ensure the legality and rationality of the penalty decision.

[0032] In one specific embodiment of the application, in combination with Figure 1 and Figure 2 , S200 includes: S210, if the review object is an electronic document, automatically parsing it to obtain a preprocessed review object; The application first automatically parses the electronic documents involved in the case. For common document types such as PDF and Word, the application extracts the text layer information, catalog bookmarks, vector objects, embedded images, and table cell structures in the document, and reconstructs the page level relationship and paragraph layout structure of the document. Table information will be output in the form of cell grid and coordinates to ensure accurate positioning to specific cells or page locations during subsequent element extraction.

[0033] S220, if the review object is a paper document, capturing an image of the paper document, and performing quality enhancement and geometric correction operations on the image to obtain a preprocessed review object.

[0034] For the scanned copy or photographic image of paper files, first perform image quality enhancement and geometric correction operations. Quality enhancement includes denoising, shadow removal, brightness and contrast equalization to ensure clear and readable key information such as text and signatures; geometric correction includes tilt correction, boundary cutting, etc. to ensure the standardization and comparability of page content in pixel space.

[0035] The present application proposes an integrated review scheme across media and documents in view of the actual situation of coexistence of electronic files (such as PDF, Word) and paper scanned files in the tobacco industry. The present application can simultaneously process files of different media sources, adopt a unified analysis and review process, and automatically complete layout structure recognition, field extraction, signature detection and compliance verification. By integrating electronic documents and paper images in the same technical framework, the problems of repeated development and information fragmentation caused by medium-specific processing in traditional schemes are avoided, and consistent review and unified output of results across media and documents are realized.

[0036] In a specific embodiment of the present application, in combination with Figure 1 and Figure 2 , S300 includes: S310, introducing empirical rules in the tobacco industry, and converting the empirical rules into empirical rule instruction templates using Prompt instructions; In the review process of files of administrative punishment cases in the tobacco industry, there are a large number of unstructured rules accumulated from long-term case handling experience, such as: which necessary files should be included in the files of a certain type of case; when a specific element appears in the punishment decision, is it necessary to supplement the relevant proof files; which pages must be signed or stamped, and which signature requirements must be in handwritten form; whether the names of the persons involved need to be kept completely consistent between different documents, etc. Such rules are often scattered in manual experience and internal case handling manuals and are difficult to directly apply in the form of fixed logic. The embodiments of the present application convert these empirical rules into executable instruction templates through the Prompt generation and injection mechanism, and input them into the visual language large model together with the file content during the review process. When analyzing the document content, the model can automatically determine whether the current document meets the conditions of the empirical rules according to the Prompt instruction requirements, and output structured compliance conclusions and evidence information. This mechanism not only improves the automatic execution capability of empirical rules, but also supports quick adjustment of Prompt templates after rule updates, without the need to redeploy or modify the model structure.

[0037] In the present application, industry rules are introduced into the code review process in the form of enhanced prompt or rule knowledge base retrieval to replace the executable logic. These alternatives can also introduce industry rules into executable logic in a similar manner. These alternatives can all complete the automatic execution of rules without changing the overall purpose of the invention.

[0038] S320, obtain the plaintext rules of the tobacco industry and constitute a rule knowledge base; In addition to the experience-based rules, the administrative punishment cases of the tobacco industry also involve a large number of explicit provisions based on laws and regulations and administrative systems, for example: when the amount involved reaches a certain numerical threshold, the amount of punishment should be calculated according to the statutory proportion; official receipts must be issued after the payment of fines; administrative approval opinions or law enforcement bases need to be supplemented in certain circumstances, etc. Such rules have clear logical conditions and legal binding force, and are suitable for management and calling using a structured knowledge base. The embodiment of the present application constructs a rule knowledge base for case files review, wherein each rule contains a trigger condition, an execution logic, required evidence and an applicable scope. During the review process, when the model detects elements matching the rule conditions, it will automatically initiate a query to the knowledge base, retrieve the corresponding review standards and processing requirements, and compare the retrieval results with the content of the document, and finally generate a compliance conclusion based on the regulations. This method ensures strict consistency between the review results and industry regulations, and supports dynamic expansion and version management of the rule base.

[0039] S330, introducing the experience-based rule template into the automated review process through the Prompt instruction, and introducing the rule knowledge base into the automated review process through knowledge base retrieval.

[0040] In the embodiment of the present application, experience-based rules and explicit provision rules are injected through two ways of Prompt instruction and knowledge base retrieval, but form a synergistic mechanism in actual execution: experience-based rules are responsible for quickly judging soft requirements such as document integrity, signature standardization, and name consistency, and explicit provision rules ensure that hard conditions such as penalty amount, penalty basis, and receipt issuance strictly meet the legal and regulatory standards. The synergistic execution of the two types of rules enables the present application to meet both business experience and regulatory requirements in a single review process, ensuring the comprehensiveness and authority of automated review.

[0041] In a specific embodiment of the present application, in combination with Figure 1 and Figure 2 S400 includes: S410, all documents in the tobacco case files are sent to the Qwen2.5-VL large model in a multi-modal form, so that the Qwen2.5-VL large model establishes a global representation of the content of the tobacco case files in a unified semantic space by feature encoding and cross-modal semantic alignment of text, layout, and image data; After completing the data access preprocessing and rule introduction, the application first sends all case file materials in a multi-modal form into Qwen2.5-VL, the input data including text content and its spatial location information in the document, layout structure features, table cell division, signature area image and attachment list image and other multi-modal information. The model establishes a global representation of the file content in a unified semantic space through feature encoding and cross-modal semantic alignment of text, layout and image data, so that subsequent element extraction and rule determination can be completed under the support of high-precision features.

[0042] The application realizes joint modeling and element extraction of multi-modal information of the file through the visual language large model, covering multiple types of data such as text, table, layout structure, signature image, etc. The model can establish semantic association in the multi-modal feature space and automatically complete cross-page and cross-document key information alignment and consistency checking according to prompt information (prompt).

[0043] The application can also be used for multi-modal feature extraction and semantic association based on the visual language large model. The alternative solution can replace the existing model with other types of multi-modal model, and then perform cross-modal feature fusion after encoding the text and image. These different types of models are consistent with the application in technical path and can complete the analysis and review of multi-modal information of the file.

[0044] S420, from the global representation, parse out key information elements, and detect the semantic and numerical consistency of the key information elements in all documents, if the consistency is abnormal, mark the corresponding key information elements and documents; After multi-modal feature modeling is completed, the application will automatically parse the key information elements in the file materials, including the subject name, unified social credit code, penalty amount, penalty basis, license number, validity period, business address, attachment list, signature and signature elements, etc. For the associated fields of multiple documents and multiple pages, the application will perform cross-document and cross-page consistency comparison to ensure that the core fields such as the subject name, penalty amount and penalty basis maintain semantic and numerical consistency in all documents. When detecting missing, contradiction or abnormality, the application will automatically mark the suspicious items for subsequent manual review or rule engine processing.

[0045] S430, calling the plaintext specified rules and empirical rules, automatically reviewing the key information elements of consistency to obtain compliance review results; After the completion of element extraction and consistency comparison, the application automatically calls the experience rules and clear rules introduced earlier. For experience rules, the application triggers through Prompt dynamic instructions. When specific elements or semantic patterns appear in the file content, the application automatically determines whether the file structure and content meet the experience requirements, such as whether the attachment list is complete, whether the names of the parties involved are consistent, whether the signatures cover all necessary documents, etc. For clear rules, the application retrieves relevant regulations and penalties from the rule knowledge base. For example, if the amount involved reaches the specified threshold, whether the penalty amount meets the proportion calculation, whether a legal invoice is issued after the fine, whether the penalty basis corresponds to the regulations, etc., to ensure that the judgment result strictly follows the industry regulations.

[0046] S440, compliance judgment is made on the compliance review results and corresponding key information elements, and a pass, fail or doubtful compliance judgment result is obtained, and evidence sources are bound for each compliance judgment result; After rule judgment execution, the application integrates multi-modal information and rule execution results to make compliance judgments on each element, output pass, fail or doubtful conclusions, and bind evidence sources for each conclusion, including page numbers, layout coordinates and image area information of the text content. All results are output in a structured format, supporting JSON file docking and visual marking in the original document for intuitive checking and review by subsequent personnel.

[0047] S450, manual review and feedback writing are performed on the compliance judgment results to obtain the final compliance judgment results.

[0048] When the confidence of the compliance judgment result is low or there is a conflict between the rules, the application automatically triggers the manual review link. The conclusion of manual review can be directly written back to the rule knowledge base or Prompt template, which is used to update the experience rules or optimize the Prompt instructions, so that the subsequent review process is continuously improved in the feedback of review, and the accuracy and stability of automatic review are improved.

[0049] The embodiment introduces a visual language large model supporting local deployment in the tobacco industry file review, which can run in the enterprise or industry special network environment, directly analyze and review file data intelligently, without relying on cloud services, avoiding the risk of sensitive data transmission. Under the premise of local deployment, the model can use the computing power resources within the enterprise to process electronic and paper files uniformly, combined with security control, log tracking and data desensitization mechanism, to ensure that the review process is completed under the conditions of high security and high compliance.

[0050] In one specific embodiment of the application, the combination of Figure 1 and Figure 2S500 includes: S510, positioning the location area of the signature and seal in the key information element, judging whether there is corresponding seal and / or signature content in the location area, and if so, extracting the seal image and / or signature content from the location area; After completing the multi-modal information extraction and compliance review, the present application will combine the layout structure analysis results and the detection ability of the visual language large model to automatically locate all areas related to seals and signatures in the file materials, record the page number, coordinates and layout index information, and extract the image and text content in the area for subsequent standard detection.

[0051] S520, integrity verification is performed on the seal image, if the verification fails, it is marked as abnormal, and if the verification passes, the continuity and authenticity of the seal are verified to obtain the final standard detection result of the seal; After extracting the seal area, the present application will first confirm whether the seal exists, and analyze the integrity of the seal using image segmentation and contour detection methods, calculate the seal boundary closure degree, area coverage ratio and continuity of cross-page saddle seal, to ensure the integrity of the seal in space and the continuity in logic. For missing, damaged or cross-page discontinuous seals, the present application will automatically mark them as non-compliant or suspicious.

[0052] S530, the clarity and legibility of the verified seal image are evaluated to obtain an evaluation result, and the seal image with clarity and legibility meeting the requirements is obtained through image enhancement according to the evaluation result; The present application further evaluates the clarity and legibility of the seal area, quantifies the imaging quality of the seal image using Laplacian variance, image contrast enhancement and other methods to ensure that the seal content is clear and visible, and the contour boundary is complete. In the cross-page saddle seal detection, the present application will perform cross-page feature point matching and geometric alignment to verify the continuity and authenticity of the cross-page seal, and prevent the occurrence of forgery and tampering.

[0053] S540, the signature content is recognized to obtain a recognition result, if the confidence of the recognition result is lower than a confidence threshold, the signature content is manually reviewed, and if the confidence of the recognition result is not lower than the threshold, the final standard detection result of the signature is obtained.

[0054] In the aspect of signature detection, the present application will confirm according to the file template and rule requirements which documents must have handwritten signatures and which only need electronic seals or official seals. For the handwritten signature area, the present application will automatically identify the signature content, and trigger the manual review link when the recognition confidence is low, to ensure that the authenticity and legality of the signature are not affected by the recognition difficulty.

[0055] In one specific embodiment of the present application, in combination withFigure 1 and Figure 2 The S600 includes: S610, integrate the final compliance determination result with the final standardization test result, and bind each conclusion in the integrated result with evidence information; After completing multimodal element extraction, rule execution, and signature compliance checks, this application will automatically bind all review conclusions with corresponding evidentiary information. For each compliance conclusion, this application will clearly indicate the corresponding document page number, layout coordinates, table cell, or image area location, and retain the text content, signature information, and relevant context, ensuring that each judgment has a complete chain of evidence.

[0056] S620, standardize all conclusions and corresponding evidence information to generate a structured report and annotation document; wherein, the structured report includes the compliance conclusion, evidence location and confidence score of each signature and seal area; After the evidence binding is completed, this application will structure all conclusions and their evidence information, uniformly adopting standardized data formats (such as JSON, XML) to ensure that each conclusion contains the necessary fields, including conclusion number, rule ID, document name, page number, evidence coordinates, text or image content, judgment result, confidence score, and related explanations. This standardized processing ensures that subsequent data storage, integration with this application, and external interface calls can all proceed smoothly.

[0057] S630, the structured report and annotation document are output in a standardized format and stored in the local deployment environment.

[0058] Once all review results are completed, this application will export structured reports and annotation documents in a standardized format and support secure storage in a localized deployment environment. The report export format can be configured according to business needs, such as JSON for system integration, and PDF or HTML for manual archiving or external display, ensuring that review conclusions can be accessed by automated systems as well as be easily viewed and archived manually.

[0059] In one specific embodiment of this application, combined with Figure 1 and Figure 2 The S700 includes: S710, The conclusions with confidence scores lower than the confidence threshold are manually reviewed to obtain the manual review results; During the automated review process, this application will mark conclusions with low confidence or conflict. Human reviewers can view the corresponding evidence information, rule basis, and reasons for judgment in the interactive interface, and make manual confirmation or corrections when necessary to ensure the reliability and authority of the final review results.

[0060] S720, updating the rule knowledge base and the experience rule instruction template in the automatic review process according to the manual review result; After the manual review is completed, the application will write the correction result back to the rule knowledge base and the Prompt template, realizing the dynamic updating of the rule base and the review instruction. For example, when some experience rules need to be modified due to policy changes or business process adjustments, the conclusions of manual correction will be automatically converted into new rules or prompt instructions, so that the updated rules and Prompt information can be directly applied to subsequent tasks when they are executed.

[0061] S730, dynamically adjusting the confidence threshold, signature clarity parameter, cross-document consistency determination standard and multi-modal information extraction strategy according to the manual review result and review statistical data.

[0062] The application will dynamically adjust the confidence threshold, signature clarity parameter, cross-document consistency determination standard and multi-modal information extraction strategy according to the manual review feedback and review statistical data, to ensure that the automatic review process can maintain stable accuracy and reliability in the case of data distribution changes or document template updates.

[0063] The application supports local deployment. Alternatives can use a cloud-edge collaborative architecture to complete data preprocessing and desensitization on the edge server, and complete model inference and rule review on the cloud; or use a distributed computing architecture to distribute different review modules on multiple computing nodes to improve the processing speed and system throughput of batch review of files.

[0064] In a second aspect, the application provides a tobacco file review system based on a visual-linguistic large model, comprising: A determination module configured to determine a review object and a review range according to the review requirements of the tobacco file; wherein the review object is all documents of the tobacco file, and the review range is the content points in the documents; A reading-in module configured to read in the review object and pre-process it to obtain a pre-processed review object; A rule module configured to introduce clear rules and experience rules in the tobacco industry into the automatic review process to form a collaborative mechanism; A review module configured to model the pre-processed review object to obtain a global representation of the review object, extract key information elements from the global representation, call the clear rules and experience rules to automatically review the key information elements to obtain a review result, and determine the compliance of the review result to obtain a final compliance determination result and corresponding evidence sources; The detection module is configured to perform standardization detection on the signatures and the signatures in the key information elements to obtain a final standardization detection result. The integration module is configured to integrate the final compliance determination result and the final standardization detection result to output a compliance conclusion, an evidence position and a confidence score of each signature area. The review module is configured to manually review the conclusion whose confidence score is lower than a confidence threshold, and adjust part of the automatic review process according to the manual review result.

[0065] In a third aspect, the present application further provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus. The memory is used to store a computer program. The processor is used to execute the program stored on the memory to realize the steps of the tobacco file review method based on the visual-linguistic large model.

[0066] The present application proposes a tobacco file review method and system based on a visual-linguistic large model, which integrates multi-modal feature extraction, cross-file content association, a large model-based rule introduction mechanism and a large model-based evidence traceability mechanism. For the case file archive materials of the tobacco industry which are not in the specified tobacco enterprise purchase case, the electronic version files and paper scanning images containing directories, attachments, photos and the like are uniformly analyzed, element extraction, signature standardization detection and rule compliance review are performed, and the results of whether the file archive is complete, whether the signature and the signature are correct, the conclusion and the evidence positioning are output. The Prompt dynamic instruction and the rule knowledge base are combined to realize the flexible introduction of the specification and standard of the industry for the case review, and the automatic, interpretable and traceable file review task is completed.

[0067] It should be noted that the terms "first" and "second" in the present application are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first" and "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0068] The above is a further detailed description of the present application in combination with specific preferred embodiments, and the specific implementation of the present application cannot be limited to these descriptions. For ordinary skilled persons in the technical field to which the present application belongs, some simple deductions or replacements can be made without departing from the concept of the present application, and all of them should be regarded as falling within the protection scope of the present application.

Claims

1. A method for reviewing tobacco case files based on a large visual-language model, characterized in that, include: S100, determine the review object and scope based on the review requirements of the tobacco case file; wherein, the review object is all documents in the tobacco case file, and the review scope is the key points of the content in the documents; S200, Read in the review object and preprocess it to obtain a preprocessed review object; S300 introduces both explicit rules and empirical rules from the tobacco industry into the automated review process to create a collaborative mechanism between the two. S400: Perform feature modeling on the preprocessed review object to obtain a global representation of the review object; extract key information elements from the global representation; call the explicit rules and empirical rules to automatically review the key information elements to obtain a review result; perform compliance determination on the review result to obtain a final compliance determination result and the corresponding evidence source; S500, Perform a standardization check on the seals and signatures in the key information elements to obtain the final standardization check result; S600, integrate the final compliance judgment result with the final standardization test result, and output the compliance conclusion, evidence location and confidence score for each seal and signature area; S700, the conclusions with confidence scores lower than the confidence threshold are manually reviewed, and some content in the automated review process is adjusted based on the results of the manual review.

2. The tobacco case file review method based on a visual-language large model according to claim 1, characterized in that, S100 includes: S110, based on the review requirements of the tobacco case file, obtain the target of the case file materials, and read and parse all documents in the case file according to the target of the case file materials, and identify all documents as the review objects; S120, Logically and business-wise associate the reviewed objects to obtain associated reviewed objects; S130, in the content of the object under review, the key points of the content are determined as the scope of review.

3. The tobacco case file review method based on a visual-language large model according to claim 1, characterized in that, S200 includes: S210, if the object of review is an electronic document, then it is automatically parsed to obtain a preprocessed object of review; S220, if the object of review is a paper document, then the image of the paper document is acquired, and the image is subjected to quality enhancement and geometric correction operations to obtain a preprocessed object of review.

4. The tobacco case file review method based on a visual-language large model according to claim 1, characterized in that, The S300 includes: S310 introduces empirical rules for the tobacco industry and uses the Prompt instruction to convert these empirical rules into empirical rule instruction templates. S320: Obtain the explicit rules and regulations of the tobacco industry and compile them into a rule knowledge base; S330, the empirical rule template is introduced into the automated review process via the Prompt command, and the rule knowledge base is introduced into the automated review process via knowledge base retrieval.

5. The tobacco case file review method based on a large visual-language model according to claim 1, characterized in that, The S400 includes: S410, send all documents in the tobacco case file into the Qwen2.5-VL large model in a multimodal form so that the Qwen2.5-VL large model can establish a global representation of the tobacco case file content in a unified semantic space through feature encoding of text, layout and image data and cross-modal semantic alignment; S420, key information elements are parsed from the global representation, and the semantic and numerical consistency of the key information elements in all documents is detected. If there is an anomaly in the consistency, the corresponding key information elements and documents are marked. S430, invoke the stated explicit rules and empirical rules to automatically review the key information elements of consistency and obtain the compliance review results; S440 makes a compliance determination on the compliance review results and the corresponding key information elements, and obtains a compliance determination result of passing, failing or questionable, and binds the source of evidence for each compliance determination result; S450, the compliance determination result is manually reviewed and feedback is written back to the final compliance determination result.

6. The tobacco case file review method based on a large vision-language model according to claim 1, characterized in that, The S500 includes: S510, Locate the location area of ​​the signature and seal in the key information element, determine whether there is a corresponding seal and / or signature content in the location area, and if so, extract the seal image and / or signature content from the location area. S520, perform integrity verification on the signature image. If the verification fails, mark it as abnormal. If the verification passes, verify its continuity and authenticity to obtain the final standardization detection result of the signature. S530: The clarity and recognizability of the verified signature image are evaluated to obtain the evaluation result, and the image is enhanced according to the evaluation result to obtain a signature image with the required clarity and recognizability. S540, the signature content is identified to obtain an identification result. If the confidence level of the identification result is lower than the confidence level threshold, the signature content is manually reviewed. If the confidence level of the identification result is not lower than the threshold, the final standardization detection result of the signature is obtained.

7. The tobacco case file review method based on a visual-language large model according to claim 1, characterized in that, The S600 includes: S610, integrate the final compliance determination result with the final standardization test result, and bind each conclusion in the integrated result with evidence information; S620, standardize all conclusions and corresponding evidence information to generate a structured report and annotation document; wherein, the structured report includes the compliance conclusion, evidence location and confidence score of each seal and signature area; S630, the structured report and annotation document are output in a standardized format and stored in the local deployment environment.

8. The tobacco case file review method based on a large vision-language model according to claim 1, characterized in that, The S700 includes: S710, The conclusions with confidence scores below the threshold are manually reviewed to obtain the manual review results; S720 updates the rule knowledge base and experience-based rule instruction templates in the automated review process based on the results of manual review. S730 dynamically adjusts the recognition threshold, signature clarity parameters, cross-document consistency judgment criteria, and multimodal information extraction strategy based on manual review results and review statistics.

9. A tobacco case file review system based on a visual-language large model, characterized in that, include: The determination module is configured to determine the review object and scope based on the review requirements of the tobacco case file; wherein, the review object is all documents in the tobacco case file, and the review scope is the key content points in the documents; The read-in module is configured to read in the review object and preprocess it to obtain a preprocessed review object; The rules module is configured to introduce both explicit and empirical rules from the tobacco industry into the automated review process to create a synergistic mechanism between the two. The review module is configured to perform feature modeling on the preprocessed review object to obtain a global representation of the review object, extract key information elements from the global representation, automatically review the key information elements by calling the explicitly defined rules and empirical rules to obtain a review result, and make a compliance determination on the review result to obtain a final compliance determination result and the corresponding evidence source. The detection module is configured to perform a standardization check on the seals and signatures in the key information elements, and obtain the final standardization check result; The integration module is configured to integrate the final compliance determination result with the final standardization test result, and output the compliance conclusion, evidence location and confidence score for each seal and signature area; The review module is configured to manually review conclusions whose confidence scores are lower than the confidence threshold, and adjust some parts of the automated review process based on the results of the manual review.

10. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the steps of the tobacco case file review method based on a large vision-language model as described in any one of claims 1-9.

Images