An administrative law enforcement file image evidence processing method and system

Abstract

The application relates to the technical field of artificial intelligence assistants, and provides an administrative law enforcement file image evidence processing method and system. The method comprises the following steps: acquiring an image evidence set of an administrative law enforcement file, wherein the image evidence set comprises multiple image evidences; performing feature extraction and hierarchical classification on each image evidence to obtain physical features and a hierarchical classification result; determining an image evidence without an identifier according to the hierarchical classification result; performing attribution determination on the image evidence without the identifier based on the physical features and correlation information of adjacent image evidences to obtain a document grouping; performing element extraction on each document grouping to obtain structured elements; performing hierarchical relationship reasoning on the structured elements to obtain a hierarchical structure; and outputting a structured processing result. The attribution determination based on the physical features and the correlation information of adjacent image evidences enables the image evidence without the identifier to be attributed to the corresponding document, the integrity of cross-page documents is maintained, and the integrity of file structuring is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence assistant technology, and in particular to a method and system for processing image evidence in administrative law enforcement case files. Background Technology

[0002] In the field of electronic processing of administrative law enforcement case files, the automatic classification and element extraction of image evidence in case files typically employ the following methods: First, converting images into text using optical character recognition (OCR), then extracting elements from fixed locations based on pre-configured templates or rules. This method heavily relies on materials with fixed layouts and neat printing. Second, inputting each image in the case file into an image classification model, outputting category labels for each image, with each image being classified independently. Third, inputting images along with open-ended prompts into a multimodal large model, directly outputting classification results or element content, with the output not constrained by a hierarchical system.

[0003] In actual case files, a single document often spans multiple pages, with only the first page containing the document title, and subsequent pages typically lacking any identifying text. The aforementioned method of processing each page independently can only determine its category based on the content of each image itself. For subsequent pages without titles, it is impossible to determine which document they belong to, resulting in the same document being broken into multiple independent parts. Subsequent pages may be misclassified to other documents or cannot be categorized at all. Feature extraction can only be performed within a single page, leading to fragmented contextual information, low structuring, and poor usability. Summary of the Invention

[0004] Based on this, it is necessary to propose a method and system for processing image evidence in administrative law enforcement case files to address the technical problem that subsequent pages without identification cannot be attributed to the original document when the same document spans multiple pages in the case file.

[0005] Firstly, a method for processing image evidence from administrative law enforcement case files is provided, comprising: acquiring an image evidence set from an administrative law enforcement case file, the image evidence set including multiple image evidences; performing feature extraction and hierarchical classification on each image evidence in the image evidence set to obtain physical features and hierarchical classification results; determining unidentified image evidence in the image evidence set based on the hierarchical classification results; for unidentified image evidence in the image evidence set, determining its attribution based on the physical features and the association information of adjacent image evidence to obtain document groups corresponding to the unidentified image evidence; extracting elements from each document group to obtain structured elements; and performing hierarchical relationship reasoning on the structured elements corresponding to each document group to obtain a hierarchical structure. The output structured processing result includes the document grouping, the structured elements, and the hierarchical structure.

[0006] Secondly, an administrative law enforcement case file image evidence processing system is provided, including a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-mentioned administrative law enforcement case file image evidence processing method.

[0007] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described administrative law enforcement case file image evidence processing method.

[0008] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described method for processing image evidence from administrative law enforcement case files.

[0009] The administrative law enforcement case file image evidence processing method and system provided in this application extracts features and classifies each image evidence in the image evidence set to obtain physical features and classification results. Based on the classification results, unidentified image evidence is identified, thus distinguishing unidentified image evidence from the image evidence set.

[0010] Attribution determination for unlabeled image evidence is made based on physical characteristics and the correlation information of adjacent image evidence. Neither type of information depends on whether the image evidence contains title or other identifying text. Unlabeled image evidence can be assigned to the document group corresponding to the document to which it belongs. The same document spanning multiple pages remains intact during the case file processing. Element extraction obtains the complete document context by document group. Hierarchical relationship reasoning preserves the hierarchical semantics in the structured elements. The output structured processing result encapsulates the document group, structured elements, and hierarchical structure in a unified manner, providing a complete data foundation for downstream document generation and case file assembly. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] in: Figure 1 This is a flowchart illustrating the administrative law enforcement case file image evidence processing method in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the end-to-end system processing relationship in an embodiment of the present invention; Figure 3This is a flowchart illustrating the process of determining the ownership of multi-page documents in an embodiment of the present invention; Figure 4 This is a structural block diagram of the computer device server in an embodiment of the present invention; Figure 5 This is a structural block diagram of the computer device client in an embodiment of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] Please see Figure 1 As shown, Figure 1 A flowchart illustrating the administrative law enforcement case file image evidence processing method provided in this embodiment of the invention includes the following steps: Step 01: Obtain the image evidence set from the administrative law enforcement case file. The image evidence set includes multiple image evidences. Step 02: Perform feature extraction and hierarchical classification on each image in the image evidence set to obtain physical features and hierarchical classification results; Step 03: Based on the hierarchical classification results, identify the unlabeled image evidence in the image evidence set; Step 04: For unlabeled image evidence in the image evidence set, determine the attribution based on physical characteristics and the association information of adjacent image evidence to obtain the document grouping corresponding to the unlabeled image evidence; Step 05: Extract elements from each document group to obtain structured elements; Step 06: Perform hierarchical relationship reasoning on the structured elements corresponding to each document group to obtain the hierarchical structure; Step 07: Output the structured processing results, which include document grouping, structured elements, and hierarchical structure.

[0015] Specifically, in step 01, the image evidence set can be a collection of case file image evidence generated and requiring processing during administrative law enforcement. Multiple image evidences can be evidence materials preserved in image form within the case file, including law enforcement documents, registration forms, receipts, on-site photos, identity documents, and handwritten records. Each image evidence can be a scanned copy or a photograph. Obtaining the image evidence set can involve receiving a batch of case file images to be processed at once. The image evidence set provides processing objects for subsequent feature extraction and hierarchical classification. The arrangement order of multiple image evidences reflects the sequential relationship of pages in the case file. Adjacent image evidences have a sequential relationship in content, and this sequential relationship provides a data foundation for utilizing the correlation information of adjacent image evidence in subsequent steps.

[0016] In step 02, feature extraction can be the process of processing image evidence and calculating image features, hierarchical classification can be the process of dividing image evidence into multiple hierarchical categories, physical features can be feature data describing the physical attributes of image evidence, and hierarchical classification results can be the classification results of image evidence in each hierarchical category. Feature extraction calculates feature data representing the physical attributes of image evidence based on the pixel data of image evidence, and hierarchical classification determines the position of image evidence in the category system based on the content of image evidence. Both feature extraction and hierarchical classification use image evidence as the processing object. Physical features and hierarchical classification results describe image evidence from two dimensions: physical attributes and category attributes, respectively, providing a basis for subsequent steps to distinguish image evidence.

[0017] In step 03, unlabeled image evidence can be image evidence in the image evidence set that does not contain identifying text such as titles. Identifying unlabeled image evidence involves recognizing image evidence without identifying text from the hierarchical classification results. The hierarchical classification results contain category information for each image evidence. Based on the category information, image evidence containing identifying text is distinguished from image evidence without identifying text, and image evidence without identifying text is identified as unlabeled image evidence. Unlabeled image evidence typically corresponds to pages after the first page of a multi-page document. These pages lack title information to identify the document to which they belong. Identifying unlabeled image evidence allows subsequent steps to perform attribution determinations for these pages.

[0018] In step 04, attribution determination can be the process of determining the document to which the unidentified image evidence belongs. Adjacent image evidence can be image evidence in the image evidence set that is adjacent to the unidentified image evidence in the case file order. Association information can be feature information describing the relationship between the unidentified image evidence and adjacent image evidence. Document grouping can be a set of image evidence belonging to the same document. Physical features characterize the physical attributes of the unidentified image evidence itself, and association information of adjacent image evidence characterizes the connection relationship between the unidentified image evidence and adjacent pages. Both types of information together constitute the basis for attribution determination. Attribution determination does not depend on whether the unidentified image evidence contains identifying text such as titles.

[0019] By determining attribution, unidentified image evidence is grouped into the document group corresponding to the document to which it belongs. The same document spanning multiple pages remains intact in the image evidence set. Subsequent element extraction is processed by document group to obtain the complete document context.

[0020] In step 05, feature extraction can be the process of extracting structured features from image evidence in a document group. Structured features can be field-based data describing the content elements of the document. Document groups contain image evidence belonging to the same document. Feature extraction uses document groups as the processing unit, aggregating feature information scattered across different pages of the document into complete structured features, avoiding contextual breaks caused by extracting features on a single page basis.

[0021] In step 06, hierarchical relationship reasoning can be the process of reasoning about the hierarchical subordinate relationships between structured elements. The hierarchical structure can be structured data describing the hierarchical subordinate relationships between structured elements. Hierarchical subordinate relationships typically exist between elements in a dossier. Hierarchical relationship reasoning preserves this hierarchical semantics, clearly presenting the correspondence between higher-level and lower-level elements in the hierarchical structure, thus providing semantically complete structured data for subsequent document generation and dossier assembly.

[0022] In step 07, the output of the structured processing result can be the process of organizing document groups, structured elements, and hierarchical structures into a unified processing result and outputting it. The structured processing result can be a data set composed of document groups, structured elements, and hierarchical structures. Document groups describe the attribution relationships of image evidence, structured elements describe the content elements of the documents, and hierarchical structures describe the hierarchical semantics between elements. After the three types of data are uniformly encapsulated, they are output to downstream processes, enabling downstream processes to directly perform document generation and case file assembly based on the structured processing result, without the need to repeatedly process image evidence.

[0023] Steps 01 to 07 above constitute the overall process of the administrative law enforcement case file image evidence processing method. The end-to-end system processing relationship is as follows: Figure 2 As shown.

[0024] In some embodiments, step 02 above includes: Each image piece of evidence undergoes orientation correction, noise reduction, background removal, and layout segmentation to obtain a normalized image; Physical features are extracted from the normalized image, including paper color features, texture features, page size features, and layout features; According to the predetermined classification rules, each image piece of evidence is classified level by level, and the next level category is predicted under the condition that the previous level category is determined. The candidate category scores at each level are normalized, and the candidate category with the highest normalized score is determined as the predicted category for the corresponding level. Based on the prediction results of each category, hierarchical classification results are obtained.

[0025] Specifically, orientation correction can be the process of correcting the tilt direction of image evidence; denoising can be the process of removing noise from image evidence; background removal can be the process of removing background interference from image evidence; and layout segmentation can be the process of dividing image evidence into regions according to the layout structure. Layout segmentation can be performed based on Document Layout Analysis (DLA) technology. After orientation correction, denoising, background removal, and layout segmentation, a normalized image is obtained. The normalized image eliminates the influence of shooting angle, scanning noise, and background interference on the content of image evidence, making the layout structure and content areas of image evidence clearly presented.

[0026] Physical features are extracted from the normalized image. These features include paper color, texture, page size, and layout characteristics. Paper color features can be represented using a paper color histogram, texture features using texture descriptors, and layout features using header / footer layout codes. The paper color histogram characterizes the background color of the paper in the normalized image; the texture descriptor characterizes the texture differences between printed and handwritten areas; the page size feature describes the size attributes of the pages in the image; and the header / footer layout code characterizes the consistency of the layout structure across pages within the same document. These physical features provide data for hierarchical classification and attribution determination.

[0027] Hierarchical classification proceeds step-by-step according to predetermined hierarchical rules. These rules can be pre-set hierarchical classification rules that specify the number of categories and the candidate sets for each category. Based on the predetermined hierarchical rules, the next level category is predicted given the determination of the previous level category. The classification at the next level is based on the classification result of the previous level. The prediction of each category constitutes a chain-like discrimination process. The discrimination probability of hierarchical classification is calculated according to equation (1): P(y1|I)×P(y2|y1,I)×P(y3|y2,y1,I) In the formula, I represents the input image evidence and the context of the image evidence; y1, y2, and y3 represent the first-level category, the second-level category, and the third-level category, respectively; P(y1|I) represents the probability of predicting the first-level category under the condition of image evidence; P(y2|y1,I) represents the probability of predicting the second-level category under the condition that the first-level category is determined; and P(y3|y2,y1,I) represents the probability of predicting the third-level category under the condition that the first-level and second-level categories are determined. The values ​​of each probability range from 0 to 1. The candidate category scores of each level are normalized. The candidate category scores can be the original scores of the multimodal large model outputting the candidate categories. The normalization process uses the normalization exponential function (Softmax), and the normalized probability of the candidate category is calculated according to formula (2): p(c)=exp(z(c)) / Σexp(z(j)),y(k1)=argmax p(c) In the formula, c represents the candidate category, z(c) represents the original score of candidate category c, exp represents the exponential function, Σexp(z(j)) represents the sum of the exponential function values ​​of all candidate categories at the corresponding level, j represents any candidate category in the candidate category set at the corresponding level, p(c) represents the normalized probability of candidate category c, and y(k1) represents the predicted category at the corresponding level, i.e., the candidate category with the largest p(c) value. The candidate category with the highest normalized score is determined as the predicted category at the corresponding level. Based on the prediction results of each category, the hierarchical classification results are obtained.

[0028] For example, for an image of an on-site inspection record, the primary candidate category is first determined to be "law enforcement document." Then, given the determination of "law enforcement document," the secondary candidate category is determined to be "record." Finally, given the determination of "record," the tertiary candidate category is determined to be "on-site inspection record." Taking the secondary category discrimination as an example, the secondary candidate categories are record, decision, and notification. The original scores for the three candidate categories are 2.0, 0.5, and 0.3, respectively. After processing with a normalized exponential function, the normalized scores for the three candidate categories are approximately 0.70, 0.16, and 0.13, respectively. The candidate category with the highest normalized score is the record, and the predicted category for the secondary category is record. Each category is discriminated level by level using the same processing method, and the hierarchical classification result is composed of the prediction results of each category.

[0029] In some implementations, hierarchical classification is performed by a multimodal large model. Driven by constrained prompts, the multimodal large model classifies image evidence hierarchically according to predetermined classification rules. The constrained prompts limit the candidate sets and output formats for each category, ensuring that the classification output of the multimodal large model conforms to the predetermined classification rules. The number of hierarchical classification levels and the categories for each level can be preset according to actual business needs, while the chain-conditional probability discrimination method remains unchanged.

[0030] In some implementations, step 04 above includes: Based on the hierarchical classification results, documents corresponding to image evidence containing the identifier are identified as candidate documents; Calculate the similarity of unidentified image evidence with each candidate document in terms of layout, contextual coherence, and physical features; The similarity scores are obtained by weighting and fusing layout similarity, contextual coherence, and physical feature similarity. Based on the fusion similarity, unlabeled image evidence is assigned to the corresponding document group.

[0031] Please see Figure 3 As shown, Figure 3 This is a flowchart illustrating the process of determining the ownership of multi-page documents according to an embodiment of the present invention. The process includes determining candidate documents, calculating similarity across three dimensions, weighted fusion, and ownership determination. Figure 3 The process shown will be executed sequentially.

[0032] Specifically, candidate documents can be a candidate set consisting of documents corresponding to image evidence containing labels, based on the hierarchical classification results. Image evidence containing labels can be image evidence containing identifying text such as titles within the image evidence set. Based on the hierarchical classification results, documents corresponding to image evidence containing labels are identified as candidate documents, and these candidate documents provide a candidate set for the attribution of unlabeled image evidence.

[0033] Layout similarity is a numerical value describing the similarity in layout structure between unlabeled image evidence and candidate documents. Contextual coherence is a numerical value describing the semantic coherence between unlabeled image evidence and adjacent image evidence. Physical feature similarity is a numerical value describing the similarity in physical features between unlabeled image evidence and candidate documents. All three similarity values ​​range from 0 to 1, with values ​​closer to 1 indicating greater similarity in the corresponding dimension.

[0034] The similarity of layout, contextual coherence, and physical features are weighted and fused to obtain the fused similarity, which is calculated according to equation (3): S(p(k),D)=α×sim_lay+β×sim_ctx+γ×sim_phy In the formula, p(k) represents the unlabeled image evidence to be assigned, D represents the candidate document, sim_lay represents the layout similarity, sim_ctx represents the contextual coherence, sim_phy represents the physical feature similarity, α represents the weight corresponding to the layout similarity, β represents the weight corresponding to the contextual coherence, and γ represents the weight corresponding to the physical feature similarity. α, β, and γ are all values ​​between 0 and 1, and α+β+γ=1. Weighted fusion combines the similarities of the three dimensions into a comprehensive value according to their weights. The larger the fused similarity value, the more consistent the unlabeled image evidence is with the candidate document across multiple dimensions. Based on the fused similarity, the unlabeled image evidence is assigned to the corresponding document group. The attribution determination integrates information from the three dimensions to determine the document to which the unlabeled image evidence belongs.

[0035] For example, an administrative penalty decision has 3 pages. Page 1 contains the table title, while pages 2 and 3 do not. Regarding the attribution determination for page 2, the candidate document is the one corresponding to the aforementioned administrative penalty decision. The similarity between page 2 and the candidate document is measured as 0.88 in format, 0.82 in contextual coherence, and 0.95 in physical feature similarity. The weights for format similarity (α) and contextual coherence (β) are set to 0.4 and 0.2 respectively. The calculated fusion similarity is 0.4 multiplied by 0.88, plus 0.4 multiplied by 0.82, plus 0.2 multiplied by 0.95, which equals 0.87.

[0036] In some implementations, contextual coherence is determined based on the cosine similarity between the semantic embedding vectors of unlabeled image evidence and adjacent image evidence, and the contextual coherence is calculated according to equation (4): sim_ctx=cos(e(k 1),e(k)) In the formula, sim_ctx represents the context coherence degree, e(k 1) represents the semantic embedding vector of the preceding image evidence adjacent to the unlabeled image evidence, e(k) represents the semantic embedding vector of the unlabeled image evidence, and cos represents the cosine similarity. The semantic embedding vector can be a vector obtained by transforming the image evidence through a multimodal large model or a text encoder. The cosine similarity is used to measure the closeness of two semantic embedding vectors in direction, and the value ranges from -1 to 1. The closer the cosine similarity value is to 1, the more consistent the directions of the two semantic embedding vectors are, and the closer the semantic content connection between adjacent pages is.

[0037] In some implementations, contextual coherence can also be determined using metrics such as Euclidean distance or structural similarity. The fusion of layout similarity, contextual coherence, and physical feature similarity can also be performed using multiplicative fusion or learning-based fusion. Multiplicative fusion multiplies the similarities of the three dimensions to obtain the fused similarity, while learning-based fusion determines the fusion parameters of the three-dimensional similarity through training.

[0038] In some implementations, the attribution determination in step 04 above can also be performed using a multimodal large language model (MLLM) verification method: the unlabeled image evidence, along with adjacent image evidence and the table name and context corresponding to the merged documents, are input into the MLLM. The MLLM explicitly infers whether the unlabeled image evidence is a continuation page of the merged documents, obtaining a continuation page determination result. The continuation page determination result is then cross-verified with the attribution result based on the fusion similarity. When the continuation page determination result and the attribution result are consistent, the attribution result is used as the final attribution of the unlabeled image evidence. When the continuation page determination result and the attribution result are inconsistent, the continuation page determination result and the attribution result are weighted and fused to obtain the final attribution.

[0039] The multimodal large model review method is suitable for case file scenarios with complex layouts and insufficient differentiation of fusion scoring, and can improve the robustness of attribution determination.

[0040] In some implementations, the above-mentioned assignment of unlabeled image evidence to corresponding document groups based on fusion similarity includes: When the maximum fusion similarity between unlabeled image evidence and each candidate document reaches a preset threshold, the unlabeled image evidence is assigned to the candidate document corresponding to the maximum fusion similarity. When the maximum fusion similarity is lower than a preset threshold, unlabeled image evidence is classified into a catch-all category, i.e., into other evidence categories.

[0041] Specifically, the preset threshold can be a pre-set numerical limit for determining whether the attribution is valid, and the catch-all category can be a category for including image evidence that cannot be attributed to any candidate document. For unlabeled image evidence, the maximum value of the fusion similarity corresponding to each candidate document is taken as the maximum fusion similarity. The maximum fusion similarity is compared with the preset threshold, and the attribution result is determined according to equation (5): a(p(k))=D Or a(p(k))=OTHER In the formula, a(p(k)) represents the attribution result of unlabeled image evidence p(k), τ represents the preset threshold, and D This represents the candidate document corresponding to the maximum fusion similarity, and OTHER represents the catch-all category, which is other evidence categories. When the maximum fusion similarity reaches the preset threshold τ, a(p(k)) becomes D. When the maximum fusion similarity is lower than the preset threshold τ, a(p(k)) is OTHER. When the maximum fusion similarity reaches the preset threshold, it means that the overall similarity between the unlabeled image evidence and a certain candidate document in multiple dimensions meets the attribution requirements, and the unlabeled image evidence is assigned to the candidate document corresponding to the maximum fusion similarity.

[0042] When the maximum fusion similarity is lower than the preset threshold, it means that the overall proximity of the unlabeled image evidence to any candidate document does not meet the attribution requirements. The unlabeled image evidence is then classified into the catch-all category. The catch-all category avoids forcibly incorporating image evidence that does not belong to any candidate document into the document.

[0043] For example, continuing with the scenario of the aforementioned administrative penalty decision, the fusion similarity between page 2 and the candidate documents is 0.87. The preset threshold is set to 0.7, and the maximum fusion similarity reaches the preset threshold. Therefore, page 2 is assigned to the document group corresponding to the aforementioned administrative penalty decision. The maximum fusion similarity between page 5 and all candidate documents is 0.55, which is below the preset threshold. Therefore, page 5 is placed in the catch-all category and will not be merged into any document.

[0044] In some implementations, step 05 above includes: Based on predefined structured element templates, element recognition is performed on image evidence in each document group; For each feature field identified, determine the field value, handwritten identifier, and initial field confidence level; When the handwritten identifier indicates that the field is handwritten, a preset handwriting penalty coefficient is applied to the initial field confidence to obtain the calibrated field confidence. Field values, handwritten identifiers, and calibrated field confidence levels are used as structured elements.

[0045] Specifically, the predefined structured element template can be a predefined set of element fields to be extracted from document groups. Element fields can include document name, parties involved, time, location, etc. Element recognition can be the process by which a multimodal large model identifies element content field by field according to the structured element template. Element recognition falls under the processing scope of Key Information Extraction (KIE). Field values ​​can be the element content corresponding to the element field. Handwritten identifiers can be identification information indicating the source of the field content, indicating whether the field has a printed or handwritten source. The initial field confidence score can be the initial confidence score value given by the multimodal large model for the field value, ranging from 0 to 1.

[0046] When the handwritten identifier indicates that the field is handwritten, a preset handwriting penalty coefficient is applied to the initial field confidence to obtain the calibrated field confidence. The calibrated field confidence is calculated according to formula (6): conf'=c×(1 λ×1[handwritten]) In the formula, conf' represents the calibrated field confidence, c represents the initial field confidence, λ represents the handwriting penalty coefficient, l[handwritten] represents the handwriting indicator function, and handwritten represents the handwriting state. When the field is from a handwritten source, l[handwritten] takes a value of 1, and when the field is from a printed source, l[handwritten] takes a value of 0. The handwriting penalty coefficient can be a pre-set coefficient used to lower the confidence of the handwritten source field, with a value ranging from 0 to 1. The recognition difficulty of handwritten content is greater than that of printed content, so the confidence of the handwritten source field is lowered. The field value, handwriting identifier, and calibrated field confidence are used as structured elements. The structured elements contain field content, handwriting identifier, and confidence information, providing a basis for judgment in subsequent manual verification.

[0047] For example, if the initial confidence level of a field is 0.92, the handwritten identifier indicates that the field is handwritten, the preset handwritten penalty coefficient is 0.25, and the calibrated field confidence level is the difference between 0.92 multiplied by 1 and 0.25, which equals 0.69, the calibrated field confidence level is lower than the confidence level of the printed source field, and the field is given priority in manual verification.

[0048] In some implementations, when encountering ambiguous or unrecognizable content during feature recognition, instead of generating speculative field values, the field values ​​are designated as blank values, and low confidence scores are generated for these blank values. The blank values ​​and low confidence scores are then processed by a manual verification process along with the structured feature output. Ambiguous or unrecognizable content includes illegible handwritten content and content whose image quality is insufficient for identification.

[0049] In some implementations, feature recognition can also be performed in conjunction with handwriting enhancement methods, including cue word enhancement and model fine-tuning enhancement. Cue word enhancement explicitly requires the multimodal large model to output handwritten identifiers and field-level confidence scores within constrained cue words, and provides a few sample examples. Model fine-tuning enhancement uses handwritten samples to perform low-rank adaptive fine-tuning (LoRA) on the multimodal large model. LoRA enhances the multimodal large model's ability to recognize handwritten content by adjusting some parameters of the multimodal large model. The fine-tuned multimodal large model is then used to perform feature recognition. Cue word enhancement and model fine-tuning enhancement can be used individually or in combination.

[0050] In some implementations, the extraction quality of feature extraction is evaluated using field-level precision, recall, and harmonic mean, which are calculated according to equation (7): P=TP / (TP+FP), R=TP / (TP+FN), F1=2×P×R / (P+R) In the formula, TP represents the number of correctly extracted fields, FP represents the number of incorrectly extracted fields, FN represents the number of missed fields, P represents precision, R represents recall, and F1 represents the harmonic mean of precision and recall. The values ​​of precision, recall, and harmonic mean all range from 0 to 1. For example, in a batch of 120 feature fields are labeled, the model extracts 100 fields, 90 of which are consistent with the labels, 10 are incorrectly extracted, and 30 are missed. The precision is 90 divided by 100, which is 0.90, the recall is 90 divided by 120, which is 0.75, and the harmonic mean is 2 multiplied by 0.90 multiplied by 0.75 and then divided by the sum of 0.90 and 0.75, which is approximately 0.82.

[0051] In some implementations, the method further includes: For each image piece of evidence, a classification confidence score is generated based on the hierarchical classification results. When image evidence is identified as containing handwritten content, a handwriting penalty coefficient is applied to the classification confidence score to obtain the calibrated classification confidence score. Based on the calibrated classification confidence level, determine the priority of manual verification; When the confidence level of a calibrated field is lower than a preset threshold, manual verification of the field value is triggered.

[0052] Specifically, the classification confidence score can be a confidence score value generated from the hierarchical classification results, ranging from 0 to 1. The classification confidence score characterizes the credibility of the hierarchical classification results. When image evidence is identified as containing handwritten content, a handwriting penalty coefficient is applied to the classification confidence score, and the calibrated classification confidence score is obtained according to the calibration method shown in equation (6). Based on the calibrated classification confidence score, the priority of manual verification is determined. Image evidence with a lower calibrated classification confidence score is given a higher priority for manual verification. The priority of manual verification is used to arrange the processing order of manual verification.

[0053] When the confidence level of a calibrated field falls below a preset threshold, manual verification of the field value is triggered. The preset threshold can be a pre-set confidence limit used to determine whether manual verification is necessary. When the confidence level of a calibrated field is below the preset threshold, the field value is determined to require manual verification. Manual verification involves checking and correcting the field values ​​manually. The confidence level of the calibrated field and the priority of manual verification together form the data basis for the verification process, ensuring that low-confidence identification results are reviewed and reducing the likelihood of erroneous identification results entering subsequent stages.

[0054] In some implementations, step 06 above includes: Based on the table recognition results in the structured elements corresponding to each document group, determine the data cells in the table and the candidate parent cells corresponding to each data cell; For each candidate parent cell, extract the geometric and semantic relationship features between the data cell and the candidate parent cell; Based on geometric and semantic relationship features, calculate the normalized score for each candidate parent cell; The candidate parent cell with the highest normalization score is determined as the parent cell of the data cell; A hierarchical structure is obtained based on the parent cell of each data cell and the hierarchical relationship between the table header and the data in the table recognition results.

[0055] Specifically, the table recognition result can be the result of recognizing table-like content in structured elements. A data cell can be a cell in the table that carries data, and a candidate parent cell can be a possible parent cell of the data cell. Based on the table recognition result, the data cells in the table and the candidate parent cells corresponding to each data cell are determined. The candidate parent cells belong to the same table as the data cells.

[0056] For each candidate parent cell, geometric and semantic relationship features between the data cell and the candidate parent cell are extracted. Geometric relationship features describe the positional relationship between the data cell and the candidate parent cell in the table, including cross-column merging, indentation hierarchy, and row-column positional relationships. Semantic relationship features describe the degree of semantic association between the data cell and the candidate parent cell. Based on the geometric and semantic relationship features, a normalized score is calculated for each candidate parent cell, using equation (8): R(c(i),c(j))=φ(c(i),c(j))^T×w, parent(c(j))=argmax R(c(i),c(j)) In the formula, c(j) represents the data cell whose parent is to be determined, c(i) represents the candidate parent cell, φ(c(i),c(j)) represents the relation feature vector composed of geometric and semantic relation features, ^T represents the transpose of the vector, w represents the weight vector of the relation features, R(c(i),c(j)) represents the normalized score of the candidate parent cell c(i) as the parent of the data cell c(j), and parent(c(j)) represents the candidate parent cell with the highest normalized score. The normalized score ranges from 0 to 1. The candidate parent cell with the highest normalized score is determined as the parent cell of the data cell. Based on the parent cell of each data cell and the hierarchical relationship between the table header and the data in the table recognition result, a hierarchical structure is obtained.

[0057] For example, an administrative penalty information table contains a first-level header of party information composed of merged cells, below which are sub-items such as name, organization name, and address. For a cell whose value is "Zhang Mou," candidate parent cells include cells containing party information, etc. The cross-column merging relationship, indentation hierarchy, and semantic association between the name cell and the party information result in the party information having the highest normalization score as the parent cell. Therefore, the parent cell of the name cell is determined to be the party information, and the hierarchical structure preserves the hierarchical relationship between the party information and the name cell.

[0058] In some implementations, step 07 above includes: Based on structured elements and hierarchical structure, multiple samplings are performed under preset constraints to obtain multiple candidate plot descriptions; Consistency assessment is performed on multiple candidate plot descriptions to determine the evidentiary plot description; The evidence description, document grouping, structured elements, and hierarchical structure are encapsulated into a structured processing result and output.

[0059] Specifically, the preset constraints can be pre-set restrictions used to constrain the generation of plot descriptions. These constraints limit the generation of plot descriptions to based on extracted elements, and do not generate content outside of these elements. Based on structured elements and hierarchical structure, multiple sampling is performed under the preset constraints. Multiple sampling can be performing multiple generation operations on the same input, with each generation operation yielding a candidate plot description. Consistency judgment is performed on multiple candidate plot descriptions. Consistency judgment is the process of comparing the degree of consistency among multiple candidate plot descriptions. The evidence plot description is determined according to equation (9): y(k2)=argmax Σ(m=1 to M)1[y(m)=y] In the formula, y(m) represents the candidate plot description generated by the m-th sampling, M represents the number of samplings, 1[y(m)=y] represents the indicator function, which takes a value of 1 when the m-th candidate plot description is consistent with the candidate result y, and a value of 0 when the m-th candidate plot description is inconsistent with the candidate result y, Σ(m=1 to M)1[y(m)=y] represents the number of M candidate plot descriptions that are consistent with the candidate result y, and y(k2) represents the evidence plot description that has obtained majority consensus support. Multiple sampling combined with consistency judgment can suppress occasional deviations in single generation, so that the evidence plot description is based on the content of majority consensus, and the verifiability of the plot description is guaranteed. The evidence plot description, document grouping, structured elements and hierarchical structure are encapsulated into a structured processing result and output to the downstream stage.

[0060] For example, if the same input is sampled 5 times, and the 4 candidate plot descriptions are consistent in their key facts, and the 1 candidate plot description differs from the other 4 candidate plot descriptions, then the consistent statement of the 4 candidate plot descriptions is determined as the evidence plot description, and the candidate plot descriptions that differ are discarded.

[0061] In some implementations, the generation of evidence plot descriptions is performed under strong constraint prompts. These prompts limit the scope of the generating roles, the range of available fields, and the handling of missing elements. When key elements are missing, supplementary prompts are generated, but plot content corresponding to the missing elements is not generated, ensuring that the generated evidence plot description is based on the extracted elements and evidentiary facts. In some implementations, methods to suppress plot description generation bias can also include retrieval enhancement or constraint decoding. Retrieval enhancement incorporates retrieval results of extracted elements during generation, while constraint decoding restricts the output of content inconsistent with the extracted elements during the generation process.

[0062] In some implementations, the structured processing results are encapsulated in JavaScript Object Notation (JSON) format. The encapsulated content includes hierarchical categories, document grouping, structured elements, table hierarchy, and evidence descriptions. The structured processing results can be directly parsed and used by downstream document generation and case file assembly processes.

[0063] In some implementations, the method further includes: The quality of the evidence descriptions is assessed, and an assessment score is obtained. Evidence descriptions with evaluation scores below a preset threshold are identified as samples to be optimized. Revise the sample to be optimized and determine the revision results; The samples to be optimized and the corresponding revision results are fed back to optimize feature extraction, hierarchical classification, element extraction, and the generation of evidence plot descriptions.

[0064] Specifically, quality assessment can be a process of quantitatively evaluating the quality of evidence plot descriptions. The assessment score can be the quantitative result obtained from the quality assessment, and the preset threshold can be a pre-set threshold for the assessment score. Evidence plot descriptions with assessment scores below the preset threshold are identified as samples to be optimized. The samples to be optimized are then revised, and the revision result is determined manually. The revision result can be the correct plot description obtained after manual revision. The samples to be optimized and the corresponding revision results are then fed back. Feedback is the process of using the samples to be optimized and the revision results to update the prompt words or model parameters used in subsequent processing, so that feature extraction, hierarchical classification, element extraction, and the generation of evidence plot descriptions are gradually optimized based on the fed-back samples.

[0065] Model parameter updates include supervised fine-tuning (SFT), direct preference optimization (DPO), and low-rank adaptive fine-tuning.

[0066] In some implementations, the quality assessment uses a manually weighted accuracy rate, which is calculated according to formula (10): Acc(w) = Σs(i) / N In the formula, N represents the number of evaluation samples, s(i) represents the human score of the i-th sample, and the human score can be 0, 0.5, or 1.0. A human score of 0 indicates an incorrect plot description, a human score of 0.5 indicates a partially correct plot description, and a human score of 1.0 indicates a correct plot description. Acc(w) represents the human weighted accuracy, which ranges from 0 to 1. For example, if 20 plot descriptions are sampled and human-scored, 12 will be scored 1.0, 6 will be scored 0.5, and 2 will be scored 0. The human weighted accuracy is equal to 12 multiplied by 1.0, plus 6 multiplied by 0.5, plus 2 multiplied by 0, divided by 20, which equals 0.75.

[0067] In some implementations, the quality assessment employs a semantic similarity combination score, which is calculated according to equation (11): score=α×BLIP2(g,r)+(1 α)×MPNet(g,r) In the formula, g represents the generated text, r represents the reference text, BLIP2(g,r) represents the semantic similarity score between the generated text and the reference text based on BLIP2 (an image-text alignment pre-trained model), MPNet(g,r) represents the semantic similarity score between the generated text and the reference text based on MPNet (a masked language pre-trained model), α represents the combined weight, and score represents the combined semantic similarity score. The semantic similarity score ranges from 0 to 1. The generated text is an evidentiary plot description, and the reference text is a manually annotated standard plot description.

[0068] This application also provides an administrative law enforcement case file image evidence processing system, the system including a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the steps of the above-mentioned administrative law enforcement case file image evidence processing method.

[0069] In some implementations, the administrative law enforcement case file image evidence processing system includes, in sequence, an image preprocessing module, a hierarchical classification module, a multi-page attribution determination module, an element extraction module, a chart-level reasoning module, an evidence plot generation module, a confidence level and manual verification module, and an evaluation and closed-loop optimization module. The output of the previous module serves as the input of the next module. The physical feature output of the image preprocessing module is connected in parallel to the hierarchical classification module and the multi-page attribution determination module. The category and confidence level output of the hierarchical classification module are connected to the multi-page attribution determination module and the confidence level and manual verification module. The structured outputs of each module are finally integrated into the evaluation and closed-loop optimization module, forming a closed-loop connection for data feedback. The image preprocessing module receives batches of case file images, performs denoising, orientation correction, background removal, and page segmentation, and extracts the physical features of each image.

[0070] The hierarchical classification module performs hierarchical classification on images, outputting each category and its calibrated confidence level. The multi-page attribution module calculates a fusion similarity score for unlabeled pages, assigning them to their respective documents or a catch-all category based on preset thresholds. The feature extraction module extracts structured features from each document group. The chart hierarchy reasoning module infers the parent-child relationships between cells in tables with merged cells or multi-level headers, outputting a hierarchical structural representation. The evidence plot generation module generates evidence plot descriptions. The confidence level and manual verification module prioritizes manual verification based on classification and field confidence levels, triggering manual verification for low-confidence results and retrieving revised results.

[0071] The evaluation and closed-loop optimization module is used to quantitatively evaluate the classification results, extraction results, and generation results, and to feed back difficult examples and revised samples for prompt word optimization and model fine-tuning.

[0072] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a server-side method for processing image evidence from administrative law enforcement case files.

[0073] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the functions or steps of a client-side method for processing image evidence from administrative law enforcement case files.

[0074] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0075] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0076] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0077] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for processing image evidence from administrative law enforcement case files, characterized in that, include: Obtain an image evidence set from administrative law enforcement case files, wherein the image evidence set includes multiple image evidences; Feature extraction and hierarchical classification are performed on each image in the image evidence set to obtain physical features and hierarchical classification results; Based on the hierarchical classification results, unidentified image evidence is determined in the image evidence set; Based on the hierarchical classification results, the documents corresponding to the image evidence containing the identifier are identified as candidate documents; Calculate the layout similarity, contextual coherence, and physical feature similarity between the unidentified image evidence and each of the candidate documents; The layout similarity, contextual coherence, and physical feature similarity are weighted and fused to obtain the fused similarity. Based on the fusion similarity, the unlabeled image evidence is assigned to the corresponding document group; For each of the document groups, element extraction is performed to obtain structured elements; Perform hierarchical relationship reasoning on the structured elements corresponding to each document group to obtain a hierarchical structure; The output structured processing result includes the document grouping, the structured elements, and the hierarchical structure.

2. The method for processing image evidence in administrative law enforcement case files according to claim 1, characterized in that, The step of assigning the unlabeled image evidence to the corresponding document group based on the fusion similarity includes: When the maximum fusion similarity between the unidentified image evidence and each of the candidate documents reaches a preset threshold, the unidentified image evidence is assigned to the candidate document corresponding to the maximum fusion similarity. When the maximum fusion similarity is lower than the preset threshold, the unidentified image evidence is classified into the catch-all category.

3. The method for processing image evidence in administrative law enforcement case files according to claim 1, characterized in that, The step of extracting features and performing hierarchical classification on each image in the image evidence set to obtain physical features and hierarchical classification results includes: Each image piece of evidence undergoes orientation correction, noise reduction, background removal, and layout segmentation to obtain a normalized image; The physical features are extracted from the normalized image, and the physical features include paper color features, texture features, page size features, and layout features; Each piece of image evidence is classified level by level according to the predetermined classification rules, wherein the next level category is predicted under the condition that the previous level category is determined. The candidate category scores at each level are normalized, and the candidate category with the highest normalized score is determined as the predicted category for the corresponding level. The hierarchical classification results are obtained based on the prediction results of each category.

4. The method for processing image evidence in administrative law enforcement case files according to claim 1, characterized in that, The step of extracting elements from each document group to obtain structured elements includes: Based on predefined structured element templates, element recognition is performed on the image evidence in each document group; For each feature field identified, determine the field value, handwritten identifier, and initial field confidence level; When the handwritten identifier indicates that the field is handwritten, a preset handwriting penalty coefficient is applied to the initial field confidence to obtain the calibrated field confidence. The field values, handwritten identifiers, and calibrated field confidence levels are used as the structured elements.

5. The method for processing image evidence in administrative law enforcement case files according to claim 4, characterized in that, The method further includes: For each image piece of evidence, a classification confidence score is generated based on the hierarchical classification results. When the handwritten identifier field indicates a handwritten source, the handwritten penalty coefficient is applied to the classification confidence score to obtain the calibrated classification confidence score. Based on the calibrated classification confidence level, the priority of manual verification is determined; When the confidence level of the calibrated field is lower than a preset threshold, the value of the field is verified.

6. The method for processing image evidence in administrative law enforcement case files according to claim 1, characterized in that, The step of performing hierarchical relationship reasoning on the structured elements corresponding to each document group to obtain a hierarchical structure includes: Based on the table recognition results in the structured elements corresponding to each document group, determine the data cells in the table and the candidate parent cells corresponding to each data cell; For each candidate parent cell, extract the geometric and semantic relationship features between the data cell and the candidate parent cell; Based on the geometric relationship features and the semantic relationship features, calculate the normalized score for each candidate parent cell; The candidate parent cell with the highest normalization score is determined as the parent cell of the data cell; The hierarchical structure is obtained based on the parent cell of each data cell and the hierarchical relationship between the header and data in the table recognition result.

7. The method for processing image evidence in administrative law enforcement case files according to claim 1, characterized in that, The output structured processing result includes: Based on the structured elements and the hierarchical structure, multiple samplings are performed under preset constraints to obtain multiple candidate plot descriptions; A consistency assessment is performed on the multiple candidate plot descriptions to determine the evidence plot description; The evidence description, document grouping, structured elements, and hierarchical structure are encapsulated into the structured processing result and output.

8. The method for processing image evidence in administrative law enforcement case files according to claim 7, characterized in that, The method further includes: The quality of the evidence description is assessed to obtain an assessment score; Evidence descriptions with evaluation scores below a preset threshold are identified as samples to be optimized. The sample to be optimized is revised, and the revision result is determined; The sample to be optimized and the corresponding revision results are fed back to optimize the feature extraction, the hierarchical classification, the element extraction, and the generation of the evidence plot description.

9. An administrative law enforcement case file image evidence processing system, characterized in that, It includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the administrative law enforcement case file image evidence processing method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the administrative law enforcement case file image evidence processing method as described in any one of claims 1 to 8.

Images