A way and system for automatically extracting waybill information based on a large language model

Abstract

This invention discloses a method and system for automatically extracting waybill information based on a large language model, belonging to the field of information processing technology. The method includes: acquiring a waybill image and performing OCR recognition to obtain several text blocks and spatial location information; filtering candidate text blocks based on semantic relevance, spatial location matching, and OCR confidence; inputting the candidate text blocks into a large language model for information extraction to obtain initial results; calculating comprehensive credibility based on OCR confidence, format rules, and cross-field logical rules; triggering secondary extraction or marking low confidence when the comprehensive credibility is lower than a preset threshold; and outputting the final extraction result. This invention, by constructing a multimodal candidate filtering and comprehensive credibility evaluation mechanism, can achieve high-accuracy extraction of cross-format waybill information without relying on format rules and model retraining, and ensures that the credibility of the extraction results is quantifiable and anomalies are manageable, possessing good industrial practical value.

Description

Technical Field

[0001] This invention relates to the field of information processing technology, and in particular to a method and system for automatically extracting waybill information based on a large language model. Background Technology

[0002] With the rapid development of the logistics industry, waybills, as the core information carrier in the logistics and transportation process, directly affect the efficiency of supply chain operations due to their information processing efficiency. Currently, waybills exist in various digital formats such as images, PDFs, and scanned copies, and their layout and structure vary significantly among different logistics companies and in different business scenarios, exhibiting typical cross-format and multi-format characteristics.

[0003] To automate the processing of waybill information, existing technologies primarily employ rule-based template-based parsing schemes or AI-based extraction schemes combining OCR and named entity recognition. The former requires manual configuration of parsing rules or templates for each format, necessitating reconfiguration with every format change, resulting in extremely high maintenance costs. While the latter possesses some generalization ability, it heavily relies on the quality of the front-end OCR recognition. When scanned images are tilted, blurry, or have complex backgrounds, OCR recognition errors cascade and amplify in subsequent processing steps, ultimately leading to information extraction failure. For example... Figure 1 As shown, current automatic waybill information extraction technologies can be mainly divided into three categories: template / rule method, OCR+NLP method, and layout understanding method, corresponding to the gradual evolution from "fixed structure dependency" to "multimodal modeling".

[0004] With the development of large language model technology, some solutions attempt to directly input OCR text into large language models for zero-sample or few-sample information extraction to avoid the costs of rule configuration and model training. However, this approach has the following prominent problems: First, large language models are sensitive to the length of the input context; directly inputting the full OCR text can easily lead to mis-extraction or omission of key fields due to interference from irrelevant information. Second, large language models have an inherent illusion problem, which may generate information that does not exist in the waybill based on contextual assumptions. Third, the output of large language models is random; even with the same input, different inferences may produce different results, lacking a quantitative evaluation and quality control mechanism for the reliability of the extraction results. Fourth, current solutions directly output the extraction results in natural language or JSON format, lacking verification and intervention mechanisms. When the model output format does not meet expectations or the field values ​​violate business rules, the system cannot automatically identify and correct them, and erroneous extraction results will be directly transmitted to downstream business systems, seriously affecting system reliability. Summary of the Invention

[0005] In view of the above-mentioned shortcomings in the current information processing technology field, the present invention provides a method and system for automatic extraction of waybill information based on a large language model, which can achieve accurate extraction of waybill information across formats and reliable quantification and anomaly intervention of the extraction results without relying on format rules and model retraining.

[0006] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0007] An automatic extraction method for waybill information based on a large language model includes:

[0008] The waybill image is acquired and OCR recognition is performed to obtain several text blocks and the spatial location information of each text block;

[0009] For each target field in the preset target field set, candidate text blocks associated with the target field are selected from the text blocks based on the semantic relevance, spatial location matching degree, and OCR confidence degree between each text block and the current target field.

[0010] Each of the target fields and its corresponding candidate text blocks is input into a large language model for information extraction to obtain the initial extraction results.

[0011] The overall credibility of the extraction result of each target field in the initial extraction result is calculated respectively. The overall credibility is determined based on the OCR confidence of the candidate text block corresponding to the extraction result, the verification result of the extraction result through the predefined format rules, and the verification result of the extraction result through the predefined cross-field logic rules.

[0012] When the overall confidence level is lower than a preset threshold, a secondary extraction of the corresponding target field is triggered or it is marked as a low-confidence result.

[0013] Output the final extraction results for each target field.

[0014] According to one aspect of the present invention, selecting candidate text blocks associated with the target field from the plurality of text blocks specifically includes:

[0015] For any text block, calculate the semantic consistency score between the text block and the current target field, the spatial position consistency score between the spatial position of the text block and the preset target area of ​​the current target field, the tag matching score between the text content of the text block and the synonym tags of the current target field, and the OCR confidence score of the text block.

[0016] The semantic consistency score, spatial location consistency score, tag matching score, and OCR confidence score are weighted and summed to obtain the matching score between the text block and the current target field.

[0017] The text blocks are sorted according to the matching scores, and the top K text blocks with the highest scores are selected as candidate text blocks.

[0018] According to one aspect of the present invention, the weight of the semantic consistency score is 0.40, the weight of the spatial location consistency score is 0.25, the weight of the tag matching score is 0.20, and the weight of the OCR confidence score is 0.15.

[0019] According to one aspect of the present invention, the comprehensive credibility is calculated by weighting and summing the OCR confidence, the pass rate of the format rule, and the pass rate of the cross-field logic rule to obtain the comprehensive credibility.

[0020] According to one aspect of the present invention, in the calculation of the overall credibility, the weight of the OCR confidence is 0.20, the weight of the format rule validation pass rate is 0.20, and the weight of the cross-field logic rule validation pass rate is 0.15; the overall credibility further includes a model generation probability component, the weight of which is 0.45.

[0021] According to one aspect of the present invention, the preset threshold includes a first threshold and a second threshold, wherein the first threshold is greater than the second threshold;

[0022] When the overall confidence level is greater than or equal to the first threshold, the current extraction value is retained as the final extraction result;

[0023] When the overall credibility is less than the first threshold and greater than or equal to the second threshold, an error correction prompt carrying the candidate evidence of the current target field is constructed, and the error correction prompt is re-inputted into the large language model for secondary reasoning to update the extracted value.

[0024] When the overall credibility is less than the second threshold, the current target field is marked as requiring manual review.

[0025] According to one aspect of the invention, the first threshold is 0.75, the second threshold is 0.55, the sampling temperature of the secondary inference is lower than the sampling temperature of the first inference, and the number of secondary inferences does not exceed 2.

[0026] According to one aspect of the invention, it further includes:

[0027] When the overall credibility is lower than the preset threshold or the verification still fails after secondary reasoning, the extraction anomaly event of the corresponding target field is sent back to the feedback module.

[0028] The feedback module performs error pattern clustering analysis based on the accumulated multiple abnormal events, and updates the field definitions, synonym tags, or format constraint rules in the subsequent structured prompt text based on the clustering results.

[0029] According to one aspect of the present invention, after the feedback module performs error pattern clustering analysis:

[0030] If the error rate of the same field type continues to decrease within a preset number of statistical windows, then the update of the structured prompt text will continue.

[0031] If the problem does not recover within a preset number of statistical windows, then efficient fine-tuning of the parameters of the large language model will be initiated.

[0032] An automatic waybill information extraction system based on a large language model includes:

[0033] The preprocessing module is used to acquire waybill images and perform OCR recognition to obtain several text blocks and the spatial location information of each text block;

[0034] The candidate filtering module is used to filter out candidate text blocks associated with the target field from the text blocks for each target field in the preset target field set, based on the semantic relevance, spatial location matching degree and OCR confidence degree between each text block and the current target field.

[0035] The inference module is used to input each of the target fields and their corresponding candidate text blocks into the large language model for information extraction and to obtain the initial extraction results.

[0036] The credibility assessment module is used to calculate the comprehensive credibility of the extraction result of each target field in the initial extraction result. The comprehensive credibility is determined based on the OCR confidence of the candidate text block corresponding to the extraction result, the verification result of the extraction result through predefined format rules, and the verification result of the extraction result through predefined cross-field logic rules.

[0037] The intervention module is used to trigger a secondary extraction of the corresponding target field or mark it as a low-confidence result when the overall confidence level is lower than a preset threshold.

[0038] The output module is used to output the final extraction results for each target field. Advantages of this invention:

[0039] 1. By using the above candidate screening technology based on three-dimensional weighted matching of "semantics + space + confidence", the full OCR text can be reduced to a limited candidate set of field associations and input into a large language model. This eliminates the noise interference of irrelevant text on the model's inference from the root, effectively suppresses the problems of model mis-sampling, missed sampling and speculative generation, and significantly improves the accuracy and stability of cross-version waybill information extraction.

[0040] 2. By using the above-mentioned comprehensive credibility quantification and evaluation technology based on the fusion and weighting of "OCR confidence + format rules + cross-field logic rules", the uncontrollable free text output of large language models can be transformed into structured data with quantifiable and comparable field-level credibility, so that each output result has a quality benchmark in an engineering sense, fundamentally solving the unreliability problem caused by the randomness of model output.

[0041] 3. Through the above-mentioned three-level dynamic intervention technology scheme of "direct pass / secondary reasoning / manual review" triggered by preset thresholds, low-confidence outputs can be automatically intercepted without human intervention, and suspicious results can be subjected to targeted error correction and re-reasoning with evidence to ensure the reliability and standardization of the final output results, significantly improving the usability of the system in industrial scenarios.

[0042] 4. By automatically feeding back failed samples to the feedback module and performing error pattern clustering analysis as described above, and updating the field definitions, synonym labels, or format constraint rules in the Prompt accordingly, the system performance can be continuously improved without retraining the model, minimizing system maintenance costs and ensuring that extraction performance adapts to changes in data distribution.

[0043] In summary, this invention can achieve accurate extraction of cross-format waybill information and reliable quantification and anomaly intervention of the extraction results without relying on format rules and model retraining. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 A schematic diagram illustrating the existing technology classification of methods for automatically extracting waybill information;

[0046] Figure 2 This is a flowchart illustrating the overall process framework of the automatic waybill information extraction method based on a large language model as described in this invention.

[0047] Figure 3 This is a diagram showing the data flow and closed-loop feedback relationship of the waybill information extraction process described in this invention.

[0048] Figure 4 This is a flowchart illustrating an automatic waybill information extraction method based on a large language model, as described in Embodiment 1 of the present invention.

[0049] Figure 5 This is a flowchart illustrating an automatic waybill information extraction method based on a large language model with closed-loop feedback, as described in Embodiment 2 of the present invention.

[0050] Figure 6 This is a schematic diagram of the structure of an automatic waybill information extraction system based on a large language model, as described in Embodiment 3 of the present invention. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] Figure 2 This paper illustrates the overall workflow framework of the automatic waybill information extraction method based on a large language model according to the present invention. This method achieves intelligent extraction of structured information from waybills of different formats through multimodal input, prompt-driven inference, consistency verification, and closed-loop continuous optimization.

[0053] Example 1

[0054] This embodiment uses an actual logistics waybill image as an input sample to explain the complete implementation process of the present invention in detail. The waybill image is an A4-sized scanned document with a resolution of 300 DPI, and includes sender information area, recipient information area, and logistics information area.

[0055] like Figure 4 As shown, an automatic extraction method for waybill information based on a large language model includes:

[0056] Step S1: Acquire the waybill image and perform OCR recognition to obtain several text blocks and the spatial location information of each text block.

[0057] In step S1 of this embodiment, the system first receives waybill data input. In this embodiment, the input is a scanned image of a waybill in JPG format.

[0058] To facilitate subsequent processing, step S1 in this embodiment specifically includes the following steps:

[0059] Step S11: Image preprocessing and standardization.

[0060] In step S11 of this embodiment, the system performs standardized preprocessing on the input waybill image. Specifically, the system scales the long side of the image proportionally to 1024 pixels and fills the short side with white pixels to 1024 pixels to avoid character distortion caused by direct stretching; at the same time, the RGB image is converted to a grayscale image, and Gaussian filtering with a kernel size of 3×3 and a standard deviation of 1.0 is used to remove noise.

[0061] In practical applications, the scaling size and Gaussian filter parameters mentioned above can be adaptively adjusted according to the quality and resolution of different input images, and are not completely fixed.

[0062] In addition, the system performs quality feature-driven adaptive enhancement. Specifically, the system calculates the image's sharpness, tilt angle, average brightness, and local contrast; when the absolute value of the tilt angle exceeds 0.5 degrees, it uses Hough lines to detect the direction of the main text and performs affine correction; when a quadrilateral page boundary is detected, it employs a four-point perspective transformation; for low-contrast pages, it uses a contrast-limited adaptive histogram equalization with a grid size of 8×8 and a cropping threshold of 2.0; and it preserves the original grayscale in areas containing barcodes or QR codes to avoid destroying the encoded texture.

[0063] Step S12: Page layout perception.

[0064] In step S12 of this embodiment, the system performs layout analysis on the standardized image to extract the structural information of the waybill. Specifically, based on keyword matching (such as "recipient", "sender", "telephone", "waybill number", etc.) and spatial distribution characteristics, the system performs preliminary positioning of key field areas and identifies the originator area, recipient area and logistics information area.

[0065] Step S13: OCR text extraction and layout construction.

[0066] In step S13 of this embodiment, the system performs OCR recognition on the detected text area to obtain the text content of each text block and its corresponding bounding box coordinates. Simultaneously, the system uses a layout diagram construction algorithm to organize the text blocks: using OCR text blocks as nodes, when the vertical overlap ratio of two text blocks is not less than 0.5 and the horizontal spacing is less than 1.5 times the height of the center character on the page, they are merged into the same text line; text lines are first clustered by vertical coordinate, then sorted by horizontal coordinate; labels and values ​​are preferentially connected to their right and bottom nearest neighbors, and aggregated into the same region using DBSCAN within a normalized distance of 0.035.

[0067] In this embodiment, the following text block (partial) is obtained after OCR recognition:

[0068]

[0069] Step S14: Constructing structured data objects.

[0070] In step S14 of this embodiment, the system constructs a structured data object for each text block. This structured data object includes at least the text content of the text block, the bounding box coordinates normalized relative to the page size, the region labels determined by layout analysis, and the OCR recognition confidence level.

[0071] Specifically, each OCR text block is represented as a data object consisting of text content, normalized bounding box coordinates, OCR confidence score, layout area labels, and reading order: where, Represents text content, Represents the normalized bounding box. Indicates OCR confidence level. Indicates page area labels. The reading order is indicated. The bounding box is normalized according to the page width W and page height H, as shown below: Normalized coordinates allow waybills with different resolutions and page sizes to be placed in the same spatial coordinate system. In this embodiment, the normalized bounding box coordinates are represented with a ratio where both page width and height are 1.0.

[0072] In this embodiment, the data object of each text block is represented as follows:

[0073] Text block 1: t1 = "Recipient: Li Si", b1 = [0.08, 0.23, 0.22, 0.28], r1 = "recipient", c1 = 0.96;

[0074] Text block 2: t2="Phone number: 138xxxx1234", b2=[0.08, 0.29, 0.36, 0.34], r2="recipient", c2=0.92;

[0075] Text block 3: t3="Address: No. XX, XX Street, Chaoyang District, Beijing", b3=[0.08, 0.35, 0.62, 0.43], r3="recipient", c3=0.88;

[0076] Text block 4: t4="Waybill Number: YT1234567890", b4=[0.64, 0.08, 0.93, 0.15], r4="logistics", c4=0.95;

[0077] Text block 5: t5="Sender: Zhang San", b5=[0.08, 0.58, 0.22, 0.63], r5="sender", c5=0.97.

[0078] Figure 3The relationship between the main data flow, control flow, and feedback flow in the waybill information extraction process of this invention is illustrated.

[0079] Step S2: For each target field in the preset target field set, based on the semantic relevance, spatial location matching degree, and OCR confidence degree between each text block and the current target field, select candidate text blocks associated with the target field from the text blocks.

[0080] In step S2 of this embodiment, the system reads the structured data objects of each text block constructed in step S1 of this embodiment, and performs candidate filtering for each target field in the preset target field set.

[0081] The target field set preset in this embodiment includes: sender name (sender_name), recipient name (recipient_name), recipient phone number (recipient_phone), recipient address (recipient_address), and waybill number (waybill_number).

[0082] In step S2 of this embodiment, the step of selecting candidate text blocks associated with the target field from the plurality of text blocks specifically includes: for any text block, calculating the semantic consistency score between the text block and the current target field, the spatial position consistency score between the spatial position of the text block and the preset target area of ​​the current target field, the tag matching score between the text content of the text block and the synonym tag of the current target field, and the OCR confidence score of the text block; performing a weighted summation of the semantic consistency score, spatial position consistency score, tag matching score, and OCR confidence score to obtain the matching score between the text block and the current target field; sorting each text block according to the matching score, and selecting the top K text blocks with the highest scores as candidate text blocks.

[0083] In this embodiment, step S2 performs the following steps for each target field:

[0084] Step S21: Calculate the score for each component.

[0085] In step S21 of this embodiment, for any text block and the current target field, the system calculates four component scores respectively:

[0086] (1) Semantic consistency score ( ): Calculate the semantic similarity between the text content of the text block and the definition of the current target field. For example, for the "Recipient Name" field, the "Recipient" in the text block "Recipient: Li Si" has a high degree of semantic similarity with the field definition, so the semantic consistency score is high.

[0087] (2) Spatial location consistency score ( This calculates the degree of spatial matching between the spatial location of the text block and the preset target area of ​​the current target field. For example, for the "Recipient Name" field, the spatial location consistency score of the text block located in the recipient area is higher than that of the text block located in the sender area.

[0088] (3) Tag matching score ( ): Calculates the degree of match between the text content of this text block and the synonym tags of the current target field. For example, for the "recipient's phone number" field, if the text content contains the keyword "phone", it matches the set of synonym tags for this field ("phone", "mobile number", "contact information", etc.).

[0089] (4) OCR confidence score ( ): Directly use the OCR of this text block to identify confidence level.

[0090] Step S22: Weighted summation and sorting filtering.

[0091] In step S22 of this embodiment, the system performs a weighted summation of the scores of the above four components to obtain the matching score SijSij between the text block and the current target field: where the four weights α, β, γ, and δ correspond to the weights of semantic consistency score, spatial location consistency score, tag matching score, and OCR confidence score, respectively.

[0092] In this embodiment, the weight of the semantic consistency score is 0.40, the weight of the spatial location consistency score is 0.25, the weight of the tag matching score is 0.20, and the weight of the OCR confidence score is 0.15. Specifically, it can be expressed as: (α,β,γ,δ)=(0.40,0.25,0.20,0.15).

[0093] The system is based on the matching score The text blocks are sorted, and the top K text blocks with the highest scores are selected as candidate text blocks. In this embodiment, K is set to 3.

[0094] Taking the "recipient's phone number" field as an example, after the system calculates the matching score for all text blocks, text block 2 ("Phone number: 138xxxx1234") has the highest matching score and is ranked first due to its high semantic consistency, location in the recipient area, and OCR confidence of 0.92. Text block 4 contains "YT1234567890" but has no phone number-related semantics, resulting in a lower matching score. Finally, the top three text blocks with the highest scores are selected as candidate text blocks for the recipient's phone number.

[0095] In practical applications, the above K value and weight allocation can be adjusted according to different business scenarios and field-level F1 values ​​on the validation set, and are not completely fixed.

[0096] Step S3: Input each target field and its corresponding candidate text block into the large language model for information extraction to obtain the initial extraction results.

[0097] In step S3 of this embodiment, the system concatenates each target field and its corresponding candidate text block (including the bounding box coordinates and region labels of each candidate text block), task instructions, field definitions, and output format constraints into a structured prompt text, and then inputs the prompt text into the multimodal large language model for inference.

[0098] The structured prompt text content constructed in this embodiment is as follows:

[0099] [Task Description]:

[0100] Extract specified fields from the input waybill OCR text and its layout structure information, and output standard JSON results.

[0101] [Input Information]:

[0102] The input includes the following: OCR-recognized text (including text content and its bounding box position information);

[0103] The layout structure information of the text block (such as area division, relative position relationship, etc.).

[0104] [Target Field Extraction]:

[0105] Sender's name (sender_name);

[0106] Recipient's name (recipient_name);

[0107] Recipient's phone number (recipient_phone);

[0108] Recipient address;

[0109] Waybill number.

[0110] [Processing Steps]:

[0111] Step 1: Understanding the layout area – Identify different functional areas (sender area, recipient area, logistics information area) based on the bounding box and text content, and use spatial relationships to help determine the field's affiliation.

[0112] Step 2: Semantic parsing – Identify field tags (such as “name”, “phone”, “address”, “waybill number”, etc.) from the text, extract the specific values ​​after the corresponding tags, and process the “tag + value” structure.

[0113] Step 3: Merging Cross-line and Broken Information – If a field is split into multiple lines or multiple text blocks, perform semantic concatenation. Address fields spanning multiple text blocks need to be completely merged.

[0114] Step 4: Conflict resolution – If there are multiple candidate values ​​for the same field, prioritize the text located in the corresponding area; if there is still a conflict, select the result with the highest confidence or the most complete text.

[0115] Step 5: Missing information handling – If a field fails to recognize valid information, it should be filled with null. Fields must not be omitted.

[0116] Output Requirements:

[0117] The output is in strict JSON format, containing no explanatory text, Markdown, or additional symbols. Field names are fixed as follows:

[0118] { "sender_name": "", "recipient_name": "", "recipient_phone": "", "recipient_address": "", "waybill_number": "" }

[0119] [Constraints]:

[0120] No conjecture or fabrication of non-existent information is permitted. Judgment should be based primarily on OCR text and layout. Address fields should be complete addresses, phone numbers should be complete numeric strings, and waybill numbers should retain their original format.

[0121] [Input Data]:

[0122] (Insert the text content, normalized bounding box coordinates, and region labels of each text block in step S1 of this embodiment here).

[0123] Subsequently, the system inputs the structured prompt text into the multimodal large language model. The model used in this embodiment is the Qwen2.5-VL-7B-Instruct multimodal large language model, which includes a visual encoder, a visual-language projection layer, and a Transformer language model based on a decoder structure. The visual encoder segments the waybill image into visual markers, the projection layer maps the visual features to the word vector space of the language model, and then inputs these features, along with the OCR text, coordinate markers, and task prompt, into the Transformer for autoregressive decoding.

[0124] In this embodiment, the large language model adopts a low randomness decoding strategy during inference: the sampling temperature is 0.1, the kernel sampling threshold is 0.9, the maximum number of generated tags is 1024, and the repetition penalty coefficient is 1.05; at the same time, it constructs finite state constraints based on the predefined JSONSchema, and only allows the generation of tag sequences that satisfy the field name, data type, required fields, and hierarchical relationship during the decoding process.

[0125] After inference using the large language model, this embodiment yields the following initial extraction results:

[0126] { "sender_name": "Zhang San", "recipient_name": "Li Si", "recipient_phone": "138xxxx1234", "recipient_address": "No. XX, XX Street, Chaoyang District, Beijing", "waybill_number": "YT1234567890" }

[0127] Step S4: Calculate the overall credibility of the extraction result of each target field in the initial extraction result. The overall credibility is determined based on the OCR confidence of the candidate text block corresponding to the extraction result, the verification result of the extraction result through predefined format rules, and the verification result of the extraction result through predefined cross-field logic rules.

[0128] In step S4 of this embodiment, the system reads the initial extraction result obtained in step S3 of this embodiment and calculates the comprehensive credibility of the extracted value of each target field.

[0129] In this embodiment, the comprehensive credibility is calculated by weighting and summing the OCR confidence, the pass rate of the format rule, and the pass rate of the cross-field logic rule to obtain the comprehensive credibility.

[0130] Step S4 in this embodiment specifically includes the following steps:

[0131] Step S41: Format rule verification.

[0132] In step S41 of this embodiment, the system performs format validity checks on the extracted values ​​of each target field to determine whether they conform to predefined format constraint rules. Specifically, this includes: verifying whether the field value conforms to the expected type (e.g., whether a numeric field is a number); and performing format matching on the field based on regular expressions (e.g., whether a phone number conforms to fixed length and number segment rules, and whether a waybill number meets a specified encoding structure). For fields that conform to the format rules, the format rule validation pass rate is... Record it as 1.0; otherwise, record it as 0.0.

[0133] Step S42: Cross-field logical rule validation.

[0134] In step S42 of this embodiment, based on the validity of the field format, the system further performs consistency verification on the semantic relationships between multiple fields to detect potential logical conflicts or unreasonable situations. Specifically, the system checks whether there is a reasonable correspondence between name, phone number, and address, and uses a rule base to determine whether the field combination is reasonable (e.g., whether the region and postal code match). For fields that pass the logical validation, the cross-field logical rule validation pass rate is... Record it as 1.0; otherwise, record it as 0.0.

[0135] Step S43: Calculate the overall credibility by weighted summation.

[0136] In step S43 of this embodiment, the system performs a weighted summation of the OCR confidence level, the pass rate of format rules, and the pass rate of cross-field logic rules to obtain the comprehensive confidence level. .

[0137] In this embodiment, the calculation of the overall confidence level includes the OCR confidence level. The weight is 0.20, and the pass rate of the format rule validation is... The weight is 0.20, and the pass rate of the cross-field logical rule validation is... The weight is 0.15; the overall credibility also includes the model generation probability component. The model generates probabilities with a weight of 0.45.

[0138] Overall credibility The calculation formula is:

[0139]

[0140] in, .

[0141] Taking the "recipient's phone number" field in this embodiment as an example: the model generates probability components. The OCR confidence score for the source text block (text block 2) is 0.96. The value is 0.92, and the format rule validation passed (the phone number is an 11-digit string, and the format is valid). Recorded as 1.0, cross-field logic validation passed (the phone number, recipient's name, and recipient's address all belong to the recipient area, and the logic is consistent). Let it be denoted as 1.0. The overall credibility is then calculated as follows:

[0142]

[0143] The overall credibility calculation results for other fields in this embodiment are as follows:

[0144]

[0145] Step S5: When the overall confidence level is lower than the preset threshold, trigger a secondary extraction of the corresponding target field or mark it as a low confidence result.

[0146] In step S5 of this embodiment, the system calculates the overall reliability of each field based on the results obtained in step S4 of this embodiment. Dynamic intervention triggered by execution thresholds.

[0147] In this embodiment, the preset threshold includes a first threshold and a second threshold, and the first threshold is greater than the second threshold; when the overall credibility is greater than or equal to the first threshold, the current extracted value is retained as the final extraction result; when the overall credibility is less than the first threshold but greater than or equal to the second threshold, an error correction prompt carrying candidate evidence of the current target field is constructed, and the error correction prompt is re-inputted into the large language model for secondary reasoning to update the extracted value; when the overall credibility is less than the second threshold, the current target field is marked as awaiting manual review.

[0148] In this embodiment, the first threshold is 0.75, the second threshold is 0.55, the sampling temperature of the secondary inference is lower than the sampling temperature of the first inference, and the number of secondary inferences does not exceed 2.

[0149] In this embodiment, the overall confidence level of all fields is greater than 0.75 (the lowest being 0.929 for the recipient's address), which is higher than the first threshold of 0.75. Therefore, all current extracted values ​​are retained as the final extraction results without triggering secondary reasoning or manual review.

[0150] To fully illustrate the handling process of this invention when the overall confidence level is lower than the first threshold, the "recipient's phone number" field is used as an example: Assuming that the overall confidence level of this field is calculated to be 0.68 in a certain extraction (between 0.55 and 0.75), the system constructs an error correction prompt carrying candidate evidence for the current target field, including a list of candidate texts for the field (including the normalized bounding box coordinates and region labels of each candidate text), OCR confidence level, and the type of verification failure. Then, the error correction prompt is re-input into the large language model for secondary inference. The sampling temperature of the secondary inference (0.05) is lower than that of the first inference (0.1) to reduce the randomness of the output. If the secondary inference result passes the verification and the overall confidence level is raised to above the first threshold, the secondary inference result is adopted. If the result after the secondary inference is still lower than the first threshold but higher than the second threshold, a maximum of 2 secondary inferences are performed. If it still fails, the field is marked as a low-confidence result. Taking the "Waybill Number" field as an example: Suppose that the overall credibility of this field is calculated to be 0.48 in a certain sampling (below the second threshold of 0.55), then the system directly marks this field as needing manual review and carries an anomaly mark in the output results for downstream systems or manual processing.

[0151] In practical applications, the specific values ​​of the first and second thresholds mentioned above can be adjusted according to the different requirements of accuracy and recall in different business scenarios, and are not completely fixed.

[0152] Step S6: Output the final extraction results for each target field.

[0153] In step S6 of this embodiment, the system outputs the structured extraction result after the above steps.

[0154] The final structured extraction result output in this embodiment is as follows:

[0155] { "sender_name": "Zhang San", "recipient_name": "Li Si", "recipient_phone": "138xxxx1234", "recipient_address": "No. XX, XX Street, Chaoyang District, Beijing", "waybill_number": "YT1234567890" }

[0156] Meanwhile, the system associates source evidence binding information with each extracted value. Taking the recipient's phone number field as an example, the evidence binding information includes: source text block identifier (text block 2), normalized bounding box coordinates ([0.08, 0.29, 0.36, 0.34]), source text content (phone number: 138xxxx1234"), overall credibility (0.966), and model version and Prompt version information, which are used for subsequent auditing and tracing.

[0157] Advantages of implementing this invention:

[0158] This embodiment encodes the visual layout information of the waybill image together with the textual semantic information, enabling the large language model to simultaneously perceive the textual semantics and spatial structure relationship in a unified input space, thus providing a stable multimodal data foundation for field location and recognition across waybill formats.

[0159] This embodiment effectively filters out irrelevant text noise that interferes with model inference by dynamically selecting the candidate text block with the highest matching score for each target field from the full OCR text block. It suppresses the problems of false extraction, missed extraction and speculative generation in large language models from the input source, and significantly improves the accuracy and stability of cross-version waybill information extraction.

[0160] This embodiment integrates four dimensions of evidence—model generation probability, OCR confidence, format rule validation pass rate, and cross-field logical rule validation pass rate—into a unified field-level credible quantifiable index, enabling the output of the large language model to be transformed into structured data with quantifiable, comparable, and traceable credibility.

[0161] This embodiment uses a three-level dynamic intervention mechanism built on the first and second thresholds. It can automatically intercept low-confidence outputs, perform targeted error correction and re-reasoning with evidence for suspicious results, and manually review and mark extremely low-confidence results. This ensures the reliability and standardization of the final output without the need for human intervention.

[0162] In summary, this embodiment can achieve accurate extraction of cross-format waybill information and reliable quantification and anomaly intervention of the extraction results without relying on format rules and model retraining, and has good industrial application value.

[0163] Example 2

[0164] The difference between this embodiment and Embodiment 1 is that this embodiment further includes a closed-loop feedback and continuous optimization process for extracting anomalies.

[0165] like Figure 5 As shown, an automatic extraction method for waybill information based on a large language model includes:

[0166] Step S1: Acquire the waybill image and perform OCR recognition to obtain several text blocks and the spatial location information of each text block.

[0167] In step S1 of this embodiment, the system first receives waybill data input. In this embodiment, the input is a scanned image of a waybill in JPG format.

[0168] To facilitate subsequent processing, step S1 in this embodiment specifically includes the following steps:

[0169] Step S11: Image preprocessing and standardization.

[0170] In step S11 of this embodiment, the system performs standardized preprocessing on the input waybill image. Specifically, the system scales the long side of the image proportionally to 1024 pixels and fills the short side with white pixels to 1024 pixels to avoid character distortion caused by direct stretching; at the same time, the RGB image is converted to a grayscale image, and Gaussian filtering with a kernel size of 3×3 and a standard deviation of 1.0 is used to remove noise.

[0171] In practical applications, the scaling size and Gaussian filter parameters mentioned above can be adaptively adjusted according to the quality and resolution of different input images, and are not completely fixed.

[0172] In addition, the system performs quality feature-driven adaptive enhancement. Specifically, the system calculates the image's sharpness, tilt angle, average brightness, and local contrast; when the absolute value of the tilt angle exceeds 0.5 degrees, it uses Hough lines to detect the direction of the main text and performs affine correction; when a quadrilateral page boundary is detected, it employs a four-point perspective transformation; for low-contrast pages, it uses a contrast-limited adaptive histogram equalization with a grid size of 8×8 and a cropping threshold of 2.0; and it preserves the original grayscale in areas containing barcodes or QR codes to avoid destroying the encoded texture.

[0173] Step S12: Page layout perception.

[0174] In step S12 of this embodiment, the system performs layout analysis on the standardized image to extract the structural information of the waybill. Specifically, based on keyword matching (such as "recipient", "sender", "telephone", "waybill number", etc.) and spatial distribution characteristics, the system performs preliminary positioning of key field areas and identifies the originator area, recipient area and logistics information area.

[0175] Step S13: OCR text extraction and layout construction.

[0176] In step S13 of this embodiment, the system performs OCR recognition on the detected text area to obtain the text content of each text block and its corresponding bounding box coordinates. Simultaneously, the system uses a layout diagram construction algorithm to organize the text blocks: using OCR text blocks as nodes, when the vertical overlap ratio of two text blocks is not less than 0.5 and the horizontal spacing is less than 1.5 times the height of the center character on the page, they are merged into the same text line; text lines are first clustered by vertical coordinate, then sorted by horizontal coordinate; labels and values ​​are preferentially connected to their right and bottom nearest neighbors, and aggregated into the same region using DBSCAN within a normalized distance of 0.035.

[0177] In this embodiment, the following text block (partial) is obtained after OCR recognition:

[0178]

[0179] Step S14: Constructing structured data objects.

[0180] In step S14 of this embodiment, the system constructs a structured data object for each text block. This structured data object includes at least the text content of the text block, the bounding box coordinates normalized relative to the page size, the region labels determined by layout analysis, and the OCR recognition confidence level.

[0181] Specifically, each OCR text block is represented as a data object consisting of text content, normalized bounding box coordinates, OCR confidence score, layout area labels, and reading order: where, Represents text content, Represents the normalized bounding box. Indicates OCR confidence level. Indicates page area labels. The reading order is indicated. The bounding box is normalized according to the page width W and page height H, as shown below: Normalized coordinates allow waybills with different resolutions and page sizes to be placed in the same spatial coordinate system. In this embodiment, the normalized bounding box coordinates are represented with a ratio where both page width and height are 1.0.

[0182] In this embodiment, the data object of each text block is represented as follows:

[0183] Text block 1: t1 = "Recipient: Li Si", b1 = [0.08, 0.23, 0.22, 0.28], r1 = "recipient", c1 = 0.96;

[0184] Text block 2: t2="Phone number: 138xxxx1234", b2=[0.08, 0.29, 0.36, 0.34], r2="recipient", c2=0.92;

[0185] Text block 3: t3="Address: No. XX, XX Street, Chaoyang District, Beijing", b3=[0.08, 0.35, 0.62, 0.43], r3="recipient", c3=0.88;

[0186] Text block 4: t4="Waybill Number: YT1234567890", b4=[0.64, 0.08, 0.93, 0.15], r4="logistics", c4=0.95;

[0187] Text block 5: t5="Sender: Zhang San", b5=[0.08, 0.58, 0.22, 0.63], r5="sender", c5=0.97.

[0188] Step S2: For each target field in the preset target field set, based on the semantic relevance, spatial location matching degree, and OCR confidence degree between each text block and the current target field, select candidate text blocks associated with the target field from the text blocks.

[0189] In step S2 of this embodiment, the system reads the structured data objects of each text block constructed in step S1 of this embodiment, and performs candidate filtering for each target field in the preset target field set.

[0190] The target field set preset in this embodiment includes: sender name (sender_name), recipient name (recipient_name), recipient phone number (recipient_phone), recipient address (recipient_address), and waybill number (waybill_number).

[0191] In step S2 of this embodiment, the step of selecting candidate text blocks associated with the target field from the plurality of text blocks specifically includes: for any text block, calculating the semantic consistency score between the text block and the current target field, the spatial position consistency score between the spatial position of the text block and the preset target area of ​​the current target field, the tag matching score between the text content of the text block and the synonym tag of the current target field, and the OCR confidence score of the text block; performing a weighted summation of the semantic consistency score, spatial position consistency score, tag matching score, and OCR confidence score to obtain the matching score between the text block and the current target field; sorting each text block according to the matching score, and selecting the top K text blocks with the highest scores as candidate text blocks.

[0192] In this embodiment, step S2 performs the following steps for each target field:

[0193] Step S21: Calculate the score for each component.

[0194] In step S21 of this embodiment, for any text block and the current target field, the system calculates four component scores respectively:

[0195] (1) Semantic consistency score ( ): Calculate the semantic similarity between the text content of the text block and the definition of the current target field. For example, for the "Recipient Name" field, the "Recipient" in the text block "Recipient: Li Si" has a high degree of semantic similarity with the field definition, so the semantic consistency score is high.

[0196] (2) Spatial location consistency score ( This calculates the degree of spatial matching between the spatial location of the text block and the preset target area of ​​the current target field. For example, for the "Recipient Name" field, the spatial location consistency score of the text block located in the recipient area is higher than that of the text block located in the sender area.

[0197] (3) Tag matching score ( ): Calculates the degree of match between the text content of this text block and the synonym tags of the current target field. For example, for the "recipient's phone number" field, if the text content contains the keyword "phone", it matches the set of synonym tags for this field ("phone", "mobile number", "contact information", etc.).

[0198] (4) OCR confidence score ( ): Directly use the OCR of this text block to identify confidence level.

[0199] Step S22: Weighted summation and sorting filtering.

[0200] In step S22 of this embodiment, the system performs a weighted summation of the scores of the above four components to obtain the matching score SijSij between the text block and the current target field: where the four weights α, β, γ, and δ correspond to the weights of semantic consistency score, spatial location consistency score, tag matching score, and OCR confidence score, respectively.

[0201] In this embodiment, the weight of the semantic consistency score is 0.40, the weight of the spatial location consistency score is 0.25, the weight of the tag matching score is 0.20, and the weight of the OCR confidence score is 0.15. Specifically, it can be expressed as: (α,β,γ,δ)=(0.40,0.25,0.20,0.15).

[0202] The system is based on the matching score The text blocks are sorted, and the top K text blocks with the highest scores are selected as candidate text blocks. In this embodiment, K is set to 3.

[0203] Taking the "recipient's phone number" field as an example, after the system calculates the matching score for all text blocks, text block 2 ("Phone number: 138xxxx1234") has the highest matching score and is ranked first due to its high semantic consistency, location in the recipient area, and OCR confidence of 0.92. Text block 4 contains "YT1234567890" but has no phone number-related semantics, resulting in a lower matching score. Finally, the top three text blocks with the highest scores are selected as candidate text blocks for the recipient's phone number.

[0204] In practical applications, the above K value and weight allocation can be adjusted according to different business scenarios and field-level F1 values ​​on the validation set, and are not completely fixed.

[0205] Step S3: Input each target field and its corresponding candidate text block into the large language model for information extraction to obtain the initial extraction results.

[0206] In step S3 of this embodiment, the system concatenates each target field and its corresponding candidate text block (including the bounding box coordinates and region labels of each candidate text block), task instructions, field definitions, and output format constraints into a structured prompt text, and then inputs the prompt text into the multimodal large language model for inference.

[0207] The structured prompt text content constructed in this embodiment is as follows:

[0208] [Task Description]:

[0209] Extract specified fields from the input waybill OCR text and its layout structure information, and output standard JSON results.

[0210] [Input Information]:

[0211] The input includes the following: OCR-recognized text (including text content and its bounding box position information);

[0212] The layout structure information of the text block (such as area division, relative position relationship, etc.).

[0213] [Target Field Extraction]:

[0214] Sender's name (sender_name);

[0215] Recipient's name (recipient_name);

[0216] Recipient's phone number (recipient_phone);

[0217] Recipient address;

[0218] Waybill number.

[0219] [Processing Steps]:

[0220] Step 1: Understanding the layout area – Identify different functional areas (sender area, recipient area, logistics information area) based on the bounding box and text content, and use spatial relationships to help determine the field's affiliation.

[0221] Step 2: Semantic parsing – Identify field tags (such as “name”, “phone”, “address”, “waybill number”, etc.) from the text, extract the specific values ​​after the corresponding tags, and process the “tag + value” structure.

[0222] Step 3: Merging Cross-line and Broken Information – If a field is split into multiple lines or multiple text blocks, perform semantic concatenation. Address fields spanning multiple text blocks need to be completely merged.

[0223] Step 4: Conflict resolution – If there are multiple candidate values ​​for the same field, prioritize the text located in the corresponding area; if there is still a conflict, select the result with the highest confidence or the most complete text.

[0224] Step 5: Missing information handling – If a field fails to recognize valid information, it should be filled with null. Fields must not be omitted.

[0225] Output Requirements:

[0226] The output is in strict JSON format, containing no explanatory text, Markdown, or additional symbols. Field names are fixed as follows:

[0227] { "sender_name": "", "recipient_name": "", "recipient_phone": "", "recipient_address": "", "waybill_number": "" }

[0228] [Constraints]:

[0229] No conjecture or fabrication of non-existent information is permitted. Judgment should be based primarily on OCR text and layout. Address fields should be complete addresses, phone numbers should be complete numeric strings, and waybill numbers should retain their original format.

[0230] [Input Data]:

[0231] (Insert the text content, normalized bounding box coordinates, and region labels of each text block in step S1 of this embodiment here).

[0232] Subsequently, the system inputs the structured prompt text into the multimodal large language model. The model used in this embodiment is the Qwen2.5-VL-7B-Instruct multimodal large language model, which includes a visual encoder, a visual-language projection layer, and a Transformer language model based on a decoder structure. The visual encoder segments the waybill image into visual markers, the projection layer maps the visual features to the word vector space of the language model, and then inputs these features, along with the OCR text, coordinate markers, and task prompt, into the Transformer for autoregressive decoding.

[0233] In this embodiment, the large language model adopts a low randomness decoding strategy during inference: the sampling temperature is 0.1, the kernel sampling threshold is 0.9, the maximum number of generated tags is 1024, and the repetition penalty coefficient is 1.05; at the same time, it constructs finite state constraints based on the predefined JSONSchema, and only allows the generation of tag sequences that satisfy the field name, data type, required fields, and hierarchical relationship during the decoding process.

[0234] After inference using the large language model, this embodiment yields the following initial extraction results:

[0235] { "sender_name": "Zhang San", "recipient_name": "Li Si", "recipient_phone": "138xxxx1234", "recipient_address": "No. XX, XX Street, Chaoyang District, Beijing", "waybill_number": "YT1234567890" }

[0236] Step S4: Calculate the overall credibility of the extraction result of each target field in the initial extraction result. The overall credibility is determined based on the OCR confidence of the candidate text block corresponding to the extraction result, the verification result of the extraction result through predefined format rules, and the verification result of the extraction result through predefined cross-field logic rules.

[0237] In step S4 of this embodiment, the system reads the initial extraction result obtained in step S3 of this embodiment and calculates the comprehensive credibility of the extracted value of each target field.

[0238] In this embodiment, the comprehensive credibility is calculated by weighting and summing the OCR confidence, the pass rate of the format rule, and the pass rate of the cross-field logic rule to obtain the comprehensive credibility.

[0239] Step S4 in this embodiment specifically includes the following steps:

[0240] Step S41: Format rule verification.

[0241] In step S41 of this embodiment, the system performs format validity checks on the extracted values ​​of each target field to determine whether they conform to predefined format constraint rules. Specifically, this includes: verifying whether the field value conforms to the expected type (e.g., whether a numeric field is a number); and performing format matching on the field based on regular expressions (e.g., whether a phone number conforms to fixed length and number segment rules, and whether a waybill number meets a specified encoding structure). For fields that conform to the format rules, the format rule validation pass rate is... Record it as 1.0; otherwise, record it as 0.0.

[0242] Step S42: Cross-field logical rule validation.

[0243] In step S42 of this embodiment, based on the validity of the field format, the system further performs consistency verification on the semantic relationships between multiple fields to detect potential logical conflicts or unreasonable situations. Specifically, the system checks whether there is a reasonable correspondence between name, phone number, and address, and uses a rule base to determine whether the field combination is reasonable (e.g., whether the region and postal code match). For fields that pass the logical validation, the cross-field logical rule validation pass rate is... Record it as 1.0; otherwise, record it as 0.0.

[0244] Step S43: Calculate the overall credibility by weighted summation.

[0245] In step S43 of this embodiment, the system performs a weighted summation of the OCR confidence level, the pass rate of format rules, and the pass rate of cross-field logic rules to obtain the comprehensive confidence level. .

[0246] In this embodiment, the calculation of the overall confidence level includes the OCR confidence level. The weight is 0.20, and the pass rate of the format rule validation is... The weight is 0.20, and the pass rate of the cross-field logical rule validation is... The weight is 0.15; the overall credibility also includes the model generation probability component. The model generates probabilities with a weight of 0.45.

[0247] Overall credibility The calculation formula is:

[0248]

[0249] in, .

[0250] Taking the "recipient's phone number" field in this embodiment as an example: the model generates probability components. The OCR confidence score for the source text block (text block 2) is 0.96. The value is 0.92, and the format rule validation passed (the phone number is an 11-digit string, and the format is valid). Recorded as 1.0, cross-field logic validation passed (the phone number, recipient's name, and recipient's address all belong to the recipient area, and the logic is consistent). Let it be denoted as 1.0. The overall credibility is then calculated as follows:

[0251]

[0252] The overall credibility calculation results for other fields in this embodiment are as follows:

[0253]

[0254] Step S5: When the overall confidence level is lower than the preset threshold, trigger a secondary extraction of the corresponding target field or mark it as a low confidence result.

[0255] In step S5 of this embodiment, the system calculates the overall reliability of each field based on the results obtained in step S4 of this embodiment. Dynamic intervention triggered by execution thresholds.

[0256] In this embodiment, the preset threshold includes a first threshold and a second threshold, and the first threshold is greater than the second threshold; when the overall credibility is greater than or equal to the first threshold, the current extracted value is retained as the final extraction result; when the overall credibility is less than the first threshold but greater than or equal to the second threshold, an error correction prompt carrying candidate evidence of the current target field is constructed, and the error correction prompt is re-inputted into the large language model for secondary reasoning to update the extracted value; when the overall credibility is less than the second threshold, the current target field is marked as awaiting manual review.

[0257] In this embodiment, the first threshold is 0.75, the second threshold is 0.55, the sampling temperature of the secondary inference is lower than the sampling temperature of the first inference, and the number of secondary inferences does not exceed 2.

[0258] In this embodiment, the overall confidence level of all fields is greater than 0.75 (the lowest being 0.929 for the recipient's address), which is higher than the first threshold of 0.75. Therefore, all current extracted values ​​are retained as the final extraction results without triggering secondary reasoning or manual review.

[0259] To fully illustrate the handling process of this invention when the overall confidence level is lower than the first threshold, the "recipient's phone number" field is used as an example: Assuming that the overall confidence level of this field is calculated to be 0.68 in a certain extraction (between 0.55 and 0.75), the system constructs an error correction prompt carrying candidate evidence for the current target field, including a list of candidate texts for the field (including the normalized bounding box coordinates and region labels of each candidate text), OCR confidence level, and the type of verification failure. Then, the error correction prompt is re-input into the large language model for secondary inference. The sampling temperature of the secondary inference (0.05) is lower than that of the first inference (0.1) to reduce the randomness of the output. If the secondary inference result passes the verification and the overall confidence level is raised to above the first threshold, the secondary inference result is adopted. If the result after the secondary inference is still lower than the first threshold but higher than the second threshold, a maximum of 2 secondary inferences are performed. If it still fails, the field is marked as a low-confidence result. Taking the "Waybill Number" field as an example: Suppose that the overall credibility of this field is calculated to be 0.48 in a certain sampling (below the second threshold of 0.55), then the system directly marks this field as needing manual review and carries an anomaly mark in the output results for downstream systems or manual processing.

[0260] In practical applications, the specific values ​​of the first and second thresholds mentioned above can be adjusted according to the different requirements of accuracy and recall in different business scenarios, and are not completely fixed.

[0261] Step S6: Output the final extraction results for each target field.

[0262] In step S6 of this embodiment, the system outputs the structured extraction result after the above steps.

[0263] The final structured extraction result output in this embodiment is as follows:

[0264] { "sender_name": "Zhang San", "recipient_name": "Li Si", "recipient_phone": "138xxxx1234", "recipient_address": "No. XX, XX Street, Chaoyang District, Beijing", "waybill_number": "YT1234567890" }

[0265] Meanwhile, the system associates source evidence binding information with each extracted value. Taking the recipient's phone number field as an example, the evidence binding information includes: source text block identifier (text block 2), normalized bounding box coordinates ([0.08, 0.29, 0.36, 0.34]), source text content ("Phone: 138xxxx1234"), overall credibility (0.966), and model version and Prompt version information, which are used for subsequent auditing and tracing.

[0266] Step S7: Closed-loop feedback and continuous optimization.

[0267] In step S7 of this embodiment, the system returns the abnormal samples that failed the verification to the feedback module, drives the iterative update of Prompt through error pattern clustering analysis, and triggers model fine-tuning when necessary.

[0268] In this embodiment, when the overall credibility is lower than the preset threshold or the verification still fails after secondary reasoning, the extraction anomaly event of the corresponding target field is sent back to the feedback module.

[0269] Step S7 in this embodiment specifically includes the following steps:

[0270] Step S71: Abnormal sample collection and cluster analysis.

[0271] In step S71 of this embodiment, the system collects and records samples that fail validation. Specifically, the system records complete information for each abnormal sample, including: original input data, model output results, fields that failed validation, and corresponding error types. Anomaly labels are automatically generated based on the validation results, including field missing anomalies, invalid format anomalies, and logical conflict anomalies, and the anomaly types are classified and coded for subsequent statistics and analysis.

[0272] The system performs distribution analysis on abnormal samples, identifies high-frequency error patterns, and extracts common problems as optimization criteria, such as frequent missed or incorrect sampling of a certain type of field. Specifically, the system encodes each abnormal sample into an error feature vector, with features including layout identifier, field type, error type, confidence interval, OCR quality level, and version used. Using the most recent 500 samples as a sliding window, the system clusters samples according to the same field and error type and compares them with the previous stable window.

[0273] The feedback module performs error pattern clustering analysis based on the accumulated multiple abnormal events, and updates the field definitions, synonym tags, or format constraint rules in the subsequent structured prompt text based on the clustering results.

[0274] Step S72: Triggering conditions and optimization action selection.

[0275] In step S72 of this embodiment, the system determines whether to trigger optimization and which optimization action to select based on the clustering analysis results. An optimization task is triggered when the field distribution stability index exceeds 0.20, the cumulative number of similar anomalies reaches 50, or the corresponding field's F1 value decreases by more than 2 percentage points. For duplicate samples, only the one with the most complete information is retained.

[0276] Optimization actions are selected based on the main sources of errors identified after clustering: when field definitions are unclear or synonym labels are missing, the field ontology is updated, the field definition is redescribed, and semantic descriptions and synonym labels are added; when the output format is incorrect, JSON Schema constraints are tightened, stricter output constraints are introduced in the Prompt, and the output structure is specified; when candidate regions are incorrect, the edge weights of the layout diagram and the candidate matching weights are adjusted.

[0277] In this embodiment, if the error rate of the same field type continues to decrease within a preset number of statistical windows, the update of the structured prompt text will continue.

[0278] Step S73: Triggering conditions for efficient parameter fine-tuning.

[0279] In this embodiment, if the problem is not resolved within a preset number of statistical windows, then efficient fine-tuning of the parameters of the large language model is initiated.

[0280] Specifically, when a cross-modal recognition error remains unresolved for two consecutive windows, the system enters LoRA incremental fine-tuning. The incremental dataset consists of manually confirmed anomalous samples and an equal number of historical normal samples, with 20% of the old-version samples retained as a replay set to prevent catastrophic forgetting. Low-rank adaptive LoRA is used for efficient parameter fine-tuning, with a low-rank dimension of 16, a scaling factor of 32, a dropout rate of 0.05, and a learning rate of 2×10⁻. 5 The batch size is set to 8, and the training is conducted in 3 rounds. The optimal parameters are selected by combining the field-level F1 score on the independent validation set, the JSON perfect match rate, and the inference latency.

[0281] Step S74: Shadow evaluation and gating release.

[0282] After the prompt update or model fine-tuning is completed, the system is safely deployed using a shadow evaluation and gated deployment strategy. Offline evaluation is first performed on a fixed regression test set and the most recent window samples, followed by online comparative testing with no more than 10% of the traffic. Traffic is only increased if the updated version's field-level F1 score is no lower than the old version, the JSON perfect match rate does not decrease, the error rate of high-risk fields decreases, and the average inference latency increase does not exceed 10%; if any condition fails, the system reverts to the old prompt or old model version.

[0283] Advantages of implementing this invention:

[0284] This embodiment uses abnormal event feedback and error pattern clustering analysis to drive the automatic updating of field definitions, synonym labels and format constraint rules, enabling the system to continuously optimize extraction performance without retraining the model.

[0285] This embodiment achieves precise decision-making and efficient execution of closed-loop optimization by quantifying trigger conditions (stability index exceeding 0.20, cumulative anomalies of 50, and F1 score drop exceeding 2%) and classifying optimization actions (field ontology update, format constraint tightening, and layout weight adjustment).

[0286] This embodiment adopts a hierarchical strategy of "first updating the Prompt / rules, and then triggering LoRA fine-tuning after two consecutive windows have not recovered" to limit most optimization scenarios to lightweight operations that do not require retraining, and only initiate incremental fine-tuning when necessary, which significantly reduces the computing cost of system operation and maintenance.

[0287] This embodiment uses shadow evaluation and gating release mechanisms to conduct offline evaluation and online low-traffic comparison tests before Prompt changes or model updates are launched. This ensures that the performance of the new version is not inferior to the old version and that the error rate of high-risk fields has decreased before full release, effectively guaranteeing the system stability and security during continuous optimization.

[0288] Example 3

[0289] like Figure 6 As shown, the system described in this embodiment includes a preprocessing module M1, a candidate screening module M2, an inference module M3, a credibility assessment module M4, an intervention module M5, and an output module M6, which are connected in sequence and whose data flow is unidirectional or bidirectional.

[0290] The preprocessing module M1 acquires the waybill image and performs OCR recognition to obtain several text blocks and their spatial location information. The spatial location information includes the normalized bounding box coordinates of each text block relative to the page size and the region labels determined through layout analysis. The preprocessing module M1 then transmits the processed text block data and its spatial location information to the candidate filtering module M2.

[0291] The candidate filtering module M2 is used to filter candidate text blocks associated with each target field in the preset target field set, based on the semantic relevance, spatial matching degree, and OCR confidence degree between each text block and the current target field. The candidate filtering module M2 then passes each target field and its corresponding candidate text block to the inference module M3.

[0292] The inference module M3 is used to input the target fields and their corresponding candidate text blocks into the large language model for information extraction, and obtain the initial extraction results. The inference module M3 then passes the initial extraction results to the credibility evaluation module M4.

[0293] The credibility assessment module M4 is used to calculate the overall credibility of the extraction result for each target field in the initial extraction result. The overall credibility is determined based on the OCR confidence of the candidate text block corresponding to the extraction result, the verification result of the extraction result through predefined format rules, and the verification result of the extraction result through predefined cross-field logical rules. The credibility assessment module M4 transmits the overall credibility of each field, along with the initial extraction result, to the intervention module M5.

[0294] Intervention module M5 is used to trigger secondary extraction or marking of the corresponding target field as a low-confidence result when the overall confidence level is lower than a preset threshold. When triggering secondary extraction, intervention module M5 sends a re-inference request to inference module M3 and receives the updated extraction result. Fields that still fail the verification after secondary inference are marked as low-confidence results. Intervention module M5 then transmits the processed final extraction result to output module M6.

[0295] The output module M6 is used to output the final extraction results for each target field. The final extraction results are in JSON format, and the extracted value of each field is associated with source evidence binding information, which includes at least the source text block identifier and normalized bounding box coordinates.

[0296] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for automatically extracting waybill information based on a large language model, characterized in that, include: The waybill image is acquired and OCR recognition is performed to obtain several text blocks and the spatial location information of each text block; For each target field in the preset target field set, candidate text blocks associated with the target field are selected from the text blocks based on the semantic relevance, spatial location matching degree, and OCR confidence degree between each text block and the current target field. Each target field and its corresponding candidate text block are constructed into a structured prompt text, which is then input into a large language model for information extraction to obtain the initial extraction results. The overall credibility of the extraction result for each target field in the initial extraction result is calculated respectively. The overall credibility is determined based on the OCR confidence of the candidate text block corresponding to the extraction result, the pass rate of the extraction result through the predefined format rules, and the pass rate of the extraction result through the predefined cross-field logic rules. When the overall confidence level is lower than a preset threshold, a secondary extraction of the corresponding target field is triggered or it is marked as a low-confidence result. Output the final extraction results for each target field.

2. The method for automatic extraction of waybill information based on a large language model according to claim 1, characterized in that, The selection of candidate text blocks associated with the target field from the plurality of text blocks specifically includes: For any text block, calculate the semantic consistency score between the text block and the current target field, the spatial position consistency score between the spatial position of the text block and the preset target area of ​​the current target field, the tag matching score between the text content of the text block and the synonym tags of the current target field, and the OCR confidence score of the text block. The semantic consistency score, spatial location consistency score, tag matching score, and OCR confidence score are weighted and summed to obtain the matching score between the text block and the current target field. The text blocks are sorted according to the matching scores, and the top K text blocks with the highest scores are selected as candidate text blocks.

3. The method for automatically extracting waybill information based on a large language model according to claim 2, characterized in that, The weight of the semantic consistency score is 0.40, the weight of the spatial location consistency score is 0.25, the weight of the tag matching score is 0.20, and the weight of the OCR confidence score is 0.

15.

4. The method for automatically extracting waybill information based on a large language model according to claim 1, characterized in that, The overall credibility is calculated by weighting and summing the OCR confidence score, the pass rate of the predefined format rules, and the pass rate of the predefined cross-field logic rules to obtain the overall credibility.

5. The method for automatically extracting waybill information based on a large language model according to claim 4, characterized in that, In the calculation of the overall credibility, the weight of the OCR confidence is 0.20, the weight of the predefined format rule verification pass rate is 0.20, and the weight of the predefined cross-field logic rule verification pass rate is 0.15; the overall credibility also includes a model generation probability component, and the weight of the model generation probability is 0.

45.

6. The method for automatically extracting waybill information based on a large language model according to claim 1, characterized in that, The preset threshold includes a first threshold and a second threshold, and the first threshold is greater than the second threshold; When the overall confidence level is greater than or equal to the first threshold, the current extraction value is retained as the final extraction result; When the overall credibility is less than the first threshold and greater than or equal to the second threshold, an error correction prompt carrying the candidate evidence of the current target field is constructed, and the error correction prompt is re-inputted into the large language model for secondary reasoning to update the extracted value. When the overall credibility is less than the second threshold, the current target field is marked as requiring manual review.

7. The method for automatically extracting waybill information based on a large language model according to claim 6, characterized in that, The first threshold is 0.75, and the second threshold is 0.55; the sampling temperature of the secondary inference is lower than the sampling temperature of the first inference, and the number of secondary inferences does not exceed 2.

8. The method for automatically extracting waybill information based on a large language model according to any one of claims 1 to 7, characterized in that, Also includes: When the overall credibility is lower than the preset threshold or the verification still fails after secondary reasoning, the extraction anomaly event of the corresponding target field is sent back to the feedback module. The feedback module performs error pattern clustering analysis based on the accumulated multiple abnormal events, and updates the field definitions, synonym tags, or format constraint rules in the subsequent structured prompt text based on the clustering results.

9. The method for automatically extracting waybill information based on a large language model according to claim 8, characterized in that, After the feedback module performs error pattern clustering analysis: If the error rate of the same field type continues to decrease within a preset number of statistical windows, then the update of the structured prompt text will continue. If the problem does not recover within a preset number of statistical windows, then efficient fine-tuning of the parameters of the large language model will be initiated.

10. A system for automatically extracting waybill information based on a large language model, characterized in that, include: The preprocessing module is used to acquire waybill images and perform OCR recognition to obtain several text blocks and the spatial location information of each text block; The candidate filtering module is used to filter out candidate text blocks associated with the target field from the text blocks for each target field in the preset target field set, based on the semantic relevance, spatial location matching degree and OCR confidence degree between each text block and the current target field. The reasoning module is used to construct structured prompt text from each target field and its corresponding candidate text block, input it into the large language model for information extraction, and obtain the initial extraction result. The credibility assessment module is used to calculate the comprehensive credibility of the extraction result of each target field in the initial extraction result. The comprehensive credibility is determined based on the OCR confidence of the candidate text block corresponding to the extraction result, the pass rate of the extraction result through the predefined format rules, and the pass rate of the extraction result through the predefined cross-field logic rules. The intervention module is used to trigger a secondary extraction of the corresponding target field or mark it as a low-confidence result when the overall confidence level is lower than a preset threshold. The output module is used to output the final extraction results for each target field.

Images