Field recognition method, system, medium and device based on visual language model

Abstract

This disclosure relates to a field recognition method, system, medium, and device based on a visual language model, belonging to the field of document processing technology. The scheme generates a structured output sequence corresponding to a document image through a visual language model in a structured output mode, parses it, and obtains a structured output result. Then, it performs a field-level quality assessment to obtain a target field quality score, which includes parameters such as structure gating and format consistency score. Based on the field recognition output conditions and the target field quality score, it outputs a structured output result or triggers field rejection processing. This scheme, through field-level quality assessment, avoids low recognition accuracy and insufficient reliability caused by document image degradation and model illusions. It requires no model training or fine-tuning, and the specific quality assessment provides interpretability for the structured output or rejection processing, facilitating a trade-off between business risk and accuracy.

Description

Technical Field

[0001] This disclosure relates to the field of document processing technology, and more specifically, to a field recognition method based on a visual language model, a field recognition system based on a visual language model, a computer-readable storage medium, and an electronic device. Background Technology

[0002] In document processing scenarios such as invoices, invoices, contracts, forms, and logistics orders, technologies such as computer vision, natural language processing, information extraction, and model inference are typically involved. Specifically, it requires extracting structured field information from the documents, such as invoice codes or numbers, dates, amounts, tax IDs, and buyer / seller names. Currently, structured fields for recognition can be generated directly from document images end-to-end using visual language models, simplifying the complex processes of traditional OCR recognition, layout analysis, and rule or model extraction.

[0003] However, when document images are degraded, such as due to blurring, occlusion, tilting, low resolution, compression noise, or perspective distortion, visual language models may suffer from insufficient accuracy and reliability in recognizing and outputting fields.

[0004] For field reliability assessment, OCR character confidence, detection score, empirical rules and other solutions can be used. However, these are difficult to apply directly to field recognition in visual language models. They usually require additional model training or fine-tuning and also have the problem of low interpretability, making it difficult to achieve a controllable trade-off between business risks and field recognition accuracy.

[0005] It should be noted that the information in the background section above is only used to enhance the understanding of the background of this disclosure. Therefore, it may include not only information that constitutes prior art known to those skilled in the art, but also information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this disclosure is to provide a field recognition method based on a visual language model, a field recognition system based on a visual language model, a computer-readable storage medium, and an electronic device, thereby overcoming, to at least a certain extent, the problems of recognition accuracy and reliability in field recognition based on visual language models being affected by document image degradation and model illusion. It does not require additional model training or fine-tuning, and has a certain degree of interpretability, making it easy to achieve a controllable trade-off between business risks and field recognition accuracy.

[0007] According to one aspect of this disclosure, a field recognition method based on a visual language model is provided. This method may include: generating a structured output sequence corresponding to a document image using a visual language model in a structured output mode; parsing the structured output sequence to obtain a structured output result; performing field-level quality assessment on the structured output result to obtain a target field quality score; the target field quality score includes at least one of a structure gating quantity and a format consistency score; the structure gating quantity is used to indicate parsing and field consistency anomalies in the structured output result; the format consistency score is used to indicate the legality anomalies of field values ​​in the structured output result for at least one of field types and field constraints; outputting a structured output result if the target field quality score meets the field recognition output conditions; and triggering field rejection processing if the target field quality score does not meet the field recognition conditions.

[0008] According to one aspect of this disclosure, a field recognition system based on a visual language model is provided. This system may include: a model prediction module for generating a structured output sequence corresponding to a document image using a visual language model in a structured output mode; a result parsing module for parsing the structured output sequence to obtain a structured output result; a quality assessment module for performing field-level quality assessment on the structured output result to obtain a target field quality score; the target field quality score includes at least one of a structure gating quantity and a format consistency score; the structure gating quantity is used to indicate parsing and field consistency anomalies in the structured output result; the format consistency score is used to indicate the legality anomalies of field values ​​in the structured output result for at least one of field type and field constraint; a result processing module for outputting a structured output result when the target field quality score meets the field recognition output conditions; and the result processing module is further used to trigger field rejection processing when the target field quality score does not meet the field recognition conditions.

[0009] According to one aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described field recognition method based on a visual language model.

[0010] According to one aspect of this disclosure, an electronic device is provided, comprising: The processor; and memory for storing the processor's executable instructions; The processor is configured to execute the aforementioned field recognition method based on a visual language model by executing executable instructions.

[0011] This disclosure provides a field recognition method based on a visual language model, a field recognition system based on a visual language model, a computer-readable storage medium, and an electronic device. The scheme generates a structured output sequence corresponding to a document image using a visual language model in a structured output mode, then parses it to obtain a structured output result. Field-level quality assessment is performed on the structured output result to obtain a target field quality score. This score includes a structure gating quantity representing parsing and field consistency anomalies in the structured output result, and a format consistency score representing field values ​​in the structured output result for legality anomalies such as field type and field constraints. Based on this, and judging from the field recognition output conditions and the target field quality score, a structured output result is output or a field rejection processing is triggered. This solution performs field-level quality assessment for visual language models' prediction of document images. Depending on actual needs, it can select fields with parsing and consistency anomalies, field values ​​with abnormalities in field type or field constraint validity, etc., thereby avoiding problems such as document image degradation and model illusion that cause low recognition accuracy and insufficient reliability. No model training or fine-tuning is required. Through specific quality assessment, the structural output or rejection handling has a certain degree of interpretability, which facilitates a controllable trade-off between business risks and field recognition accuracy.

[0012] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0014] Figure 1 This illustration shows one of the step flow diagrams of the field recognition method based on a visual language model provided in an embodiment of this disclosure.

[0015] Figure 2 An exemplary schematic diagram of the process for triggering two-stage re-decoding provided in an embodiment of this disclosure is shown.

[0016] Figure 3 This is an exemplary flowchart illustrating the steps of the field recognition method based on a visual language model provided in this embodiment of the present disclosure.

[0017] Figure 4 An exemplary schematic diagram of the structured output sequence weighting provided in an embodiment of this disclosure is shown.

[0018] Figure 5Exemplary embodiments provided by this disclosure are shown. and Comparison diagram.

[0019] Figure 6 A schematic diagram of the calibration curve provided in an embodiment of this disclosure is shown as an example.

[0020] Figure 7 An exemplary diagram illustrates a curve representing the field acceptance rate versus the field error rate provided in an embodiment of this disclosure.

[0021] Figure 8 An exemplary schematic diagram of the process for locating and rereading the region of interest provided in an embodiment of this disclosure is shown.

[0022] Figure 9 An exemplary structural diagram of a field recognition system based on a visual language model provided in an embodiment of this disclosure is shown.

[0023] Figure 10 An exemplary diagram illustrates the specific implementation architecture of the field recognition system based on a visual language model provided in this disclosure.

[0024] Figure 11 An electronic device for implementing a field recognition method based on a visual language model is illustrated by example of an embodiment of this disclosure. Detailed Implementation

[0025] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0026] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0027] Currently, when using the Vision Language Model (VLM) to extract key information (KIE) from document images, issues such as model illusion and document image degradation may cause unreliable output for some fields, including but not limited to at least one of the following: The field values ​​meet the rules, format, and other requirements, but are inconsistent with the actual content of the document image; for example, the number format is correct but there are inconsistent characters. Generate values ​​for fields that do not exist; Generate non-existent fields or additional fields; The structured output sequence can be parsed, but the field semantics are incorrect.

[0028] This disclosure provides a field recognition method based on a visual language model, which can perform field-level quality assessment of the visual language model generation process. It overcomes, to some extent, the problems in field recognition using visual language models where accuracy and reliability are affected by document image degradation and model illusion. It requires no additional model training or fine-tuning and has a degree of interpretability, facilitating a controllable trade-off between business risks and field recognition accuracy. Specifically, see the accompanying figures below.

[0029] Figure 1 This is one of the flowcharts illustrating the steps of a field recognition method based on a visual language model provided in an embodiment of this disclosure. Figure 1 As shown, the method may include steps 101 to 105 as follows.

[0030] Step 101: In structured output mode, generate the structured output sequence corresponding to the document image through the visual language model.

[0031] In this embodiment, the structured output mode is used to constrain the generation process of the visual language model, ensuring that the objects and content extracted by the visual language model conform to the requirements defined by the structured output mode, thus achieving constrained decoding. The structured output mode can be configured before field recognition begins, or it can be adjusted during field recognition based on business needs and model feedback.

[0032] In an optional embodiment of the method disclosed herein, the structured output mode includes at least one of structure syntax constraints, field set constraints, field type constraints, and field value format constraints.

[0033] In this embodiment of the disclosure, the constraints imposed by the structured output mode on the generation of the visual language model may include structural grammar constraints to restrict the model to output recognition results in a structured grammar; field set constraints to prevent the model from producing phantom keys outside the document image or omitting fields to be recognized in the document image; field type constraints to prevent the recognition results output by the model from not matching the corresponding field type; and field value format constraints to restrict the field value format output by the model to conform to the format requirements of the corresponding field. The above-described structured output mode is for illustrative purposes only, and those skilled in the art can select and expand upon it according to actual needs. This embodiment of the disclosure does not impose specific limitations in this regard.

[0034] In this embodiment of the disclosure, under the constraint of a structured output mode, one or more visual language models can be invoked. The document image to be recognized is input, and the visual language model performs information extraction, token segmentation, and prediction to obtain a structured output sequence. This structured output sequence can refer to a structured character sequence output based on the document image, such as semantic characters, structural characters, and other special characters.

[0035] Step 102: Parse the structured output sequence to obtain the structured output result.

[0036] In this embodiment of the disclosure, based on the obtained structured output sequence, characters can be parsed to determine possible fields and their corresponding values, resulting in a structured output result organized in a structured manner. The content, type, format, and relationship of the fields and field values ​​should conform to the constraints of the aforementioned structured output pattern. In an optional embodiment of the method of this disclosure, if the aforementioned parsing fails or is inconsistent with the structured output pattern, the visual language model can be regenerated; furthermore, during regeneration, the structured output pattern can be adjusted, employing more lenient or more stringent restricted decoding constraints.

[0037] Step 103: Perform field-level quality assessment on the structured output results to obtain the target field quality score; The target field quality score includes at least one of the following: structure gating quantity and format consistency score; The structure gating quantity is used to indicate parsing and field consistency anomalies in the structured output results; The format consistency score is used to indicate the legality anomalies of field values ​​in the structured output results for at least one of the following: field type and field constraint.

[0038] In this embodiment, the obtained structured output results are evaluated at the field level to determine whether they meet business quality requirements, thereby informing subsequent execution steps. The field-level quality evaluation can be a thorough and comprehensive assessment of quality within and between fields. The target field quality score obtained from the field-level quality evaluation can include one of structural gating parameters and format consistency scores, or a fusion of structural gating parameters and format consistency scores. Among them, the structure gating quantity can be obtained by verifying and evaluating the parsing anomalies and field consistency of the structured output results to determine whether there are inconsistencies such as whether the parsing was successful, whether there are duplicates, illusions, or omissions in the fields; the format consistency score can be used to verify and evaluate the legality of the field values ​​of the structured output results, such as whether the field value type does not meet the field type requirements, or whether the field value content does not meet the field constraint requirements. Field types can include integer, decimal, enumeration, string, and date / time types. Field constraints can include constraints such as the range of values ​​within the field, character length, regular expression relationships, check bits, check sums, etc., and can also include constraints on the logical relationships between fields, such as the consistency of uppercase and lowercase amounts, the ratio of tax amount to amount, and the logical order of dates.

[0039] The structural gating quantity and format consistency score in the above target field quality score are only used as examples. Those skilled in the art can select and expand them according to actual needs. This disclosure does not impose specific limitations on them.

[0040] Step 104: If the quality score of the target field meets the field recognition output conditions, output the structured output result.

[0041] Step 105: If the quality score of the target field does not meet the field recognition conditions, trigger the field rejection process.

[0042] In this embodiment, the field identification output conditions can be configured based on factors such as business quality requirements, data processing conditions, and the calculation method of the target field quality score. This allows for decisions on the execution of subsequent steps based on the field-level quality feedback of the model identification process and results according to the target field quality score. For example, if the target field quality score meets the field identification output conditions, it indicates that the field-level quality feedback of the model identification process and results meets the business quality requirements, and the structured output result can be output to the requester. If the target field quality score does not meet the field identification output conditions, it indicates that the field-level quality feedback of the model identification process and results does not meet the business quality requirements, thereby triggering field rejection processing.

[0043] In this embodiment of the disclosure, the target field quality score may include a single score, several different scores, or a fusion of several different scores. The field identification output conditions may set thresholds, attribution intervals, or other judgment conditions for a single score or a fusion of several different scores under the target field quality score, or may set corresponding thresholds, attribution intervals, or other judgment conditions for several different scores under the target field quality score, and then combine the judgment results of different scores to decide on subsequent steps; or, the aforementioned schemes may be combined among several different scores.

[0044] In this embodiment of the disclosure, the field rejection processing includes rejecting the structured output results whose target field quality scores do not meet the field recognition conditions, and may also include further processing, such as quality problem analysis, problem improvement suggestions, generation parameter adjustment, model re-recognition, etc.

[0045] In this embodiment of the disclosure, document images can be obtained when extracting key information. This allows for input into a visual language model; it also allows for configuration of field sets. Each field At least include field keys Field type Field constraints Field threshold This allows for subsequent consistency and validity checks. Visual language models can extract structured output sequences from document images. The structured output sequence is parsed to obtain the structured output result. From this field The field values ​​in the structured output can be denoted as: .

[0046] In an optional embodiment of the method disclosed herein, the target field quality score includes a structural gating quantity. In this embodiment of the disclosure, the target field quality score may include a structure gating quantity, and may further include other scores. The structure gating quantity may be obtained by parsing and field consistency auditing, and may be applied to the field identification output condition decision, or to the field identification output condition decision after score fusion, thereby triggering the execution of subsequent steps.

[0047] In an optional embodiment of the method disclosed herein, the structural gating quantity can be configured with corresponding value rules to determine the value of the corresponding structural gating quantity based on different audit results. For example, the value rules for the structural gating quantity can be as follows: When parsing the structured output sequence fails, the structure gate variable is set to its first value. When there are unknown field keys in the structured output, the structure gating variable is the second value; When there are duplicate field keys in the structured output, the structure gating value is the third value. When a required field key is missing in the structured output, the structure gating value is the fourth value. When only field key spelling errors exist in the structured output, the structure gating value is the fifth value. When there are no anomalies in the parsing and consistency of the structured output, the structure gating variable is the sixth value. Among them, the first value is the smallest, and the sixth value is the largest.

[0048] In this embodiment, the structural gating quantity can be configured with value rules for several scenarios, including parsing failure, presence of unknown field keys, presence of duplicate field keys, presence of missing required field keys, presence of field key spelling errors, and no parsing and consistency errors. These scenarios are merely examples. Based on this, a minimum first value is set for parsing failure, and a maximum sixth value is set for parsing and consistency errors. For scenarios such as presence of unknown field keys, duplicate field keys, missing required field keys, and field key spelling errors, second, third, fourth, and fifth values ​​are set respectively within the range of the first value and the sixth value. The magnitude of these values ​​is not limited, thus distinguishing different abnormal situations. Simultaneously, the lowest first value indicates the lowest quality assessment and the highest degree of abnormality; the highest sixth value indicates the highest quality assessment and no abnormality. The magnitudes of other values ​​can be set according to actual needs.

[0049] For example, the structural gating quantity is Then the rule for taking the value can be expressed as: When parsing the structured output sequence fails, ; For unknown field keys, constraints in a visual language model can include a whitelist of field key sets. The range of fields used to constrain the identification and extraction of data exists. hour, ; For duplicate field keys, then in any , When they are the same or similar, ; For required field keys, the constraints of a visual language model can include the set of required keys. The key information extracted for constraint field value identification is then... There exists hour, ; For field key spelling errors, for any field key k If it exists The normalized edit distance is satisfied by the following formula (1): (1) In the aforementioned formula (1), To set a threshold for spelling errors, if For the minimum bond length, then ,and If so, it is determined that there is a spelling error in the field key. .

[0050] When there are no anomalies in the parsing and consistency of the structured output results, ; in, For example, , , , Desirable to .

[0051] The values ​​0 and 1 mentioned above are for illustrative purposes only. Those skilled in the art can set any different values ​​that satisfy the aforementioned size relationship according to actual business needs and data processing conditions. This disclosure does not impose any specific restrictions on this.

[0052] Based on the above rules for determining the values ​​of structural gating quantities, step 103 can determine the structural gating quantities for parsing and consistency auditing, which can then be further applied to subsequent steps in the decision-making process.

[0053] In an optional method embodiment of this disclosure, the triggering condition for field rejection processing when the target field quality score does not meet the field identification conditions includes the structure gating value being less than the sixth value.

[0054] In this embodiment, the sixth value corresponds to the case where there are no parsing and consistency anomalies, and this value is the largest. Therefore, field rejection processing can be triggered when the structural gating quantity is less than the sixth value, as in the aforementioned value rules. If 0 < 1, rejection processing is triggered, which can result in rejecting the output this time and regenerating the structured output result.

[0055] For example, Figure 2 A schematic diagram of the process for triggering two-stage re-decoding according to an embodiment of this disclosure is shown. , , , Desirable to After parsing the structured output sequence to obtain the structured output result, the value of the structure gating variable is judged. If parsing fails, If parsing is successful, check the unknown field key; if it exists... When the unknown field key If no unknown field key exists, perform a duplicate field key check; if a duplicate field key exists... If there are no duplicate field keys, a check for missing required field keys will be performed. There exists hour, If the field key does not exist, a field key spelling error check will be performed; if any field key exists... k When determining the presence of a field key spelling error using the aforementioned formula (1) When there is no field key spelling error Based on this, according to The value of , If the value is less than 1, an exception type is output, and a two-stage re-decoding is triggered to regenerate the structured output sequence. Y .

[0056] In an optional method embodiment of this disclosure, the structural gating quantity includes a global structural gating quantity uniformly applied to the field.

[0057] In this embodiment of the disclosure, the structural gating variable can be uniformly applied to all fields, thereby serving as a global structural gating variable. , indicating anomalies in the overall structured output.

[0058] In an optional method embodiment of this disclosure, the structural gating quantity includes a field-level gating quantity applied to fields that exhibit parsing and consistency anomalies.

[0059] In this embodiment of the disclosure, the structural gating variable can be applied separately to fields with parsing and consistency anomalies, thereby serving as a field-level gating variable. , indicating specific anomalies in the structured output.

[0060] In an optional method embodiment of this disclosure, the target field quality score includes a format consistency score.

[0061] In this embodiment of the disclosure, the target field quality score may include a format consistency score, and may further include other scores, such as the aforementioned structural gating quantity. The format consistency score can be obtained by validating the validity of field values, and can be applied to the field identification output condition decision, or to the field identification output condition decision after score fusion, thereby triggering the execution of subsequent steps.

[0062] Based on the format consistency score, step 103 above includes at least step A.

[0063] Step A: In the structured output, perform a validity check on the field types and / or field constraints to obtain a format consistency score.

[0064] In this embodiment, field types and constraints can be configured for each field. For example, in the aforementioned field set, each field includes field types and constraints. Specific details can be found in the description of step 103 above; to avoid repetition, they will not be repeated here. In the structured output, the format of the field values ​​is validated for consistency based on the validation rules provided by the field types and constraints. For example, whether the field value type conforms to the validation rules for integer, decimal, enumeration, string, and date / time fields; whether the field value's range, length, and association logic with another field value conform to the validation rules; and, for example, whether the format consistency score is valid. The range of values ​​can be The lower the score, the lower the format consistency, and vice versa.

[0065] In an optional method embodiment of this disclosure, the aforementioned legality verification may include hard verification based on all rules, soft verification based on different subsets of rules, and verification based on consistency constraints between fields.

[0066] Hard validation, based on all rules, assigns the maximum format consistency score when all rules are met and the minimum score when no rule is met. Soft validation, based on different rule subsets, breaks down all rules into subsets such as value range, character length, regular expression relationships, checksums, and checkpoints. And calculate the hit score for each rule subset. After fusion, the format consistency score is obtained, as shown in the following formula (2): (2) As shown in formula (2) above, the hit scores of each rule subset are weighted and fused. The fusion weight corresponding to each rule subset is greater than or equal to 0, and the sum of the fusion weights is zero.

[0067] When there are consistency constraints with logical relationships between fields, such as requiring A cost and B cost to be case-sensitive, or requiring period C to precede period D, other format consistency scores will be obtained. and format consistency score for consistency between fields. When the weighted fusion is performed, the following formula (3) can be used for calculation: (3) Thus, the format consistency score after fusion is obtained. .

[0068] It should be noted that, without departing from the above technical principles, the calculation method for format consistency score is not limited to the above formula. Other similar consistency score calculation methods and score fusion methods can also be used to achieve the same or similar technical effects.

[0069] In an optional embodiment of the method disclosed herein, the target field quality score further includes at least the initial field quality score.

[0070] In this embodiment of the disclosure, the target field quality score may include an initial field quality score, or further include other scores, such as the aforementioned structure gating quantity and format consistency score. The initial quality score may be obtained from decoding uncertainty, coverage weight, etc., and may be applied to the field recognition output condition decision, or to the field recognition output condition decision after score fusion, thereby triggering the execution of subsequent steps.

[0071] For example, the target field is the quality score. Including initial field quality score Format consistency score With structural gating quantity The following formula (4) can be used to calculate: (4) Formula (4) above shows one fusion calculation method. Those skilled in the art can also choose other fusion methods according to actual needs, such as summation, averaging, and weights can be set for weighted fusion. This disclosure does not impose specific restrictions on this.

[0072] In an optional embodiment of the method disclosed herein, the obtained target field quality score can also be... Perform field-level calibration to obtain And select field thresholds based on calibration results. The system performs field identification and output condition checks to meet risk assessment needs or target coverage. For example, field thresholds can be configured based on field type; for instance, the threshold for short fields such as invoice code and tax number can be greater than the threshold for longer fields such as company name and address.

[0073] Figure 3 A flowchart illustrating the steps for calculating the initial field quality score provided in this embodiment of the disclosure is shown below. Figure 3 As shown, it may include the following steps 301 to 303.

[0074] Step 301: For each field, determine the character range of the field value in the structured output sequence, and determine the character range of the corresponding word in the structured output sequence.

[0075] In this embodiment of the disclosure, when the target quality score includes an initial quality score, the calculation of the initial quality score can be performed based on the structured output sequence. Specifically, for each field, the character range of the field value can be determined in the structured output sequence, and the character range of the corresponding word in the structured output sequence can be determined. A field-based mapping relationship can be established between the field value character range and the word character range, thereby achieving field alignment. In an optional method embodiment of this disclosure, field alignment can be achieved in any of the following ways: (1) Insert field boundary markers and remove them during post-processing; (2) Structure the output sequence Parse into an AST (Abstract Syntax Tree), and obtain the character range of field values ​​and the corresponding word character range of the mapped fields based on the AST; (3) Locate the character range of the field value based on the field path or JSON Pointer, and the character range of the corresponding word in the mapped field.

[0076] For example, this could be done by targeting a field and determining the field value within the structured output sequence. The character range of the field value in the data is and the token field in The range of metacharacters in the text is .

[0077] Step 302: Calculate the coverage weight of the field value character range relative to the word meta character range.

[0078] In this embodiment of the disclosure, based on determining the field value character range and the word character range, the coverage weight of the field value character range relative to the word character range can be calculated, that is, the ratio of the intersection of the field value character range and the word character range to the word character range. For example, assuming that the word character range is completely outside the field value character range, the coverage weight is zero, which can be expressed as: Assuming the entire range of lexical characters lies within the range of characters in the field value, the coverage weight is 1, which can be represented as: Assuming the lexical character range covers both the field value character range and other characters, such as structural symbols, the coverage weight can take a value between zero and 1, and can be represented as follows: Therefore, by using coverage weights, word character ranges containing other characters can be weighted less, or word character ranges that do not overlap with the field value character range can be masked with a coverage weight of zero.

[0079] For example, coverage weight It can be calculated using the following formula (5): (5) in, .

[0080] Therefore, the field lexical set of a visual language model can be represented as .

[0081] In an optional embodiment of the method disclosed herein, step B may be included before step 202.

[0082] Step B: Reduce the weight of special characters within the lexical character range.

[0083] In this embodiment of the disclosure, when special characters exist within the lexical character range, these special characters can be weighted down, such as by setting an additional weight of zero for masking, or by setting an additional weight less than or equal to 1 for weight reduction. Special characters may include quotation marks, decimal points, separators, spaces, etc. For example, additional weights can be set for decimal points in monetary amounts, thousands separators, etc. Thus, coverage weights and additional weights can be combined into one. For example, whitespace, quotation marks, separators, etc. Desirable to Decimal points, thousands separators, etc. Desirable to .

[0084] Figure 4 A schematic flowchart illustrating the weighted structured output sequence provided in an embodiment of this disclosure is shown. Figure 4 As shown, based on the structured output sequence {"invoice_no":"12345678","amount":"123.45"}, the character range of the field values ​​is determined. and the metacharacter range of the token The token sequence is weighted accordingly. For example, the weights of {"invoice_no":"}, {","amount":"}, and {"} are reset to 0, the weight of {12345678} is reset to 1, and the weight of {123.45} is reset to between 0 and 1, thus obtaining the weighted set of field tokens. .

[0085] Step 303: Calculate the initial field quality score of the prediction field using coverage weights and decoding uncertainty; decoding uncertainty is used to represent the word-level prediction uncertainty of the visual language model in the process of generating structured output sequences.

[0086] In this embodiment, decoding uncertainty can characterize the accuracy, reliability, and stability of word-level recognition during the generation of structured output sequences in the visual language model decoding process. Therefore, based on the aforementioned coverage weights, decoding uncertainty can be used to statistically analyze the initial field quality score of the predicted field. The initial quality score can be obtained through word-level statistical aggregation at the field dimension, thereby reflecting the reliability of the field under document image degradation conditions, and ultimately determining subsequent output or rejection processing. The decoding uncertainty can be represented by the prediction probability provided by the visual language model, or it can be approximated using other indicators.

[0087] In an optional embodiment of the method disclosed herein, the aforementioned decoding uncertainty may include at least one of lexical probability, lexical log probability, lexical entropy, and lexical approximate entropy.

[0088] In this embodiment of the disclosure, during the decoding process of the structured output sequence generated by the visual language model, lexical-level statistics can be recorded to characterize the decoding uncertainty in the model's inference process. For example, this could include the lexical probability of each lexical step. It can also include the log probability of lexical units. It can also include lexical entropy. or top-order entropy of lexical units Calculate the approximate entropy of lexical units using distribution. Among them, lexical entropy The following formula (6) can be used for calculation: (6) In the above formula (6), the lexical entropy is calculated by the lexical probability and the lexical log probability.

[0089] You can also use top- Lexical approximate entropy The following formula (7) is used for calculation: (7) In the above formula (7), K The value can be chosen arbitrarily according to actual needs. For example, Can be taken For example, take .

[0090] In one optional embodiment of the method disclosed herein, there may also be a probability that the visual language model cannot return the word at each step. or top- When using the distribution to calculate lexical entropy or lexical approximate entropy, other alternative indicators can be used to represent decoding uncertainty. For example, the same document image at different temperatures can be calculated using the following formula (8). downsampling The candidate set of fields is obtained this time. The sampling consistency rate is used as a measure of decoding uncertainty. (8) For example, in the above formula (8) Can be taken .

[0091] Furthermore, the target field quality score can be defined as Alternatively, it can be combined with other indicators, for example, the following formula (9) can be used for fusion calculation: (9) in, A quality assessment index at the lexical level; .

[0092] Alternatively, consistency voting scores can be obtained by voting on the consistency of fields obtained from various document image decoding strategies, such as greedy algorithms, bundle search, and sampling, to obtain a quality score; or, external discriminators and verifiers can be introduced, such as using OCR (Optical Character Recognition) or ROI (Region of Interest) to determine the consistency score between its field values ​​and document image evidence, as a decoding uncertainty.

[0093] In an optional embodiment of the method disclosed herein, step 203 may also include steps C1 to C2.

[0094] Step C1: Calculate field-level statistical indicators based on coverage weights and decoding uncertainty; field-level statistical indicators include at least one of field weighted average log probability, field weighted average entropy, and field minimum log probability.

[0095] Step C2: Calculate the initial field quality score of the prediction field based on field-level statistical indicators.

[0096] In this embodiment, field-level statistical indicators can be calculated first from coverage weights and decoding uncertainty, and then the initial field quality score can be calculated further using the field-level statistical indicators. Field-level statistical indicators can be obtained by aggregating or otherwise processing the lexical-level coverage weights and decoding uncertainty. For example, let the field-level coverage weights be as shown in formula (10): (10) Based on the aforementioned formula (10), when the field-level statistical indicator is the field-weighted average log probability, the following formula (11) can be used to calculate it: (11) in, denoted as the log-probability of a word.

[0097] Alternatively, based on the aforementioned formula (10), when the field-level statistical indicator is the field-weighted average entropy, the following formula (12) can be used to calculate it: (12) in, It can be by Replacement.

[0098] Alternatively, based on the aforementioned formula (10), when the field-level statistical indicator is the minimum logarithmic probability of the field, the following formula (13) can be used to calculate it: (13) in, denoted as the log-probability of a word.

[0099] In an optional embodiment of the method disclosed herein, the initial field quality score is calculated by calculating a mapping based on the field weighted average log probability and the field weighted average entropy to obtain the initial field quality score.

[0100] In this embodiment of the disclosure, when calculating the initial field quality score, it can be based on and Calculate the mapping, and Normalization to The mapping can be a non-linear activation sigmoid function, calculated as shown in formula (14): (14) in, , ; As a bias term, it can be... The value can also be determined by fitting a validation set.

[0101] In an optional embodiment of the method disclosed herein, the initial field quality score is calculated by means of: for short fields whose field length is less than or equal to the short field length threshold, a mapping is calculated based on the field weighted average log probability, the field weighted average entropy, and the field minimum log probability to obtain the initial field quality score.

[0102] In this embodiment, fields with a length less than or equal to the short field length threshold can be considered short fields, such as codes, numbers, tax numbers, dates, etc. The specific short field length threshold can be set according to actual needs. Therefore, the minimum logarithmic probability of the field can be introduced for calculation, such as using the following formula (15) to calculate the conservative mapping: (15) in, Suitable for short fields.

[0103] In an optional embodiment of the method disclosed herein, the initial field quality score is calculated by calculating the linear score of the segments based on the field weighted average log probability and the field weighted average entropy, thereby obtaining the initial field quality score.

[0104] In this embodiment of the disclosure, the linear quality score of the segmented pruning can also be calculated based on the field weighted average log probability and the field weighted average entropy to obtain the initial field quality score. For example, the linear segmentation can be calculated using the following formula (16):

[0105] in, , Clip the value to .

[0106] In an optional method embodiment of this disclosure, the above These parameters can be determined through temperature scaling, isotonic regression, curve fitting, or minimizing the field-level Brier score on the validation set to calculate initial field quality scores for different types of fields. For example, temperature scaling can learn scalar parameters. Make Isochronous regression can learn a monotonic piecewise function that matches the output field quality score to the actual accuracy. Curve fitting can be determined on the validation set using Bayesian optimization. Parameters such as these.

[0107] For example, the above Short fields are acceptable , , For long fields, this can reduce... Or take To weaken Influence.

[0108] It should be noted that the calculation method for field quality scores is not limited to the above formula. Other equivalent uncertain aggregation and mapping methods can also be used to achieve the same or similar technical effects.

[0109] In this embodiment of the disclosure, when calculating the field quality score, the decoding uncertainty can be verified first to reduce the impact of errors and improve the effectiveness of quality analysis and evaluation. Figure 5 The embodiments provided in this disclosure are shown. and Comparison diagram, in which K =20, representing the original entropy. With approximate entropy The curves showing the comparison between the two values ​​are: the vertical axis represents the entropy value, and the horizontal axis represents the token index. Figure 6 This diagram illustrates a calibration curve provided in an embodiment of the present disclosure. The vertical axis represents the empirical accuracy, and the horizontal axis represents the prediction confidence, demonstrating the calibration mapping. The shape of the function, under the ideal curve standard, is the change of the curve from before calibration to after calibration.

[0110] In an optional embodiment of the method disclosed herein, the target field quality score not meeting the field identification condition includes the target field quality score being less than the field threshold.

[0111] In this embodiment of the disclosure, the target field quality score can be used for decision-making. With field threshold Comparisons are made. For example, if the target field's quality score does not meet the field's identification criteria, it can be represented as... Thus, for the field Trigger field rejection handling; conversely, When the target field quality score meets the field recognition criteria, the structured output result can be output. The target quality score may be influenced by factors such as structure gating, field uncertainty, field length, character content, and inter-field consistency. These influencing factors can also be used individually as decision conditions, such as... When the field length is abnormal, the character content is abnormal, or the consistency check between fields fails, the field... Trigger field rejection handling.

[0112] In this embodiment of the disclosure, when the field thresholds are different, there is a certain balance between the field error rate and the field acceptance rate. The corresponding field threshold can be selected for different document images and fields to be identified according to actual needs. Figure 7 This illustration shows a graph illustrating the field acceptance rate versus field error rate provided in an embodiment of this disclosure. Figure 7 As shown, the risk upper limit of the field error rate is represented by a dashed line. The short field curve A, which has a relatively higher risk, includes fields such as tax ID, amount, date, and number. The long text field curve B includes fields such as name and address. Based on this, the field threshold of the short field can be selected according to the position where the short field curve A intersects with the dashed line, and the field threshold of the long field can be selected according to the position where the long field curve B intersects with the dashed line, so as to balance the business requirements of field error rate and field acceptance ratio.

[0113] In an optional method embodiment of this disclosure, the target field quality score not meeting the field identification conditions includes an anomaly in the legality check of the format consistency score.

[0114] In this embodiment of the disclosure, when making decisions, the abnormal situations represented by the format consistency score can be analyzed. If an anomaly is found in the legality check, it can be considered that the target field's quality score does not meet the field identification conditions, thereby affecting the field. Triggering field rejection processing; conversely, only when the format consistency score indicates that all valid validations are normal can the target field's quality score be considered to meet the field recognition criteria, thus outputting the structured output result. For example, or Time to field Trigger field rejection processing. The structured output result is output at that time.

[0115] In an optional embodiment of the method disclosed herein, the triggering condition for field rejection processing may further include the presence of at least one risk field in the structured output result.

[0116] In this embodiment of the disclosure, risk fields can also be set, such as tax ID, amount, date, number type, etc. These fields are included in the structured output results. Belongs to risk field If this occurs, the structured output result can be considered to have a certain risk, thus triggering field rejection processing.

[0117] In an optional implementation of this disclosure, the field rejection handling includes any one of the following: setting the output to null, outputting a candidate, triggering two-stage re-decoding, or triggering a fallback link verification.

[0118] In this embodiment of the disclosure, the field rejection handling can be implemented in different ways. For example, it can be outputting empty, that is, the recognition output of the document image is empty, indicating that the recognition failed this time; it can be outputting candidates, that is, outputting multiple candidate recognition results for the document image, and can also mark the reliability, confidence, etc. of the candidate results for the requester's reference; it can be triggering two-stage re-decoding, that is, the visual language model re-recognizes and decodes the document image; it can also be triggering fallback link verification, that is, through OCR recognition and rule judgment, KIE (Key Information Extraction) model or retrieval verification, the consistency of the fallback result is verified, and further processing is carried out according to the verification result.

[0119] In an optional embodiment of the method disclosed herein, the field rejection processing includes triggering a two-stage re-decoding, and the termination condition for the two-stage re-decoding includes any one of the following: The number of retries for two-phase decoding is greater than or equal to the maximum number of retries; The computation time for two-stage re-decoding is greater than or equal to the delay budget; The computational cost of two-stage re-decoding is greater than or equal to the computational budget; The improvement in the quality score of the target field is less than the quality improvement threshold; The target field quality score meets the field identification output conditions.

[0120] In this embodiment, the two-stage re-decoding can be repeatedly executed in a loop if the termination condition is not met, and the execution will terminate when the termination condition is met. The termination condition can be set based on the quality score of the target field that triggered the two-stage re-decoding, the process of the two-stage re-decoding, the result of the two-stage re-decoding, etc., and can also take into account the efficiency requirements and calculation conditions of document image recognition.

[0121] Specifically, termination conditions may include the number of retries in the repeated loop being greater than or equal to the maximum number of retries. If a structured output that meets the field recognition output conditions is not obtained after multiple repetitions, a maximum number of retries can be set to terminate the two-stage re-decoding in a timely manner to avoid wasting ineffective time resources and reduce the impact on other inference tasks. If the number of retries is greater than or equal to the maximum number of retries, after terminating the two-stage re-decoding, the output can be left blank, candidate outputs can be selected based on field quality scores during multiple retries, or the recognition result of the last retry can be output, etc. For example, the maximum number of retries... Can be taken .

[0122] Termination conditions may include computation time greater than or equal to the delay budget. The latency budget can include a pre-allocated delay time for the field recognition task of document images, used to control the time consumption of document image recognition and avoid affecting the overall business efficiency. The latency budget can be set according to the overall business efficiency requirements, specific recognition efficiency requirements, etc. The calculation time can start from the execution of the second-stage re-decoding and be forcibly terminated when the time is greater than or equal to the latency budget; or, during repeated loop execution, the cumulative time of the already executed second-stage re-decoding can be determined, and the estimated time of the next second-stage re-decoding can be determined. The execution of the next second-stage re-decoding can be forcibly terminated when the sum of the cumulative time and the estimated time is greater than or equal to the latency budget. After terminating the second-stage re-decoding, the output can be left blank, or candidate outputs can be selected based on the field quality score in multiple retries, or the recognition result of the last retry can be output, etc.

[0123] Termination conditions may include computational cost being greater than or equal to computational budget. The computational budget can include pre-allocated computational costs for document image field recognition tasks, used to control resource usage and consumption in document image recognition, avoiding impact on overall business power consumption and efficiency. The computational budget can be based on the occupancy and idle status of computational resources, the computational resources required for the field recognition task, the amount and duration of available computational resources, etc. The computational cost can accumulate from the start of the two-stage re-decoding execution, and be forcibly terminated when the accumulated computational cost is greater than or equal to the computational budget; alternatively, during repeated loop execution, the accumulated cost can be determined for the executed two-stage re-decoding, and the estimated cost for the next two-stage re-decoding can be determined, with the next two-stage re-decoding execution forcibly terminated when the sum of the accumulated cost and the estimated cost is greater than or equal to the computational budget. After terminating the two-stage re-decoding, the output can be left blank, candidate outputs can be selected based on field quality scores during multiple retries, or the recognition result of the last retry can be output, etc.

[0124] Termination conditions can include an improvement in the target field quality score that is less than a quality improvement threshold. Since two-stage re-decoding is triggered when the target field quality score no longer meets the field recognition output conditions, the improvement effect on the target field quality score can be evaluated after two-stage re-decoding to analyze whether to terminate it. Specifically, this could be the improvement in the target field quality score in adjacent loops after multiple repeated iterations, or the improvement in the target field quality score relative to the first iteration in each repeated loop. The quality improvement threshold can be used to evaluate the magnitude of change in the target field quality score. When the improvement in the target field quality score is greater than or equal to the quality improvement threshold, the two-stage re-decoding can be considered to have achieved effective field recognition quality improvement; conversely, when the target field quality score is less than the quality improvement threshold, i.e., ... If the improvement in field recognition quality after the second-stage re-decoding is deemed too low, and continuing the loop does not yield the expected results, then the second-stage re-decoding can be terminated. After terminating the second-stage re-decoding, the output can be left blank, candidate outputs can be selected based on the field quality score during multiple retries, or the recognition result of the last retrieval can be output, etc. For example, a quality improvement threshold... Can be taken to .

[0125] Termination conditions may include the target field quality score meeting the field recognition output condition. Since two-stage re-decoding is triggered when the target field quality score does not meet the field recognition output condition, the target field quality score will meet the field recognition output condition after two-stage re-decoding. When the two-stage re-decoding is terminated, the structured output result obtained in that iteration can be output.

[0126] In an optional method embodiment of this disclosure, the field rejection processing includes triggering two-stage re-decoding, which includes at least one of dynamic adjustment constraint re-decoding, local re-decoding, and region of interest re-decoding.

[0127] In this embodiment of the disclosure, after triggering the two-stage re-decoding, the execution method of the two-stage re-decoding can also be selected according to actual business needs and resource conditions. Specifically, the execution method of the two-stage re-decoding can be dynamically adjusted constraint re-decoding. In the loop repetition of the two-stage re-decoding, the rule constraint strength can be increased, such as adjusting from a loose regular expression to a strict regular expression; the character set range can also be narrowed, such as adjusting the tax number to only allow the output of numbers and uppercase letters; the visual language model can also be adjusted, such as lowering the sampling temperature T, adopting greedy search, bundle search, etc.

[0128] The two-stage re-decoding can be performed as partial re-decoding, that is, only regenerating the target field. By keeping other fields unchanged and performing targeted re-decoding in a localized manner, the quality of field recognition can be improved, while reducing computational resource pressure and increasing re-decoding efficiency. Specifically, the path of the target field can be located based on AST, JSON Pointer, etc., and then restricted generation can be performed on that path. Alternatively, the identified target field can be locked as context, and the visual language model can be required to output only that target field.

[0129] The two-stage re-decoding can be performed as region of interest (ROI) re-decoding, which involves rereading the ROI regions in the document image. Targeted rereading of key information within the ROI region can improve field recognition quality, reduce computational resource pressure, and increase re-decoding efficiency. Specifically, during ROI re-decoding, the field ROI region can be determined through OCR alignment. For example, a full-image OCR of the document image can be performed to obtain a set of text boxes, which can then be matched with field values ​​or their anchor keywords, such as "invoice number," "tax number," or "amount," to obtain the ROI. Alternatively, layout analysis can be used to align the document image, such as using layout templates like page blocks and table cells to locate the target field's location, thus obtaining the ROI. Keyword anchor point positioning can also be used, such as expanding the ROI by using a fixed template position near the field name or by expanding the keyword bounding box. Based on this, the ROI can be cropped from the document image and then input into a visual language model to regenerate the target field. Before inputting the ROI image into the visual language model, enhancement processing can be performed, such as at least one of super-resolution, denoising, and tilt correction.

[0130] For example, Figure 8 A schematic diagram illustrating the process of locating and rereading the region of interest according to an embodiment of this disclosure is shown. Figure 8As shown, ROIs are located in document images through OCR recognition, layout analysis, and rule-based judgment. Based on the ROI location, corresponding ROI images can be cropped from the document image, and these images are then enhanced with denoising, super-resolution, and tilt correction. The ROI images are then input into a visual language model for ROI rereading, and a structured output sequence is obtained through restricted decoding. The structured output results are then parsed to obtain the final quality assessment and output decision control.

[0131] In this embodiment of the disclosure, the different execution methods of the two-stage re-decoding can be executed in one or more combinations, and the order of different execution methods can be preset in the combination. For example, when the two-stage decoding is triggered, dynamic adjustment constraint re-decoding is performed first to reduce decoding uncertainty; if the target quality score still does not meet the field recognition output conditions, local re-decoding can be performed, only the target field is regenerated, and other fields remain unchanged; if the target quality score still does not meet the field recognition output conditions, ROI rereading can be performed, the ROI region is determined by ROI positioning, the ROI image is obtained by cropping, and the ROI image is enhanced and then reread and generated; if the target quality score still does not meet the field recognition output conditions, other rejection processing is adopted, such as triggering fallback link verification, falling back to OCR and rule recognition, or KIE model, and outputting or setting it to empty after consistency verification.

[0132] In an optional embodiment of the method disclosed herein, step 105 may include step D.

[0133] Step D: Output structured output results and field identification information, which includes at least one of field quality information and evidence of reasons for rejection.

[0134] In this embodiment, when the target quality score meets the field recognition output conditions, a structured output result that meets business requirements can be output. Furthermore, field recognition information can also be output to reflect the performance of the visual language model in the recognition and reasoning process, such as analytical stability, accuracy, and reliability. This field recognition information may include field-level field quality information, and may also output evidence of the reason for rejection if rejection has occurred. This evidence of the reason for rejection can be referred to the foregoing description and includes at least the following: , or , Reasons for triggered format consistency check failure, key audit exception types, This disclosure does not impose specific restrictions on the values ​​and their basis.

[0135] In an optional embodiment of the method disclosed herein, step 101 may include step E.

[0136] Step E: In structured output mode, prohibit duplicate field keys and / or a whitelist of qualified key sets, and obtain the structured output sequence corresponding to the document image through a visual language model under the condition of prohibiting duplicate field keys.

[0137] In this embodiment, the visual language model recognition process is performed under the constraints of the structured output model. Further recognition constraints can be set to standardize the model output and improve field quality. Specifically, duplicate field keys can be prohibited from being output, or a whitelist of key sets can be defined to limit the range of field output and avoid the problem of model illusion divergence causing the output results to deviate from the document image content. Other details can be found in the description of step 101 above; to avoid repetition, they will not be repeated here.

[0138] This disclosure provides a field recognition method based on a visual language model. The method generates a structured output sequence corresponding to a document image using a visual language model in a structured output mode, then parses it to obtain a structured output result. Field-level quality assessment is performed on the structured output result to obtain a target field quality score. This score includes a structure gating quantity representing parsing and field consistency anomalies in the structured output result, and a format consistency score representing field values ​​in the structured output result for legality anomalies such as field type and field constraints. Based on this, a structured output result is output or a field rejection handling is triggered, depending on the field recognition output conditions and the target field quality score. This method performs field-level quality assessment on the visual language model's prediction of document images. Depending on actual needs, it can select parsing and consistency anomalies, and field value anomalies such as field type and field constraint legality anomalies, thereby avoiding low recognition accuracy and insufficient reliability caused by document image degradation, model illusions, etc. It eliminates the need for model training or fine-tuning, and the specific quality assessment makes the structured output or rejection handling somewhat interpretable, facilitating a controllable trade-off between business risks and field recognition accuracy.

[0139] The following are system embodiments of this disclosure, which can be used to execute the method embodiments of this disclosure. For details not disclosed in the device embodiments of this disclosure, please refer to the method embodiments of this disclosure.

[0140] This disclosure also provides one example structural diagram of a field recognition system 900 based on a visual language model. Specifically, refer to... Figure 9As shown, the system 900 may include: a model prediction module 901, used to generate a structured output sequence corresponding to a document image through a visual language model in a structured output mode; a result parsing module 902, used to parse the structured output sequence to obtain a structured output result; a quality assessment module 903, used to perform field-level quality assessment on the structured output result to obtain a target field quality score; the target field quality score includes at least one of a structure gating quantity and a format consistency score; the structure gating quantity is used to indicate parsing and field consistency anomalies in the structured output result; the format consistency score is used to indicate the legality anomalies of field values ​​in the structured output result for at least one of field types and field constraints; a result processing module 904, used to output a structured output result when the target field quality score meets the field recognition output conditions; the result processing module is also used to trigger field rejection processing when the target field quality score does not meet the field recognition conditions.

[0141] This disclosure provides an optional system embodiment where the target field quality score includes a structure gating quantity. The structure gating quantity is determined according to the following rules: when parsing the structured output sequence fails, the structure gating quantity takes a first value; when an unknown field key exists in the structured output result, the structure gating quantity takes a second value; when a duplicate field key exists in the structured output result, the structure gating quantity takes a third value; when a required field key is missing in the structured output result, the structure gating quantity takes a fourth value; when only a field key spelling error exists in the structured output result, the structure gating quantity takes a fifth value; and when there are no parsing and consistency errors in the structured output result, the structure gating quantity takes a sixth value. Among these, the first value is the smallest, and the sixth value is the largest.

[0142] In an optional system embodiment of this disclosure, the structural gating quantity includes at least one of the following: a global structural gating quantity uniformly applied to fields; and a field-level gating quantity applied to fields with parsing and consistency anomalies.

[0143] In an optional system embodiment of this disclosure, the target field quality score includes a format consistency score. The quality assessment module 903 is specifically used to perform legality checks on field types and / or field constraints in the structured output results to obtain a format consistency score. The legality checks include at least one of the following: hard checks based on all rules; soft checks based on different subsets of rules; and checks based on consistency constraints between fields.

[0144] In an optional system embodiment of this disclosure, the target field quality score further includes at least an initial field quality score. The quality assessment module 903 is specifically used to: for each field, determine the field value character range corresponding to the field value in the structured output sequence, and determine the corresponding word character range in the structured output sequence; calculate the coverage weight of the field value character range relative to the word character range; and use the coverage weight and decoding uncertainty to statistically calculate the initial field quality score of the predicted field; the decoding uncertainty is used to represent the word-level prediction uncertainty of the visual language model in the process of generating the structured output sequence.

[0145] In an optional system embodiment of this disclosure, the decoding uncertainty includes at least one of lexical probability, lexical log probability, lexical entropy, and lexical approximate entropy; the quality assessment module 903 is specifically used to calculate field-level statistical indicators based on coverage weight and decoding uncertainty; the field-level statistical indicators include at least one of field weighted average log probability, field weighted average entropy, and field minimum log probability; and calculate the initial field quality score of the predicted field based on the field-level statistical indicators.

[0146] In an optional system embodiment of this disclosure, the initial field quality score is calculated in at least one of the following ways: calculating a mapping based on the field weighted average log probability and the field weighted average entropy to obtain the initial field quality score; for short fields whose field length is less than or equal to the short field length threshold, calculating a mapping based on the field weighted average log probability, the field weighted average entropy, and the field minimum log probability to obtain the initial field quality score; and calculating the segmented linear score based on the field weighted average log probability and the field weighted average entropy to obtain the initial field quality score.

[0147] In an optional system embodiment of this disclosure, the quality assessment module 903 is further configured to perform weight reduction processing on special characters in the metacharacter range.

[0148] In an optional system embodiment of this disclosure, the target field quality score not meeting the field identification conditions includes any of the following: the target field quality score is less than the field threshold; the format consistency score indicates that there is an anomaly in the legality verification.

[0149] In an optional system embodiment of this disclosure, the triggering condition for field rejection processing includes the structure gating value being less than the sixth value.

[0150] In an optional system embodiment of this disclosure, the triggering condition for field rejection processing further includes the presence of at least one risk field in the structured output result.

[0151] In an optional system embodiment of this disclosure, the field rejection processing includes any one of setting the output to null, outputting a candidate, triggering two-stage re-decoding, and triggering a fallback link verification.

[0152] In an optional system embodiment of this disclosure, the field rejection processing includes triggering a two-stage re-decoding. The termination condition for the two-stage re-decoding includes any one of the following: the number of retries for the two-stage re-decoding is greater than or equal to the maximum number of retries; the computation time for the two-stage re-decoding is greater than or equal to the latency budget; the computation cost for the two-stage re-decoding is greater than or equal to the computation budget; the improvement in the quality score of the target field is less than the quality improvement threshold; or the quality score of the target field meets the field recognition output condition.

[0153] In an optional system embodiment of this disclosure, the field rejection processing includes triggering two-stage re-decoding, which includes at least one of dynamic adjustment constraint re-decoding, local re-decoding, and region of interest re-decoding.

[0154] In an optional system embodiment of this disclosure, the result processing module 904 is specifically used to output structured output results and field identification information, wherein the field identification information includes at least one of field quality information and evidence of rejection reasons.

[0155] In an optional system embodiment of this disclosure, the structured output mode includes at least one of structure syntax constraints, field set constraints, field type constraints, and field value format constraints.

[0156] In an optional system embodiment of this disclosure, the model prediction module 901 is specifically used to, in structured output mode, prohibit duplicate field keys and / or limit key set whitelist, and obtain the structured output sequence corresponding to the document image through a visual language model under the condition of prohibiting duplicate field keys.

[0157] This disclosure also provides a second example structural diagram of a field recognition system 1000 based on a visual language model. For example... Figure 10 As shown, the system 1000 may include an input module 1001, a model prediction module 1002, a result parsing module 1003, a quality assessment module 1004, and a result processing module 1005. The model prediction module 1002 may include a restricted decoding unit 10021 and a statistical recording unit 10022; the quality assessment module 1004 includes a key consistency audit unit 10041, a field alignment unit 10042, a lexical processing unit 10043, and a quality assessment subunit 10044; the result processing module 1005 may include a decision control unit 10051, a rejection handling unit 10052, and a result output unit 10053.

[0158] The input module 1001 can provide document images, structured output modes, and field sets; it can also include configurations such as field key whitelists and required field key sets.

[0159] The constrained decoding unit 10021 of the model prediction module 1002 retrieves the visual language model and generates a structured output sequence under the constraint of the structured output mode. The statistical recording unit 10022 can perform word-level decoding uncertainty statistics for the constrained decoding process of the visual language model, including word probability, word log probability, word entropy, and word approximate entropy. It can also characterize the reliability of the constrained decoding process of the visual language model by using consistency rate, self-consistency voting, etc.

[0160] The result parsing module 1003 can parse the structured output sequence provided by the restricted decoding unit 10021 to obtain the corresponding structured output result.

[0161] In the quality assessment module 1004, the consistency audit unit 10041 can determine the anomaly type corresponding to the field key and the value of the structure gating quantity based on the value rules of the structure gating quantity. On the basis of key consistency audit, the field alignment unit 10042 can locate the field value character range and lexical character range from the structured output sequence and structured output results, and map the token to obtain the field lexical set. The lexical processing unit 10043 can perform weighted calculations based on coverage weight and additional weight to reduce weight or block structure characters, special characters or other characters. The quality assessment subunit 10044 can perform field-level statistical aggregation based on decoding uncertainty, structure gating quantity, etc., and can analyze the format consistency score of field type and field constraint legality anomalies, and then integrate different assessment results to obtain the target field quality score.

[0162] In the result processing module 1005, the decision control unit 10051 can determine whether the structured output result meets the field recognition output condition by comparing the target field quality score with the field threshold, and thus decide to output the structured output result or trigger field rejection processing.

[0163] When the decision control unit 10051 triggers the second-stage re-decoding of a field in the result processing module 1005, the rejection processing unit 10052 can call other modules and units to perform the second-stage re-decoding. Depending on the execution method of the second-stage re-decoding, different modules and units can be called. For example, dynamic constraint adjustment re-decoding can call the constrained decoding unit 10021 to regenerate the structured sequence for the document image; local re-decoding can call the result parsing module 1002 to perform local re-decoding of the target field. The structured sequence and structured results obtained from the second-stage re-decoding can be re-evaluated by the quality assessment module 1004, and then the decision control unit 10051 of the result processing module 1005 makes subsequent decisions.

[0164] In the result processing module 1005, the result output unit 10053 can output the structured output result when the decision control unit 10051 makes a decision and outputs the structured output result. It can also provide field-level quality information, evidence of rejection reasons, etc.

[0165] The specific details of each module in the field recognition system based on the visual language model have been described in detail in the corresponding field recognition method based on the visual language model, so they will not be repeated here.

[0166] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0167] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0168] In exemplary embodiments of this disclosure, an electronic device capable of implementing the above-described methods is also provided. Those skilled in the art will understand that various aspects of this disclosure can be implemented as systems, methods, or program products. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0169] The following reference Figure 11 To describe an electronic device 1100 according to such an embodiment of the present disclosure. Figure 11 The electronic device 1100 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0170] like Figure 11 As shown, the electronic device 1100 is manifested in the form of a general-purpose computing device. The components of the electronic device 1100 may include, but are not limited to: at least one processing unit 1110, at least one storage unit 1120, a bus 1130 connecting different system components (including storage unit 1120 and processing unit 1110), and a display unit 1140.

[0171] The storage unit stores program code, which can be executed by the processing unit 1110, causing the processing unit 1110 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 1110 can perform actions such as... Figure 1 Or the field recognition method based on the visual language model shown in Figure 3.

[0172] Storage unit 1120 may include readable media in the form of volatile storage units, such as random access memory (RAM) 11201 and / or cache memory 11202, and may further include read-only memory (ROM) 11203.

[0173] Storage unit 1120 may also include a program / utility 11204 having a set (at least one) program module 11205, such program module 11205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0174] Bus 1130 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0175] Electronic device 1100 can also communicate with one or more external devices 1200 (e.g., field disks, pointing devices, Bluetooth devices, etc.), and with one or more devices that enable users to interact with electronic device 1100, and / or with any device that enables electronic device 1100 to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be performed via input / output (I / O) interface 1150. Furthermore, electronic device 1100 can also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 1160. As shown, network adapter 1160 communicates with other modules of electronic device 1100 via bus 1130. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1100, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0176] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0177] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible implementations, various aspects of this disclosure may also be implemented as a program product including program code, which, when the program product is run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0178] The program product for implementing the above-described method according to embodiments of the present disclosure may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0179] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0180] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0181] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0182] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0183] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0184] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention described herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not invented by this disclosure. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

Claims

1. A field recognition method based on a visual language model, characterized in that, The method includes: In structured output mode, a structured output sequence corresponding to the document image is generated through a visual language model; The structured output sequence is parsed to obtain the structured output result; Perform field-level quality assessment on the structured output to obtain a target field quality score; the target field quality score includes at least one of structure gating quantity and format consistency score; the structure gating quantity is used to indicate parsing and field consistency anomalies in the structured output; the format consistency score is used to indicate the legality anomalies of field values ​​in the structured output for at least one of field type and field constraint; If the quality score of the target field meets the field identification output conditions, the structured output result is output. If the quality score of the target field does not meet the field recognition conditions, the field rejection process is triggered.

2. The method according to claim 1, characterized in that, The target field quality score includes a structural gating quantity, and the rules for determining the value of the structural gating quantity are as follows: When parsing the structured output sequence fails, the structure gate variable is set to the first value. When there is an unknown field key in the structured output, the structure gating quantity is the second value; When there are duplicate field keys in the structured output, the structure gating quantity is the third value; When the required field key is missing in the structured output, the structure gating value is the fourth value. When only field key spelling errors exist in the structured output, the structure gating value is the fifth value. When there are no anomalies in the parsing and consistency of the structured output results, the structure gating quantity is the sixth value; Among them, the first value is the smallest, and the sixth value is the largest.

3. The method according to claim 1, characterized in that, The structural gating quantity includes at least one of the following: A global structural gating variable uniformly applied to the aforementioned fields; Field-level gating applied to fields that exhibit parsing and consistency anomalies.

4. The method according to claim 1, characterized in that, The target field quality score includes a format consistency score. The process of performing field-level quality assessment on the structured output to obtain the target field quality score includes at least the following: In the structured output, the legality of field types and / or field constraints is checked to obtain a format consistency score. The legality verification includes at least one of the following: Hard validation based on all rules; Soft validation based on different rule subsets; Validation based on consistency constraints between fields.

5. The method according to claim 1, characterized in that, The target field quality score further includes at least the initial field quality score, and the process of performing field-level quality assessment on the structured output to obtain the target field quality score includes at least: For each field, determine the character range of the field value corresponding to the field value in the structured output sequence, and determine the character range of the corresponding word in the structured output sequence; Calculate the coverage weight of the field value character range relative to the word character range; The initial field quality score of the field is statistically predicted using the coverage weight and decoding uncertainty; the decoding uncertainty is used to represent the word-level prediction uncertainty of the visual language model in the process of generating the structured output sequence.

6. The method according to claim 5, characterized in that, The decoding uncertainty includes at least one of the following: lexical probability, lexical log probability, lexical entropy, and lexical approximate entropy. The step of statistically predicting the initial field quality score of the field using the coverage weight and decoding uncertainty includes: Field-level statistical indicators are calculated based on the coverage weights and the decoding uncertainty; the field-level statistical indicators include at least one of the following: field weighted average log probability, field weighted average entropy, and field minimum log probability. The initial field quality score for predicting the field is calculated based on the field-level statistical indicators.

7. The method according to claim 6, characterized in that, The initial field quality score is calculated using at least one of the following methods: The initial field quality score is obtained by calculating the mapping based on the weighted average log probability of the field and the weighted average entropy of the field. For short fields whose length is less than or equal to the short field length threshold, a mapping is calculated based on the field weighted average log probability, the field weighted average entropy, and the field minimum log probability to obtain the initial field quality score; Based on the weighted average log probability and the weighted average entropy of the field, the linear score of the segment is calculated to obtain the initial field quality score.

8. The method according to claim 5, characterized in that, Before calculating the coverage weight of the field value character range relative to the lexical character range, the method further includes: Within the defined metacharacter range, special characters are subject to reduced weighting.

9. The method according to claim 1, characterized in that, The target field quality score does not meet any of the following field identification conditions: The target field's quality score is less than the field threshold; A format consistency score indicates an anomaly in the validity check.

10. The method according to claim 2, characterized in that, The triggering conditions for the field rejection processing include when the structure gating value is less than the sixth value.

11. The method according to claim 1, characterized in that, The triggering conditions for the field rejection processing also include the presence of at least one risk field in the structured output result.

12. The method according to claim 1, characterized in that, The field rejection handling includes any one of the following: setting the output to null, outputting a candidate, triggering two-stage re-decoding, or triggering a fallback link verification.

13. The method according to claim 12, characterized in that, The field rejection handling includes triggering a two-stage re-decoding, and the termination condition of the two-stage re-decoding includes any one of the following: The number of retries for the two-stage re-decoding is greater than or equal to the maximum number of retries; The computation time for the two-stage re-decoding is greater than or equal to the delay budget; The computational cost of the two-stage re-decoding is greater than or equal to the computational budget; The improvement in the quality score of the target field is less than the quality improvement threshold; The target field quality score satisfies the field identification output condition.

14. The method according to claim 12, characterized in that, The field rejection processing includes triggering two-stage re-decoding, which includes at least one of dynamic adjustment constraint re-decoding, local re-decoding, and region of interest re-decoding.

15. The method according to claim 1, characterized in that, The output of the structured output result includes: The structured output result is output, along with field identification information, which includes at least one of field quality information and evidence of reasons for rejection.

16. The method according to claim 1, characterized in that, The structured output mode includes at least one of the following: structure syntax constraints, field set constraints, field type constraints, and field value format constraints.

17. The method according to claim 1, characterized in that, In the structured output mode, generating the structured output sequence corresponding to the document image through a visual language model includes: In the structured output mode, a whitelist of prohibited duplicate field keys and / or limited key sets is used to obtain the structured output sequence corresponding to the document image through the visual language model under the condition of prohibiting duplicate field keys.

18. A field recognition system based on a visual language model, characterized in that, The system includes: The model prediction module is used to generate a structured output sequence corresponding to a document image using a visual language model in structured output mode. The result parsing module is used to parse the structured output sequence to obtain the structured output result; A quality assessment module is used to perform field-level quality assessment on the structured output results to obtain a target field quality score. The target field quality score includes at least one of a structure gating quantity and a format consistency score. The structure gating quantity is used to indicate parsing and field consistency anomalies in the structured output results. The format consistency score is used to indicate the legality anomalies of field values ​​in the structured output results for at least one of field types and field constraints. The result processing module is used to output the structured output result when the quality score of the target field meets the field identification output condition. The result processing module is also used to trigger field rejection processing when the quality score of the target field does not meet the field recognition conditions.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the field recognition method based on a visual language model as described in any one of claims 1-17.

20. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the field recognition method based on a visual language model according to any one of claims 1-17 by executing the executable instructions.

Images