Adaptive form layout analysis method and system based on YOLO and OCR

Abstract

This invention discloses an adaptive form layout parsing method and system based on YOLO and OCR. The method includes: inputting a form image into a layout detection model to detect layout regions in the form image, outputting multiple layout units and region type information corresponding to each layout unit; for each detected layout unit, calling a text recognition module to perform text recognition to obtain the text content and character-level coordinate information of the corresponding layout unit; performing fusion modeling to generate a region adjacency graph of the form layout; based on the region adjacency graph, dynamically generating field parsing paths by combining preset rule constraints, performing key-value pair matching, parsing cross-region fields or multi-line fields, and generating a structured form result. This method can solve the problem that traditional rule-driven or template-based form parsing methods are highly sensitive to layout changes and struggle to adapt to complex form scenarios with different sources, shooting conditions, and field arrangement methods.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to an adaptive form layout parsing method and system based on YOLO and OCR. Background Technology

[0002] In the processing of structured or semi-structured documents such as invoices, contracts, application forms, and registration forms, traditional rule-driven or template-based form parsing methods are highly sensitive to layout changes and are difficult to adapt to complex form scenarios with different sources, shooting conditions, and field arrangement methods.

[0003] While existing OCR technology can recognize text content, it lacks the ability to understand the overall structure of a form, and cannot accurately determine field boundaries, field semantics, and key-value relationships. This often leads to problems such as field mismatch, row and column confusion, and cross-regional misrecognition.

[0004] Meanwhile, existing deep learning methods often focus only on a single aspect of text detection or text recognition, lacking the ability to collaboratively model the target region localization results and text semantic results. This results in low parsing accuracy and weak generalization ability in scenarios with complex forms, non-standard layouts, and variable number of fields, making it difficult to scale up applications in real-world business scenarios. Summary of the Invention

[0005] The summary section introduces a series of simplified concepts, which will be further explained in detail in the detailed description section. The summary section of this invention is not intended to limit the key features and essential technical features of the claimed technical solution, nor is it intended to determine the scope of protection of the claimed technical solution.

[0006] To address the problem that traditional rule-driven or template-based form parsing methods are highly sensitive to layout changes and struggle to adapt to complex form scenarios with different sources, shooting conditions, and field arrangements, this invention proposes, in a first aspect, an adaptive form layout parsing method based on YOLO and OCR, comprising: The form image is input into the layout detection model to detect the layout regions in the form image and output multiple layout units and the region type information corresponding to each layout unit. The region type information includes at least one of the following: title region, field name region, field value region, table region, and explanatory text region. For each of the detected layout units, the text recognition module is invoked to perform text recognition in order to obtain the text content and character-level coordinate information of the corresponding layout unit; Based on the region type information, spatial location information, text content and character-level coordinate information of each layout unit, a fusion model is performed to generate a region adjacency relationship graph of the form layout; Based on the region adjacency graph, and combined with preset rule constraints, field parsing paths are dynamically generated to perform key-value pair matching between field name regions and field value regions without relying on fixed templates, and to parse cross-region fields or multi-row fields to generate structured results for forms.

[0007] Secondly, this invention also proposes an adaptive form layout parsing system based on YOLO and OCR, comprising: The detection unit is used to input the form image into the layout detection model to detect the layout area in the form image and output multiple layout units and the area type information corresponding to each layout unit. The area type information includes at least one of the following: title area, field name area, field value area, table area and explanatory text area. The recognition unit is used to call the text recognition module to perform text recognition for each of the detected layout units in order to obtain the text content and character-level coordinate information of the corresponding layout unit. The fusion unit is used to perform fusion modeling based on the region type information, spatial location information, text content and character-level coordinate information of each layout unit to generate a region adjacency relationship graph of the form layout; The generation unit is used to dynamically generate field parsing paths based on the region adjacency graph and in combination with preset rule constraints, so as to perform key-value pair matching on field name regions and field value regions without relying on fixed templates, parse cross-region fields or multi-line fields, and generate structured results of the form.

[0008] Thirdly, an electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program stored in the memory to implement the steps of the adaptive form layout parsing method based on YOLO and OCR as described in any of the first aspects above.

[0009] Fourthly, the present invention also proposes a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the adaptive form layout parsing method based on YOLO and OCR according to any one of the above claims in the first aspect.

[0010] In summary, the adaptive form layout parsing method based on YOLO and OCR proposed in this application detects layout regions in a form image by inputting the form image into a layout detection model. It outputs multiple layout units and their corresponding region type information, where the region type information includes at least one of a title region, field name region, field value region, table region, and explanatory text region. For each detected layout unit, a text recognition module is invoked to perform text recognition, obtaining the text content and character-level coordinate information of the corresponding layout unit. Based on the region type information, spatial location information, text content, and character-level coordinate information of each layout unit, a fusion model is performed to generate a region adjacency graph of the form layout. Based on the region adjacency graph, and combined with preset rule constraints, a field parsing path is dynamically generated to perform key-value pair matching between field name regions and field value regions without relying on a fixed template. This allows for the parsing of cross-region fields or multi-line fields, generating a structured form result. This improves the parsing accuracy in complex form scenarios. Because layout understanding is performed first, followed by text recognition, relationship modeling, and dynamic parsing, it effectively reduces field mismatches, row / column confusion, and cross-region misidentification. It also enhances adaptability to diverse layouts. This solution doesn't base its parsing on fixed templates, but rather on real-time detected layout units and adjacency relationships, thus adapting to documents from different sources and with different layouts, such as invoices, contracts, application forms, and registration forms. It also improves robustness to non-standard shooting conditions. In actual business, forms often come from scanned documents, mobile phone photos, compressed images, and even printed copies. Through preprocessing, region detection, and local recognition, the effects of lighting, tilt, and local blurring can be reduced. Furthermore, it enhances the ability to extract complex fields completely. Traditional methods often only extract simple single-line fields, while this solution can further process multi-line fields, cross-region fields, and nested table fields. For example, addresses, medical descriptions, remarks, additional instructions, and approval opinions can all be preserved relatively completely. At the same time, it reduces manual configuration costs. Since the system doesn't need to write separate templates and coordinate rules for each form, when faced with new form styles, as long as the overall layout type is still within the recognizable range, the system can be quickly deployed, which is particularly important for actual engineering deployments. Finally, it improves the usability of structured output. The output is not unordered text, but structured data with field semantics, hierarchical relationships and table substructures, which facilitates subsequent database entry, field validation, automatic review, process triggering, information comparison and knowledge extraction. Attached Figure Description

[0011] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit this specification. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating an adaptive form layout parsing method based on YOLO and OCR, provided for embodiments of this application; Figure 2 A schematic diagram of an adaptive form layout parsing system based on YOLO and OCR provided in this application embodiment; Figure 3 This is a schematic diagram of an electronic device structure that provides an adaptive form layout parsing method based on YOLO and OCR, as an embodiment of this application. Detailed Implementation

[0012] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus. The technical solutions of the embodiments of this application will now be clearly and completely described in conjunction with the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0013] To address the issue that traditional rule-driven or template-based form parsing methods are highly sensitive to layout changes and struggle to adapt to complex form scenarios with varying sources, shooting conditions, and field arrangements, please refer to [link to relevant documentation]. Figure 1 This is a flowchart illustrating an adaptive form layout parsing method based on YOLO and OCR provided in an embodiment of this application. Specifically, it may include steps S110 to S140.

[0014] S110, Input the form image into the layout detection model to detect the layout area in the form image, and output multiple layout units and the area type information corresponding to each layout unit, wherein the area type information includes at least one of title area, field name area, field value area, table area and explanatory text area; S120, for each of the detected layout units, call the text recognition module to perform text recognition in order to obtain the text content and character-level coordinate information of the corresponding layout unit; S130, Based on the region type information, spatial location information, text content and character-level coordinate information of each layout unit, a fusion model is performed to generate a region adjacency relationship diagram of the form layout; S140, based on the region adjacency relationship graph, and combined with preset rule constraints, dynamically generate field parsing paths to perform key-value pair matching between field name regions and field value regions without relying on fixed templates, parse cross-region fields or multi-line fields, and generate structured results for the form.

[0015] Understandably, traditional form processing often falls into two problems. The first is that it only performs text recognition without understanding the layout structure. While this yields several text fragments, it cannot identify which fragment represents a field name, which represents a field value, or whether a particular field value belongs to a left-hand label, a top-hand label, or an adjacent cell. The second problem is its reliance on templates or fixed rules. Once the form style changes—for example, fields change from horizontal to vertical, or key-value pairs that were originally on the same line are split into two lines—the parsing logic is prone to failure. This solution addresses these problems through four layers. The first layer is the layout detection layer. A layout detection model is used to identify layout units in the form image, such as title areas, field name areas, field value areas, table areas, and explanatory text areas. This way, the system no longer deals with a large number of disorganized characters but first segments the form into functionally meaningful area objects. The second layer is the text recognition layer. Text recognition is performed individually on each layout unit, yielding not only the text content but also character-level coordinate information. The role of character-level coordinate information is not merely to record text position, but more importantly, to provide fine-grained positional basis for subsequent judgments of multi-line fields, broken fields, and cross-region fields. The third layer is the fusion modeling layer. It unifies region type information, spatial location information, text content, and character-level coordinate information to construct a region adjacency graph. This graph is not an image in the ordinary sense, but a structural representation. Each node in the graph represents a layout unit, and the edges between nodes represent relationships such as positional adjacency, alignment, vertical correspondence, horizontal association, and semantic association. In this way, the system no longer views each text box in isolation, but understands the hierarchical relationships between text within the overall layout structure. The fourth layer is the dynamic parsing layer. Based on the region adjacency graph and combined with preset rule constraints, it dynamically generates field parsing paths. Dynamic generation means that it does not pre-assuming that the form must follow a fixed template, but automatically determines whether the field name should prioritize searching for the field value to the right, downwards, or further concatenating in adjacent description areas based on the actual structure of the current form, thereby completing key-value pair matching and complex field parsing.

[0016] For example, in S110, the different functional areas of the form are first distinguished at the image level. This is because different text areas serve different purposes in a form context. Title areas typically identify the document's topic; field name areas are usually labels such as "name," "ID number," and "address"; field value areas contain the specific content entered by the user or generated by the system; table areas usually contain row and column organization; and explanatory text areas often contain notes, filling instructions, disclaimers, etc. If these areas are not distinguished first, relying solely on text recognition results can easily lead to the following misjudgments: mistaking titles for field names; mistaking explanatory text for field values; mistaking table headers for ordinary field names; and incorrectly concatenating multiple adjacent field values ​​into a single long field. Therefore, it is necessary to first functionally partition the layout at the image level. YOLO-based detection models are suitable for this task because they can complete multi-target localization and classification in a single forward inference process, directly outputting the bounding boxes and categories of each layout area. In form scenarios, although different layout areas all belong to text-related regions, they often exhibit learnable differences in aspects such as size ratio, text density, position distribution, surrounding white space, and alignment. The layout detection model learns these differences to distinguish region types. This is also a crucial foundation for adaptive processing. Subsequent parsing does not directly rely on the form template but rather on the layout units actually detected in the current image. If the input form style changes, as long as the model can still recognize these layout units and their types, the subsequent process can continue. Therefore, the first step is to obtain the form image to be parsed. This form image can originate from scanned documents, mobile phone photos, tablet images, webpage screenshots, image files exported from office systems, or bitmaps generated from document format conversions. The form image can be a color image or a grayscale image. Before the image enters the layout detection model, image preprocessing can be performed. Preprocessing can include image size normalization, brightness correction, contrast enhancement, noise reduction, perspective distortion correction, rotation correction, and edge cropping. The purpose is to reduce the impact of factors such as shooting angle, uneven lighting, shadow occlusion, and paper skew on subsequent layout detection. After preprocessing, the form image is input into the layout detection model. The layout detection model is preferably an improved YOLO model. This improvement can be reflected in the following aspects: First, a multi-scale feature fusion mechanism can be introduced. Forms contain large title areas, small field name areas, and possibly short but long explanatory text areas. By fusing features at different resolution levels, the model's ability to detect targets of different sizes can be enhanced. Second, an adaptive anchor box adjustment mechanism can be introduced. Because the shapes of areas in a form differ significantly from those in natural scenes—for example, field name areas are usually short in width and moderate in height, explanatory text areas may be wide and flat, and table areas may be large—these improvements can be implemented.By obtaining prior boxes that better match the form layout distribution based on the training set statistics, the accuracy of region regression can be improved. Thirdly, higher attention can be given to small target text regions and densely arranged regions during training. For example, enhanced sampling can be performed on small region samples, and training samples can be added for overlapping scenarios of adjacent regions to improve the model's ability to distinguish adjacent field blocks. After inference, the layout detection model outputs multiple layout units. Each layout unit includes at least bounding box coordinates and a region type label. Bounding box coordinates can be represented by the coordinates of the top left and bottom right corners, or by the coordinates of the center point, width, and height. The region type label includes at least one of the following: title region, field name region, field value region, table region, and explanatory text region. Post-processing can also be performed on the detection results. For example, when two similar regions highly overlap and have similar confidence levels, a more reasonable candidate box can be retained based on a non-maximum suppression strategy; when the boundaries between the field name region and the field value region are close, the overlapping boxes can be fine-tuned through positional constraints to reduce region crosstalk during subsequent text recognition.

[0017] For example, in S120, the detected layout regions can be transformed from image regions into text regions with location semantics. Layout detection can only tell the system what type a region is, but not its specific content. For example, knowing that a region is a field name region is not enough; it is also necessary to know whether it contains "date of birth" or "ID number"; knowing that a region is a field value region is not enough; it is also necessary to know whether its content is "March 6, 1992" or "4401...". Text content alone is still insufficient, because the complex relationships in forms often depend on the fine-grained arrangement of text within regions. For example, a field value region may contain two lines of content, the first being the main value and the second being supplementary information; or a region may contain two candidate options, "male" and "female," one of which is marked with a checkmark; or the amount and currency symbol may be located in different positions on a ticket. To handle these situations, character-level coordinate information needs to be preserved, enabling the system to analyze the text composition and spatial distribution at a finer granularity. Therefore, the key to this step is not just OCR recognition, but simultaneously preserving the text content and its spatial location. In this way, subsequent semantic judgments can be made not only using text content, but also using character positions for row attribution, cross-region splicing, and table cell attribution. For each layout unit output by S110, the system crops the corresponding region image from the original or preprocessed form image based on its bounding box. During cropping, the boundary can be appropriately extended outwards to avoid truncating edge characters due to overly tight detection boxes, but the extension range should not be too large to avoid introducing content from adjacent regions. For the cropped region image, the text recognition module is invoked for recognition. The text recognition module can employ an OCR model based on convolutional neural networks and sequence modeling networks, or a sequence recognition model based on visual encoders and decoders. For table regions, a recognition strategy suitable for densely distributed table text can be adopted as needed. For explanatory text regions, a recognition path more suitable for long text recognition can be used. The recognition result includes at least the text content and character-level coordinate information. Character-level coordinate information can be represented as the bounding box coordinates of each character in the region image, or as the character center point coordinates, character outline coordinates, or character baseline information. Then, by combining the position of the layout unit within the entire form, the character-level coordinates are mapped back to the original image coordinate system, resulting in a positional representation under a unified reference. The text recognition module can also output character-level or word-level confidence scores. When performing key-value matching subsequently, if the text confidence score of a certain region is low, its priority in the candidate path can be reduced, or a secondary recognition strategy can be triggered. In some implementations, differentiated recognition strategies can be adopted for different region types.For example: for the title area, larger font spacing and centering are permissible; for the field name area, short text labels can be prioritized; for the field value area, the ability to recognize mixed text such as numbers, dates, numbers, and names can be enhanced; for the explanatory text area, the recognition of long sentences with multiple lines can be improved; for the table area, local grid segmentation can be performed before cell text recognition. For example, in an insurance application, "Insured's Name" is a field name area, and the corresponding field value area might be "Zhang San"; the "Contact Address" field value area might contain an entire line or even two lines of text; and the "Additional Notes" area might be a long, continuous explanatory text. Targeted OCR recognition for these different areas can improve the overall recognition quality.

[0018] For example, in S130, since the information from a single region is insufficient, the relationships between regions must be further represented to truly understand the form structure. Fields in a form are typically not isolated. A field name corresponds to a field value because there is a certain regularity between them on the page. For example, a field name might be to the left of a field value, above it, or in the same list as the field value. Text in a table is determined by row and column relationships. Explanatory text is often located below a field region to supplement constraints or provide instructions. In other words, a form is not simply a collection of text, but a structure with spatial and semantic organizational relationships. Relying solely on nearest distance cannot correctly establish this structure, as the nearest text block may not correspond to the value. For example, in a medical examination form, "blood pressure" might be "120 / 80" to the right, but the adjacent pulse text box below it might be even closer. Using only Euclidean distance can easily lead to mismatches. Therefore, a comprehensive relationship representation must be introduced. The region adjacency diagram is designed for this purpose. It represents each layout unit as a node, and the vertical adjacency, horizontal adjacency, row alignment, column alignment, boundary overlap, containment relationship, and text semantic relevance between nodes as edges, thus forming a graph model that reflects the overall layout structure. Based on this graph, subsequent inferences about field affiliation can be made more reasonable, rather than relying solely on local distances. Each layout unit can be treated as a graph node. Each node contains at least the following attributes: region type, bounding box coordinates, region center point coordinates, width, height, OCR text content, character-level coordinate set, and recognition confidence information. Then, edge relationships are constructed between nodes. Edge relationships can include at least the following categories: The first category is positional adjacency. If node A is to the left of node B and their vertical projections overlap significantly, a "left adjacency" relationship can be established; if node A is above node B and their horizontal projections overlap significantly, a "top adjacency" relationship can be established. The second category is alignment. For example, if the left boundaries of two regions are close, it may indicate that they belong to the same column; if the upper boundaries of two regions are close, it may indicate that they belong to different units in the same row. The third category is spacing. There is usually a small and stable spacing between field names and field values, while the spacing between field groups is often larger. Spacing relationships can help determine pairing priority. The fourth type is containment or dependency relationships. For example, if character-level coordinates show that some text is located inside a table area, then these text nodes can be attached to nodes in that table area; some descriptive text may be located within the boundaries of a field group, and a weak dependency relationship can also be established. The fifth type is text semantic relationships. For example, if the OCR text in one area is "Name" and the OCR text in another area is "Zhang San," then the former is more likely a field name semantically, and the latter is more likely a field value; if the text in one area is "Remarks," then its adjacent lines of text may belong to descriptive content.Semantic relationships can be obtained through keyword dictionaries, text classifiers, word vector similarity, or rule matching. After graph construction, a region adjacency graph is formed. This graph can represent the organization of different regions within the entire form. For example, in a discharge record form, the system can see that the name nodes for the fields "Patient Name," "Gender," and "Age" are distributed in the same row, with the three field value nodes connected to their right sides; below, there is a large "Summary of Condition" description area node; and at the bottom, there is a table area node. This graph structure reflects the true semantics of the form better than a simple list of text strings. Weights can be assigned to the edges after graph construction. Weights can comprehensively consider adjacency direction, distance, alignment, text type compatibility, and semantic matching. This allows the system to prioritize searching for possible key-value relationships along high-weight paths during subsequent dynamic parsing.

[0019] For example, in S140, instead of pre-writing a hard-coded form template, the system dynamically determines how to find field values ​​and organize the output results based on the relationship graph already constructed from the current form. Using a traditional template approach, it's typically necessary to pre-define "the range of pixels to the right of field name A is the field value" and "the fixed position in the next row after field name B is the field value." This method is effective for fixed layouts, but it easily fails when the form source differs, the scan ratio differs, the field arrangement is adjusted, or the area expands or contracts. This solution dynamically generates field parsing paths. A field parsing path can be understood as the process by which the system starts from a field name node and searches for the most likely corresponding field value node in the area adjacency graph according to a series of constraints. This process is not fixed but changes with the layout relationships in the current graph. Some fields may prioritize searching to the right, some may prioritize searching downwards, and some fields need to be extended from a single field value node to the subordinate areas below to form a complete value. In this process, the role of rule constraints is not to replace structural understanding, but to provide boundary conditions and priority principles for dynamic searching. For example, field name areas are typically not paired with title areas, field values ​​generally do not prioritize long jumps, matching within table areas should be prioritized within the table, and multi-row field values ​​should prioritize merging adjacent and semantically continuous text areas. Through these constraints, the system can maintain parsing stability without relying on fixed templates. A set of rule constraints can be pre-established. This set of rule constraints can include the following: First, type constraints. Field name areas prioritize matching with field value areas, and do not prioritize establishing primary matching relationships with title areas or explanatory text areas. Explanatory text areas are generally used as supplementary information rather than direct field values, unless certain fields explicitly allow the explanatory area to be incorporated into the field value. Second, direction constraints. For common forms, the field value is more likely to be to the right or below the field name; therefore, horizontal right adjacency priority and vertical downward adjacency priority can be set. However, this priority is not absolutely fixed and can be adaptively adjusted according to the overall layout characteristics of the current form. For example, in a vertical field form, the downward priority can be increased. Third, distance constraints. The distance between adjacent field names and values ​​is generally within a reasonable range; if the candidate value area is too far from the field name area, its matching priority decreases. Fourth, alignment constraints. Areas horizontally aligned in the same row or vertically aligned in the same column are more likely to form a matching relationship. Fifth, semantic constraints. For example, if the field name text is "date of birth", then the candidate value text in date format has a higher matching confidence; if the field name is "amount", then numeric text is more likely to be the expected; if the field name is "address", then long text, multi-segment text, and text with province, city, and district structures are more likely to be the correct value. Sixth, continuity constraints. If a field value is distributed across multiple rows or regions, then these regions should have sequential continuity in character coordinates, should be spatially adjacent, and should be semantically able to be pieced together to form a complete expression.The region adjacency graph can be used as the search space. Starting from a field name node, the system scores adjacent nodes according to rule constraints and selects the candidate value node with the highest score. If this value node is determined to be the starting region of a multi-line value, the system continues to search for additional value regions that satisfy continuity constraints below or in adjacent directions until a complete field value is formed. An example of parsing cross-region fields is as follows: In an application form, the "Current Address" field is split into two regions. The first region is located to the right of the field name and contains "Shanghai Pudong New Area," while the second region is located at the next line indentation and contains "Room XX, XX, XX Road." If the system only matches the first region, the value is incomplete. Through dynamic graph search, the system can first match the first value region, and then, based on the indentation alignment, distance, and semantic continuity below, identify the second region as a continuation of the same field, thus merging and outputting the complete address. An example of parsing multi-line fields is as follows: The "Chief Complaint" field in a medical record form typically contains a multi-line text. The field name "Main Complaint" is located on the left, while the right side contains a large text area. The OCR result may be automatically segmented into several lines of text. The system can determine that these lines belong to the same field value based on character-level coordinates and line spacing, and then concatenate them in their original order. For table areas, the table area can be treated as a special parsing unit. The system can first organize the cell text within the table nodes according to row and column relationships, and then output the table content as a nested structure. In this way, the table will not be mistakenly parsed as several ordinary fields. Finally, the system generates a structured result for the form. This result can be represented using key-value pairs, hierarchical dictionaries, list structures, or tokenized data formats. For example, when outputting in JSON format, it can contain different levels of content such as titles, ordinary fields, explanatory information, and table substructures.

[0020] In some examples, inputting the form image into the layout detection model to detect layout regions in the form image includes: The form image is processed by multi-scale feature fusion based on the improved YOLO object detection model to enhance the detection capability of small font regions, multi-line text regions and irregularly arranged regions. The detection anchor box is adaptively adjusted based on the size distribution characteristics of the target area in the form image to improve the positioning accuracy of different field areas, table areas and explanatory text areas; Output the bounding box coordinates, confidence level, and region type information of each layout unit.

[0021] For example, the form image to be parsed is first acquired. This form image can come from a scanning device, a mobile terminal camera, an office system export module, or a paper document to image conversion module. The acquired image may have issues such as resolution differences, shooting angle shifts, paper curvature, local shadows, background interference, and compression distortion. Before inputting it into the layout detection model, image preprocessing can be performed. Preprocessing operations can include grayscale normalization, color space conversion, image scaling, edge cropping, rotation correction, perspective correction, contrast enhancement, and noise suppression. For example, for a contract homepage image taken with a mobile phone, the four corner positions can be estimated based on the page edges to correct perspective distortion, and then local enhancement can be performed on areas with uneven brightness to make the main form area clearer. The preprocessed form image is then input into an improved YOLO object detection model. In some implementations, this model can employ a structure including a backbone feature extraction network, a feature pyramid fusion network, and a detection head. The backbone network is used to extract image features at different levels, the feature fusion network is used to fuse shallow localization details with deep semantic features, and the detection head is used to output the region location and region type. To adapt to form scenarios, a form layout dataset can be pre-built during the training phase, and the title area, field name area, field value area, table area, and explanatory text area can be labeled. Multi-scale feature fusion processing can be achieved by simultaneously extracting high-resolution, medium-resolution, and low-resolution feature layers. For example, for an inpatient registration form, the header title "Basic Information Table for Admitted Patients" is a large target, requiring stable recognition based on higher-level semantic features; while labels such as "Name," "Age," and "Gender" are smaller and closer together, relying more on shallow or medium-level detailed features for accurate positioning; the medical history description area in the middle has a large range and includes multiple rows, requiring a comprehensive assessment of multiple scale features. Anchor box adaptive adjustment can be performed based on cluster analysis of the width and height distribution of all labeled layout areas in the training set to obtain several prior box shapes that better fit the form document scenario. For example, after statistical analysis of a large number of form samples, it was found that field name areas are more concentrated in narrow rectangles, explanatory text areas are more concentrated in wide rectangles, and table areas are often larger. In this case, a more realistic set of anchor boxes can be generated for these distributions, making it easier for the model to fit these layout areas. After model inference, multiple layout units are output. Each layout unit can include bounding box coordinates, region confidence score, and region type information. The bounding box coordinates identify the region's location range in the original form image, the confidence score reflects the model's trustworthiness of the detection result, and the region type information identifies whether the layout unit belongs to a title region, field name region, field value region, table region, or descriptive text region. Post-processing can also be performed on the detection results.For example, highly overlapping candidate boxes of the same type can be suppressed, excessively small noisy candidate boxes can be filtered, and fine-grained boundary correction can be performed on field name and field value regions with overly close boundaries. This can further reduce false positives and false negatives in layout detection results.

[0022] In some examples, the step of calling a text recognition module to perform text recognition for each detected layout unit to obtain the text content and character-level coordinate information of the corresponding layout unit includes: Perform OCR recognition on each of the layout units to obtain the text string within the layout unit; Obtain the position coordinates of each character or text fragment in the original form image; The text string is associated with and stored with the corresponding character-level coordinate information to serve as the basis for subsequent field attribution determination and key-value pair matching.

[0023] For example, for each layout unit output by the layout detection module, the corresponding region image is cropped from the original form image based on its bounding box coordinates. During cropping, a small amount of edge redundancy can be added to avoid truncation of edge characters due to the detection box being too close to the edge. For instance, for the field value region "Shenzhen City, District, Road, Number," if the detection box is too close to the edge, the last character may be cut off; therefore, a preset pixel margin can be added around the edges for cropping. After cropping, OCR recognition is performed on each layout unit. The text recognition module used can be an OCR model suitable for Chinese, English, numbers, and mixed character scenarios. In some implementations, for the title region, a recognition strategy suitable for large font sizes and short text can be adopted; for the field value region, the recognition capability for mixed content such as dates, numbers, amounts, and names can be enhanced; for the explanatory text region, the recognition capability for multi-line natural language sentences can be enhanced; and for local text in the table region, a finer-grained cell-level text recognition process can be used. After OCR recognition, the text string within the layout unit is output. For example, a field name area outputs "Contact Number," and its right-hand field value area outputs "13812345678"; a description text area outputs "This form is only for hospital admission registration; all information must be filled in truthfully"; a cell in a table area outputs the number of days of hospitalization. Further, the position coordinates of each character or text fragment in the text string within the original form image are obtained. These position coordinates can be the coordinates of the character's bounding box or the coordinates of the text fragment's bounding box. Specifically, the character positions can be obtained first in the local area coordinate system, and then, based on the offset of the layout unit in the original image, these character coordinates are mapped back to the original form image coordinate system, thus ensuring that all text objects are within a unified spatial reference frame. Subsequently, the text string is associated and stored with the corresponding character-level coordinate information. For example, a data record can be created for each layout unit, containing the area identifier, area type, area bounding box coordinates, OCR text result, and the position box information of each character in the text. For multi-line text areas, the line number to which the character belongs, the line order, and the fragment grouping result can also be recorded for subsequent field concatenation. For example, in an application form, the field value area corresponding to "Contact Address" might be recognized as two lines of text: the first line reads "Xihu District, Hangzhou City, Zhejiang Province," and the second line reads "Room 1208, XX Building, XX Road." After preserving the character-level coordinate information, the system not only knows the text content of the area but also which part is on the first line, which part is on the second line, and the spacing and indentation between the two lines.

[0024] In some examples, the process of generating a region adjacency graph for the form layout based on the fusion modeling of region type information, spatial location information, text content, and character-level coordinate information of each layout unit includes: Extract the vertical, horizontal, alignment, spacing, and containment relationships between layout units; Extract the semantic features of fields in the text content corresponding to the layout unit; Based on the region type information, the spatial location information, and the field semantic features, an adjacency relationship is established between layout units to generate a region adjacency relationship graph that represents the structural relationship of the form.

[0025] For example, each layout unit obtained from layout detection is treated as a graph node. For each node, attributes such as region type information, bounding box coordinates, region center point, width, height, OCR text content, character-level coordinate set, and recognition confidence are recorded. Next, the vertical, horizontal, alignment, spacing, and containment relationships between layout units are extracted. For example, if the right boundary of node A is located to the left of the left boundary of node B, and the two have a large vertical projection overlap, a right-side adjacency relationship from A to B can be established; if node A is above node B and the two are horizontally aligned, a vertical correspondence relationship can be established; if a field value region falls entirely within the table area, a containment relationship can be established. In some implementations, alignment relationships can be further refined into left alignment, right alignment, center alignment, top boundary alignment, and baseline alignment. Spacing relationships can be represented by horizontal distance, vertical distance, and normalized distance between regions. For example, in a personnel information registration table, the horizontal spacing between "name" and its right-hand field value may be small, while the horizontal spacing between "name" and the next field group is significantly larger. By comparing these spacings, the discriminative ability of subsequent matching can be improved. Then, the semantic features of the fields in the text content corresponding to the layout units are extracted. These semantic features can be obtained through keyword recognition, category dictionary matching, statistical linguistic feature analysis, or semantic vector representation. For example, "name," "gender," "date of birth," and "address" typically represent field name semantics; "Zhang San," "male," "December 1, 1998," and "No. X, X Road, Chaoyang District, Beijing" typically represent field value semantics; "description," "remarks," and "notes" are closer to descriptive text semantics. By introducing semantic features, the system can further distinguish areas that are similar in appearance but have different functions. Based on area type information, spatial location information, and field semantic features, adjacency relationships are established between layout units. Adjacency relationships can include edge type and edge weight. Edge weight can comprehensively reflect the relative position rationality, alignment rationality, distance rationality, and semantic compatibility between two areas. For example, a field name area can be assigned a higher weight to a field value area within a short distance to the right, while a field name area can be assigned a lower weight to a distant descriptive text area. Finally, an area adjacency relationship graph is generated to represent the structural relationships of the form. This diagram can represent both the adjacency relationships between regular key-value areas and the special relationships between description areas, table areas, and ordinary field areas. For example, in the first page of an inpatient medical record, there is a vertical structural relationship between the header node and the body text node; there are horizontal or vertical pairings between several field name nodes and field value nodes in the body text; the lower middle table area nodes contain multiple table cell relationships; and the bottom description nodes have an auxiliary description relationship with the field groups above them.

[0026] In some examples, the dynamic generation of field parsing paths based on the region adjacency graph and preset rule constraints, to perform key-value pair matching between field name regions and field value regions without relying on a fixed template, includes: Establish a rule constraint library, which includes at least the relative position constraints between the field name area and the field value area, the same row priority constraint, the proximity priority constraint, the area type constraint, and the text semantic consistency constraint. Based on the layout structure detected in the form to be parsed, candidate matching paths are dynamically selected from the region adjacency graph; Based on the rule constraint library, candidate matching paths are filtered and sorted to determine the target matching relationship between the field name area and the field value area.

[0027] For example, first, a rule constraint library is established. This library can be pre-configured in the system or optimized through historical form sample statistics. Rule constraints include at least the following: First, relative position constraints. Field value areas are typically located to the right, below, or to the lower right of the field name area, so different priority levels can be set for these directions. For example, for horizontal forms, the priority of right-side candidates can be set higher than that of lower-side candidates; for business scenarios with many vertical forms, the priority of lower-side candidates can be increased. Second, same-row priority constraints. When the field name area and a field value area are in the same horizontal band and have a small vertical deviation, they can be considered to have a key-value relationship. For example, in a layout like "Name: Zhang San Gender: Male," same-row priority constraints can effectively prevent "Name" from incorrectly matching the value of "Date of Birth" below. Third, proximity priority constraints. When multiple field value candidate areas have reasonable directions and compatible types, the one closer to the field name area is preferred. For example, when there are two text blocks to the right of "Contact Number," the shorter text block that is closer is more likely to be the actual contact number, while a long explanatory text block further away should not be selected. Fourth, area type constraints. Field name areas are preferentially paired with field value areas and are not directly paired with title or descriptive text areas to form primary key value relationships. Table areas are preferentially parsed separately within the table and are not directly used as ordinary field values ​​to compete for data. Fifth, text semantic consistency constraints. There should be a semantic fit between field name text and candidate field value text. For example, "date of birth" tends to match date format content, "amount" tends to match numeric amount format, "ID number" tends to match long sequences of numbers and letters, and "gender" tends to match short texts such as "male" and "female". In some implementations, candidate matching paths are dynamically selected from the region adjacency graph based on the layout structure detected in the form to be parsed. Specifically, starting from each field name area node, the candidate field value nodes with adjacent edges are traversed to form a set of candidate paths. For complex scenarios, paths can be allowed to extend to two-hop or multi-hop adjacent nodes, for example, a field name can be connected to a primary value area first and then to a continuation area. Subsequently, the candidate matching paths are filtered and sorted based on the rule constraint library. Different constraints can be assigned different weights during sorting. For example, location rationality has a high weight, semantic consistency has a medium-high weight, distance constraints have a medium weight, and type compatibility is a hard constraint. The candidate path with the highest score that meets a preset threshold is determined as the target matching relationship. For example, in an employment registration form, there is a short text box to the right of the "Place of Origin" field name, with the explanatory text "Please fill in to the province / city level" below. After the candidate paths are generated, the short text box on the right satisfies constraints such as right-side priority, proximity priority, and field value type compatibility with "Place of Origin." Although the explanatory text below is adjacent, it does not meet the field value semantic compatibility constraint. Therefore, the system will confirm the short text box on the right as the target matching value.For example, in some vertically formatted forms, there are field values ​​below "Patient Name" and "Contact Number," while another group of fields is on the right. In this case, the system determines, based on the current layout, that multiple field names have a strong vertical alignment with their corresponding value areas below, thus dynamically increasing the score of the lower priority path and avoiding incorrect matching along the right side.

[0028] In some examples, parsing cross-region fields or multi-row fields includes: When field values ​​are distributed across multiple adjacent regions or multiple lines of text, multiple text fragments are merged based on character-level coordinate continuity, region adjacency, and text semantic integrity. When a table area is detected, the row and column relationships of the cell text within the table area are analyzed, and the table content is output as an independent structured sub-result. When a descriptive text area is detected, the content corresponding to the descriptive text area is marked as descriptive information to distinguish it from the parsing results of the field name area and the field value area.

[0029] For example, when a field value is detected to be distributed across multiple adjacent regions or multiple lines of text within the same region, the system merges multiple text fragments based on character-level coordinate continuity, region adjacency, and text semantic integrity. Character-level coordinate continuity is primarily used to determine whether multiple text fragments have a natural spatial continuity. For instance, the first text region to the right of a field name "Contact Address" is identified as "Gulou District, Nanjing City, Jiangsu Province," and the adjacent region below it is identified as "Room 1203, Building A, No. 88, XX Road." If the two regions have a reasonable horizontal indentation, a small vertical spacing, and the lower region is directly below or slightly indented from the upper region, they can be considered spatially continuous. Region adjacency is used to determine whether these regions are adjacent to each other in the global layout diagram and jointly attached to the same field name node. If two field value fragments both have a reasonable adjacency link with the same field name region, they are more likely to belong to the same field value rather than different fields. Text semantic integrity is used to determine whether the concatenation result constitutes reasonable semantics. For example, address fields typically allow consecutive location expressions, medical condition descriptions typically allow consecutive sentences, and remarks fields typically allow complete explanatory paragraphs. If two fragments are semantically coherent after concatenation, the merging reliability is higher; if the semantics are significantly fragmented after concatenation, the likelihood of merging should be reduced. In some implementations, for multi-line fields, text fragments can be grouped according to character line numbers, line spacing, and line indentation. For example, the "chief complaint" field value in a medical record table is "recurrent chest pain for three days, accompanied by chest tightness for one hour." After OCR recognition, two lines of text and their character-level positions can be obtained. The system merges them into a complete field value based on their characteristics of being within the same field value area, vertically continuous, and semantically coherent. In some implementations, when a table area is detected, the row and column relationships of the cell text within the table area are analyzed. Specifically, several units in the table can be divided based on table lines, cell boundaries, or text clustering results, and then the table structure can be reconstructed based on the row and column positions of the cells. The reconstructed table content can be output as an independent structured sub-result, for example, represented as a list nested dictionary. In this way, the content such as "Item Name," "Quantity," "Unit Price," and "Amount" in the table will not be mistakenly split into ordinary flat fields, but will retain its table semantic structure. In some implementations, when a descriptive text area is detected, the corresponding content in the descriptive text area is marked as descriptive information. This marking can be reflected through field type labels, output level labels, or metadata fields. For example, texts such as "All content in this table must be filled in by the individual" or "The following items are for medical staff only" are not output as ordinary key-value pairs, but are stored separately as descriptive information or remarks to distinguish them from the parsing results of the field name area and field value area. For example, in a hospital registration form, the "Current Address" field value may be output across two rows; there is a "Past Medical History" table in the middle; and there is a "Filling Instructions" section at the bottom.This example solution can merge two lines of addresses into a single complete field value, organize the medical history table into an independent table structure, mark the filling instructions as explanatory information, and finally output a structured result that is more consistent with the semantics of real documents.

[0030] In some cases, while field name regions and field value regions may appear to be in the same row or horizontal band at the overall bounding box level, this visual alignment does not necessarily mean they belong to the same key-value pair semantically. This phenomenon is often not caused by a disordered field layout itself, but rather by the combined effects of the field name's font size, bolding style, colon position, leading line at the end, detection box clipping deviation, and the layout of adjacent field groups. Therefore, in some examples, the process of dynamically generating field parsing paths based on the region adjacency graph and predefined rule constraints to perform key-value pair matching between field name regions and field value regions includes: Based on the character-level coordinate information of the characters within the field name area, extract the character baseline information, text centroid information, and tail guide symbol position information of the field name text; Based on the coordinate information of the first character or the first text fragment in the candidate field value region, a semantic connection baseline is constructed between the field name region and the candidate field value region; Based on the continuity between the semantic connection baseline and the character baseline information, the target matching direction between the field name region and the candidate field value region is determined to correct key-value mismatches in visual pseudo-alignment scenarios.

[0031] In some examples, during key-value direction inference, instead of directly relying on the overall bounding box of the field name area, it delves deeper into the character layout level within the field name, extracts the character baseline information, text center-of-gravity information, and the position information of the trailing guiding symbol of the field name text, and reconstructs the visual semantic direction actually pointed to by the field name based on this. When the system has completed layout unit detection and OCR recognition, for each field name area, instead of immediately searching for the field value area on its right or below in the traditional way, it first enters the fine-grained analysis stage of the field name. First, perform character-level parsing on the OCR recognition results within the field name area to obtain the bounding box coordinates, character center coordinates, and character arrangement order of each character in the field name. Based on these character coordinates, the character baseline of the field name can be fitted. For example, for the field name "Document Type:", the system can recognize the horizontal arrangement relationship of the characters "证", "件", "类", "型", ":", and obtain an approximately horizontal baseline. For vertically arranged field names or line-break field names, corresponding vertical or broken-line baselines can be fitted respectively. Secondly, extract the text center-of-gravity information of the field name. The text center-of-gravity information is not only used to judge the overall position, but more importantly, it is used to measure which side the field name area visually leans towards. For example, if the last valid character of the field name and the colon are both concentrated at the right end of the area, it means that the field is more likely to look for the field value to the right; if the field name is in a stacked form and the last character at the end of the last line is at the bottom, it means that the possibility of looking down is increased. Subsequently, recognize the trailing guiding symbol in the field name area. The trailing guiding symbol can include a colon, an underscore, a dot guiding line, a short horizontal guiding line, a bracket guiding area, etc. In many forms, although these guiding symbols do not directly carry text semantics, they have a strong hinting effect on the layout direction of the field value. For example, the colon in "Name:" usually indicates that the field value is on the right; if there is a blank line below "Contact Phone Number", it means that the filled content may be within or after the line; the dot guiding line in some forms indicates that the value area expands along the direction of the guiding line. After determining the character baseline, text center-of-gravity, and trailing guiding symbol of the field name, the system starts from the field name area and evaluates the surrounding candidate field value areas. For each candidate value area, extract the coordinate information of its first character or the first text segment. Then, starting from the last valid character or the trailing guiding symbol of the field name and ending with the first character of the candidate value area, construct a semantic connection baseline. In some embodiments, the system calculates the angular difference, extension continuity, and whether it passes through an unreasonable intermediate area between the semantic connection baseline and the character baseline of the field name. For example, if the character baseline of the field name is horizontally to the right and the first character of a certain candidate value area is exactly on the right and slightly lower along the same horizontal band, then the direction consistency between its semantic connection baseline and the character baseline is relatively high, and the matching priority is increased.If another candidate value region, although centered in approximately the same row, has its first character located further to the upper right or separated from the end of the field name by another field group, its consistency decreases. In some implementations, this scheme can also work in conjunction with existing same-row priority constraints and proximity priority constraints, but baseline consistency within the field name should be given higher weight. For example, in a hospital registration form, to the right of "Marital Status:" is an exclusive field "Married / Unmarried," and below it in the same column is a "Occupation:" field. If only the frame position is considered, both candidate directions may be valid; however, by analyzing the baseline continuity between the colon at the end of the field name and the first character of the exclusive field on the right, the area on the right can be preferentially identified as the target value region. As another example, in an approval form, the field name "Applicant Unit" is relatively long, and the colon at the end is slightly downward due to printing offset. To the right of it is the value box of another column, "Approval Opinion," and below it is the actual value of "Applicant Unit." At this point, the system can determine the candidate area below based on the position of the center of gravity of the last line of characters in the field name and the direction of the semantic connection baseline. Although the candidate area below is not dominant in the center of the frame, it matches the internal layout structure of the field name better, thus correcting the mismatch result.

[0032] In some cases, the parsing of field values ​​may be partially cut off by non-text occlusions. Non-text occlusions refer to non-text objects such as stamps, checkmarks, handwritten annotations, imprints, confirmation marks, and signature lines. These differ from ordinary noise because they typically have a defined shape, position, and overlapping relationship, and often partially cover the originally continuous field value content, causing OCR to segment the same field value into two or more discontinuous text fragments. Therefore, in some examples, the parsing of cross-region or multi-line fields includes: Detect non-text occlusions within the field value range; When two text segments are located on opposite sides of the same non-text occluder, a cross-occlusion continuation candidate relationship is established based on the consistency of character height, baseline direction, and field semantics of the two text segments, as well as the spatial breakage relationship caused by the non-text occluder. Based on the cross-occlusion continuation candidate relationship, text fragments located on both sides of the non-text occlusion body are spliced ​​together to restore the content of the same field value that was cut off by occlusion.

[0033] In some examples, occlusion can be viewed not merely as noise, but as a structural event causing field value breaks. The consistency of text fragments on both sides of the occluded object in character height, baseline direction, field semantics, and spatial layout can be used to recover the possible relationship that they belong to the same field value. After the system completes layout detection and OCR recognition, for each field value region, it analyzes not only the text content but also whether there are non-text occluders within that region. Non-text occluders can be identified through various visual features. One approach is based on color anomaly regions. For red stamps, blue stamps, and colored checkmarks, the color distribution that is significantly different from black text in the color space can be used for location. Another approach is based on texture anomalies. Stamps or handwritten annotation areas often have edge continuity, filling patterns, and local textures different from printed characters. Yet another approach is based on shape features. For example, official seals often have circular or elliptical outlines, checkmarks present as slanted broken lines or small hook-shaped structures, and embossed stamps may exhibit partially translucent boundaries. After identifying the occlusion, the system further analyzes whether there are text fragments on both sides of the occlusion. "Both sides" is not limited to strictly left and right sides; it can also refer to relative positions such as above and below the occlusion, or upper left and lower right, as long as these fragments share the same main direction in character arrangement. Subsequently, the system extracts the character height, character width, baseline direction, character spacing, and text category features of the text fragments on both sides. For example, for a string of coded field values, the characters before and after the occlusion often have consistent font size, collinear baselines, and stable character spacing; for addresses or descriptive text, the text fragments before and after the occlusion may have continuity at the natural language level. In some implementations, the system establishes cross-occlusion continuation candidate relationships. This relationship can be established based on several conditions: First, the text fragments on both sides are located within the same field value region, or within adjacent field value sub-regions associated with the same field name region. Second, the positional distribution of the text fragments on both sides relative to the occlusion conforms to the spatial pattern of the occluded body, rather than being two independent normal fields. Third, the differences in character height, baseline direction, and font style between the two sides are small. Fourth, the semantic types of the two sides are compatible with the current field name. For example, when the field name is "ID number", the segments before and after the occlusion should exhibit a continuous encoding pattern; when the field name is "address", the continuity of natural language phrases is emphasized. After establishing candidate relationships, the system executes different concatenation strategies based on the field type. If the field is an encoded field, such as ID number, unified social credit code, medical record number, hospitalization number, contract number, etc., the system prioritizes inferring the number of characters that may be missing in the occluded area based on fixed length rules, encoding bit length rules, or format constraints, and outputs the segments on both sides of the occlusion as the first and second halves of the same field value.For example, for a unified social credit code like "9144****1234567890", if the sum of the lengths of the segments before and after the occlusion is less than the standard length, the middle section can be marked as a missing area, but the same field value object will still be output. If the field is a natural language field such as address, unit name, medical condition description, or approval opinion, the system usually does not forcibly infer the specific missing character, but retains the break mark at the occlusion position or only splices the segments, and records in the result that the field was occluded. In some implementations, if the occlusion is a checkmark or a short handwritten annotation, and does not cover the shape of the characters on a large scale, the system can further use a local context recovery strategy. For example, if the field value is "RMB One Thousand Two Hundred Yuan Only", and the middle part is partially covered by a small confirmation stamp, but the semantics before and after are highly complete, the system can recover the overall field value based on the amount format and contextual language habits. For example, in a hospital registration form, the recognition result of the "Current Address" field is "Nanshan District, Shenzhen" in the first half and "No. X Road, Science Park" in the second half, with the middle area covered by a "Verified" red stamp. The system first identifies the outline of the red seal, then discovers natural language text fragments on the same baseline to its left and right, both of which are associated with the "Current Address" field name. Therefore, a cross-obscuration continuation relationship is established, and the complete address field is output. For example, in an appendix to a business license copy, the "Unified Social Credit Code" is obscured by a circular official seal. Although the system cannot recover the exact characters that are completely obscured, it can treat the two segments as a single encoded field based on the encoding length and field semantics, retaining a missing area indication instead of incorrectly splitting them into two independent fields.

[0034] In some cases, the field name itself may be split into multiple layout regions. This splitting sometimes stems from the form design itself, such as narrow column layout causing label wrapping; sometimes it comes from scanning, cropping, or layout detection processes, such as large letter spacing within the field name, layered printing, or being misidentified as two independent label blocks. As a result, in the subsequent key-value matching stage, the system treats these split field name fragments as multiple independent field names, leading to repeated matching, incorrect matching, or even outputting multiple incomplete field names for the same field value region. Based on this, in some examples, the dynamic generation of field parsing paths based on the region adjacency graph, combined with preset rule constraints, to perform key-value pair matching between the field name region and the field value region includes: When a field value region forms a candidate association relationship with multiple field name candidate regions at the same time, the multiple field name candidate regions are aggregated based on the semantic continuation relationship and layout splicing relationship between the multiple field name candidate regions to generate a composite field name node. Based on the association between the composite field name node and the field value area, the target key value matching relationship is determined. The layout splicing relationship includes at least one of the following: vertical stacking relationship, horizontal adjacent relationship, font consistency relationship, and edge spacing relationship.

[0035] In some examples, when a field value region forms candidate associations with multiple field name candidate regions simultaneously, it is not immediately treated as a multi-field shared value. Instead, the system reversely determines whether these field name candidate regions essentially belong to the same split composite field name, and aggregates the field names if necessary before performing key-value matching. After completing the construction of the region adjacency graph, the system counts the number of candidate associations between each field value region and its surrounding field name candidate regions. When a field value region is found to form high-scoring candidate relationships with two or more field name candidate regions simultaneously, the system does not immediately perform many-to-one matching, but instead enters the field name aggregation judgment stage. First, the semantic continuation relationship between these field name candidate regions is extracted. The semantic continuation relationship can be judged by the rationality of the words after text concatenation. For example, if candidate label one is identified as "contact location" and candidate label two is identified as "address", then concatenating the two constitutes "contact address", which is a highly reasonable field name semantically. Similarly, concatenating "work unit" and "and position" constitutes the complete semantic unit "work unit and position". In some implementations, tag dictionary matching, word boundary judgment, phrase templates, or semantic similarity analysis can be used to evaluate the rationality of this continuation. Secondly, the layout and splicing relationships between these candidate field name regions are extracted. These relationships can include at least vertical stacking, horizontal adjacency, consistent font, similar font size, small margins, and maintaining an overall directional correspondence with the same field value region. For example, if two candidate tags are arranged one above the other, with relatively aligned left and right boundaries, and both located to the left of the same value region, it indicates they are more likely to be the same field name split into two lines. If two candidate tags are horizontally adjacent, have the same font style, and the spacing between them is smaller than the spacing with other fields, it indicates they may have originally been a continuous tag. In some implementations, the system can combine semantic continuation scores and layout and splicing scores to determine whether these candidate field name regions should be aggregated. If the aggregation conditions are met, multiple candidate field name regions are merged into a composite field name node. This node can inherit the joint bounding box of each candidate region or retain internal order information for subsequent reference. For example, for the two-line labels "Work Unit" and "Job Title", the system can record that the composite field name node is composed of an upper and lower line segment, with the overall text being "Work Unit and Job Title". After the composite field name node is established, the system uses this node to replace the original multiple fragment nodes in subsequent matching in the region adjacency graph. The candidate edges of field values ​​that were originally connected to each fragment node are re-converged to the composite node, and the matching score is recalculated based on the new overall direction and overall semantics. This avoids a field value region being repeatedly competed for by multiple incomplete labels. For example, in an employee registration form, the two lines on the left are identified as "Work Unit" and "Job Title" respectively, while the right side is a long text region "R&D Manager of [Company Name] Technology Co., Ltd."If a direct match were used, the system might assign the value region to two separate labels. Using this approach, the system first identifies a value region on the right that is associated with two labels on the left. Then, based on semantic continuation and stacking relationships, it merges the two labels into "Work Unit and Position," ultimately achieving a correct one-to-one match. For example, in a historical scanned file, "Contact Person's Phone Number" is printed as two consecutive short blocks, "Contact Person" and "Contact Number," with the value region "Zhang San 13812345678." The system can determine that these two labels share a common correspondence with the same composite value region on the right. If semantic analysis suggests it should represent a composite field name, label aggregation can be completed first, followed by a decision on whether to further decompose the internal structure of the value region.

[0036] In some cases, considering that the internal structure of a table area is not always a simple row-column intersection structure, some cells may also contain hidden relationships between local field names and values. In other words, a table area is not necessarily a single semantic plane; some cells may contain a second layer of structure. If the system treats all cells within a table area as "ordinary table values" as soon as it detects them, it will ignore the local key-value relationships within these cells, resulting in an overly flat structural representation and even the loss of field semantics. Therefore, in some examples, when a table area is detected, row-column relationship analysis is performed on the cell text within the table area, and the table content is output as an independent structured sub-result, including: Divide the table range into cells; Perform local field-triggered feature recognition on the text within each cell to determine whether the corresponding cell belongs to a pure table value range or a local field range; When a cell is a local field, perform local key-value pair matching on the field name text and field value text within that cell; When a cell contains only a pure table value, output the table value content according to the row and column position of the cell.

[0037] In some examples, after detecting a table region, instead of directly assuming all cells within the table belong to a single value domain, the system further distinguishes between cells belonging to a pure table value domain and local field domains. For cells belonging to local field domains, local field parsing is performed again within the cell. Once the system has detected that a layout unit belongs to a table region, it first performs cell segmentation within the table region. Cell segmentation can be accomplished through table line detection, text clustering, projection segmentation, row and column boundary estimation, etc. The segmentation result should at least obtain the boundary range of each cell and its row and column position in the table. After completing cell segmentation, the system performs local field trigger feature recognition on the text within each cell. Local field trigger features refer to signals that indicate "the cell contains not just a regular value, but more like a small field structure." These signals can include one or more of the following: the presence of colons in the cell text, a structure with short labels followed by long values, the presence of typical field name keywords, the presence of local checkbox structures, and patterns of short text labels combined with numbers or dates, etc. For example, in a cell with text "Blood pressure: 120 / 80", the colon is a clear trigger feature; in a cell with text "Height 172cm", although there is no colon, the combination of "height" followed by a number and unit conforms to the short label and value combination structure; a cell with text "Agree" is more like a regular value and may not trigger local field parsing. In some implementations, the system determines whether a cell belongs to a pure table value domain or a local field domain based on the strength of the trigger feature. For cells with pure table value domains, the system maintains the conventional table parsing method, that is, expressing its semantics by the row and column position of the cell in the table. For example, in a cost table, the cell "200.00" may only represent the amount of a certain project and does not need to be split further. For cells with local field domains, the system further performs local field name recognition and field value recognition within the cell boundaries. At this time, the cell is equivalent to a mini form, and lightweight layout analysis and text segmentation can also be performed inside it. In some implementations, local field parsing can adopt a simplified key-value matching process. For example, if a cell contains a colon, the preceding short text can be used as a candidate field name, and the following text as a candidate value, then confirmed by combining the number format, date format, and unit format. If a cell contains multiple local fields, such as "Opinion: Agreed Date: 2025-04-20", it can be further divided into two micro-field groups and output separately. Ultimately, when outputting structured results, the table still maintains its status as an independent structured sub-result, but the content of some cells is no longer just a flat string, but can be output as a nested object containing internal key-value pairs. For example, the cell in row 3, column 2 of the table can be output as "{Field Name: Blood Pressure, Field Value: 120 / 80}", instead of simply outputting "Blood Pressure: 120 / 80".For example, in a medical examination report, several cells in the "Examination Results" column might display "Blood Pressure: 120 / 80", "Pulse: 72 beats / min", and "Temperature: 36.6°C". With this solution, these cells are recognized as local field fields. The system not only retains their row and column information but also outputs the local field names and values ​​("Blood Pressure", "Pulse", "Temperature") within the cells. Similarly, in an approval process table, a cell might display "Review Comment: Agreed". The system can parse this as a local field structure, rather than a simple string.

[0038] In some cases, considering that multiple semantically mutually exclusive candidate options exist simultaneously in the field value region, and the truly valid value is not determined by the text itself, but by checkmarks, circles, fill marks, or local stroke variations, the candidate option text in these scenarios is usually in a valid position and geometrically normal. Therefore, relying solely on OCR text content and positional relationships, the system can easily output all candidate options together, resulting in redundant results or semantic errors. Based on this, in some examples, the method of dynamically generating field parsing paths based on the region adjacency graph and pre-defined rule constraints to perform key-value pair matching between the field name region and the field value region includes: Create an exclusive candidate group containing multiple candidate option texts in the field value area; Extract check marks, enclosing marks, filling marks, local stroke thickening marks, or imprint marks from the adjacent areas of each candidate option text as evidence for selection; Based on the strength of evidence for each candidate option text, the target option text in the exclusive candidate group is determined, and the target option text is used as the field value that matches the corresponding field name area.

[0039] In some examples, for candidates with mutually exclusive semantics, they are not directly output as ordinary field value strings. Instead, an exclusive candidate group is first established, then selection evidence is extracted from the vicinity of each candidate, and finally the target output item is determined within the exclusive group based on the strength of the selection evidence. When the system detects multiple short text candidates in a field value area, and these text candidates semantically constitute an exclusive set with the field name, the system first establishes an exclusive candidate group. The establishment of the exclusive candidate group can be based on a pre-defined dictionary of mutually exclusive words, a dictionary of exclusive field names, or statistics of candidate sets. For example, when "male" and "female" appear under the "gender" field, they can be directly identified as an exclusive candidate group; when "married" and "unmarried" appear under the "marital status" field, an exclusive group can also be established. After establishing the exclusive candidate group, the system extracts selection evidence from the vicinity of each candidate text area. Selection evidence can include check marks, enclosing marks, filling marks, local stroke thickening marks, imprint marks, strikethrough marks, or symbols associated with the candidate. The selection methods may differ across forms, therefore, selection evidence extraction should be flexible. For example, in some paper forms, a checkmark may appear before a selected item; in others, it may be circled; in approval forms, an unselected item may be marked with a diagonal line; and in scanned documents after electronic printing, a selected item may be near a solid square. The system can extract this evidence through local shape detection, edge direction statistics, pixel density analysis, and connected component features. In some implementations, the system analyzes not only the small area to the left or right of the candidate item itself, but also its upper and lower neighbors and surrounding areas. For example, for "male / female" side-by-side options, a checkmark may be located on the left side of the text; for "yes / no" vertically arranged options, a circle may surround the entire text; and for "agree / disagree," a strikethrough may be drawn on a specific item. The system can employ differentiated neighborhood sampling strategies for different layout formats. After extracting the selection evidence, the system calculates the strength of selection evidence for each candidate item in the exclusive candidate group. The strength of evidence can be comprehensively considered based on the existence of traces, the consistency of trace positions, the binding distance between traces and candidate text, the reliability of trace shapes, and the existence of conflicting multiple pieces of evidence. For example, if a checkmark is located exactly to the left of "male" and very close, then the evidence strength for "male" is high; if there are no traces around "female," then its strength is low. If there are weak traces around both candidate options, further adjudication can be made by combining explanatory text or local rules. In some implementations, if a field does not have an explicit checkmark, but there is an explanatory area stating "retained items are valid items" or "cross out inapplicable items," the system can also feed back the constraint rules in the explanatory text into the exclusion determination. For example, if "yes" remains intact and "no" is crossed out in the two candidate options "yes," then "yes" should be output. Another example is that if "disagree" is crossed out in "agree / disagree," then "agree" should be output.Ultimately, the system no longer outputs all text from the exclusive candidate group together. Instead, it outputs the target option text as the field value after evaluation, while other candidate options can be retained as auxiliary information or ignored directly. For example, in a hospital registration form, the "Gender" field displays "Male and Female," with a checkmark to the left of "Male" and no mark for "Female." After the system creates an exclusive candidate group, it only outputs "Male." Another example is in a questionnaire where "Do you smoke: Yes or No" is selected, so the final output is "No." Yet another example is in an approval form where "Review Comments: Agree or Disagree" is crossed out, so the system determines "Agree" as a valid result based on exclusive evidence.

[0040] In some cases, due to factors such as scan compression, image noise, character adhesion, or background filling, the blank boundaries between two adjacent fields may be visually smoothed out, causing the layout detection results to incorrectly merge two field value regions that should be independent into an excessively long region. This phenomenon can be called false extension of field boundaries. Based on this, in some examples, after calling the text recognition module to perform text recognition for each detected layout unit to obtain the text content and character-level coordinate information of the corresponding layout unit, the method further includes: When the length of the field value region exceeds the preset length range of the same type of field value region, and there are multiple text clusters within the field value region, the potential boundary valley positions in the field value region are determined based on the character-level coordinate distribution, the gap density curve between characters, and the local background pixel sparsity. Based on the potential boundary valley locations, the field value region is further segmented to generate multiple candidate sub-regions; The multiple candidate sub-regions are re-added to the region adjacency graph to perform a rematch between the field name region and the field value region.

[0041] In some examples, when a field value region exhibits abnormal length, significant separation of internal text clusters, or potential gaps and troughs in character distribution, the original layout detection result may not be directly accepted. Instead, the system utilizes the character-level coordinate distribution, inter-character gap density variation, and local background sparsity after OCR output to perform secondary boundary segmentation on the field value region, thereby restoring the originally independent multiple field value sub-regions. After the system has performed OCR recognition on the layout unit and obtained character-level coordinates, it will perform length reasonableness checks on each field value region. The so-called length reasonableness can be obtained by comparing it with the statistical length range of similar field value regions. For example, in the same form or form of the same business type, field values ​​such as "contact number," "postal code," "age," and "gender" are usually not particularly long. If the length of a certain field value region significantly exceeds the average length of similar field value regions in the same column, it can be marked as a suspicious long region. After determining that a field value region is a suspicious long region, the system further analyzes whether there are multiple text clusters within it. Text clusters can be obtained through character-level coordinate clustering, character spacing statistics, or text fragment grouping. For example, if a long region consists of an eleven-digit phone number in the first half and a six-digit postal code in the second half, even though it is connected by noise, there may still be gaps between characters that are slightly larger than normal character spacing. The system can construct a character gap density curve by sorting the horizontal coordinates of all characters and calculating the gap length between adjacent characters. Subsequently, the system analyzes the potential boundary trough locations in the character gap density curve. A boundary trough refers to a location within a long region where the character distribution suddenly becomes sparse, and the background pixels relatively increase, indicating that this location may have originally been a blank boundary between two fields. To avoid misjudging normal spaces within long text as boundaries, the system also combines local background pixel sparsity, character cluster width ratio, and expected field type for comprehensive judgment. In some implementations, the analysis of local background pixel sparsity can be achieved by observing the proportion of binarized background near the candidate segmentation location, the interruption of connected components, and the horizontal projection valley value. If a location not only has an abnormally large character spacing but also a significantly increased background proportion, it is more likely to be a true boundary rather than an internal character gap. After identifying the potential boundary trough location, the system performs a secondary segmentation of the field value region at that location, generating two or more candidate sub-regions. Each segmented sub-region retains the original OCR text fragment and character coordinate set, and can also be re-sent to the lightweight OCR module for local correction and recognition. Subsequently, these sub-regions are re-added to the region adjacency graph and participate again in the matching between field names and field values. For example, in a low-quality scanned employee registration form, the values ​​of "contact number" and "postal code" are clustered together into a long region due to scanning noise, and the OCR output is "13812345678510000".The system detected that the field's value range was significantly longer than similar fields, and that there was a clear trough between the characters "8" and "5" in the spacing. Furthermore, the numeric patterns to the left and right of this position conformed to the formats of a mobile phone number and a postal code, respectively. Therefore, the system segmented the region at this trough, recovering two sub-regions: "13812345678" and "510000," which were then re-matched with the field names "Contact Number" and "Postal Code," respectively. In some implementations, to prevent accidental segmentation of truly long fields such as addresses, remarks, and explanatory text, the system also incorporates field type and semantic analysis for protection. For example, if the field name is "Current Address" or "Remarks," even if the region is long, it should not be easily segmented simply because of the presence of spaces or line breaks. Secondary segmentation is only performed when the region length is abnormal, text clusters are clearly separated, local boundary troughs are obvious, and the segmented result better matches the layout of surrounding fields.

[0042] Please see Figure 2 An embodiment of the adaptive form layout parsing system based on YOLO and OCR in this application includes: The detection unit 21 is used to input the form image into the layout detection model to detect the layout area in the form image and output multiple layout units and the area type information corresponding to each layout unit. The area type information includes at least one of the following: title area, field name area, field value area, table area and explanatory text area. The recognition unit 22 is used to call the text recognition module to perform text recognition for each of the detected layout units in order to obtain the text content and character-level coordinate information of the corresponding layout unit. The fusion unit 23 is used to perform fusion modeling based on the region type information, spatial location information, text content and character-level coordinate information of each layout unit to generate a region adjacency relationship diagram of the form layout; The generation unit 24 is used to dynamically generate field parsing paths based on the region adjacency relationship graph and in combination with preset rule constraints, so as to perform key-value pair matching on the field name region and the field value region without relying on a fixed template, parse cross-region fields or multi-line fields, and generate the structured result of the form.

[0043] like Figure 3 As shown, this application embodiment also provides an electronic device 300, including a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor. When the processor 320 executes the computer program 311, it implements the steps of any of the above-mentioned methods of the adaptive form layout parsing method based on YOLO and OCR.

[0044] Since the electronic device described in this embodiment is the device used to implement the adaptive form layout parsing system based on YOLO and OCR in this application embodiment, those skilled in the art can understand the specific implementation method and various variations of the electronic device in this embodiment based on the method described in this application embodiment. Therefore, how the electronic device implements the method in this application embodiment will not be described in detail here. Any device used by those skilled in the art to implement the method in this application embodiment is within the scope of protection of this application.

[0045] In practical implementation, when the computer program 311 is executed by the processor, it can achieve the following: Figure 1 Any of the corresponding implementation methods in the embodiments.

Claims

1. A YOLO and OCR-based adaptive form layout analysis method, characterized in that, include: The form image is input into the layout detection model to detect the layout regions in the form image and output multiple layout units and the region type information corresponding to each layout unit. The region type information includes at least one of the following: title region, field name region, field value region, table region, and explanatory text region. For each of the detected layout units, the text recognition module is invoked to perform text recognition in order to obtain the text content and character-level coordinate information of the corresponding layout unit; Based on the region type information, spatial location information, text content and character-level coordinate information of each layout unit, a fusion model is performed to generate a region adjacency relationship graph of the form layout; Based on the region adjacency graph, and combined with preset rule constraints, field parsing paths are dynamically generated to perform key-value pair matching between field name regions and field value regions without relying on fixed templates, and to parse cross-region fields or multi-row fields to generate structured results for forms.

2. The method of claim 1, wherein, The step of inputting the form image into the layout detection model to detect the layout regions in the form image includes: The form image is processed by multi-scale feature fusion based on the improved YOLO object detection model to enhance the detection capability of small font regions, multi-line text regions and irregularly arranged regions. The detection anchor box is adaptively adjusted based on the size distribution characteristics of the target area in the form image to improve the positioning accuracy of different field areas, table areas and explanatory text areas; Output the bounding box coordinates, confidence level, and region type information of each layout unit.

3. The method of claim 1, wherein, For each of the detected layout units, the text recognition module is invoked to perform text recognition to obtain the text content and character-level coordinate information of the corresponding layout unit, including: Perform OCR recognition on each of the layout units to obtain the text string within the layout unit; Obtain the position coordinates of each character or text fragment in the original form image; The text string is associated with and stored with the corresponding character-level coordinate information to serve as the basis for subsequent field attribution determination and key-value pair matching.

4. The method of claim 1, wherein, The process involves fusing and modeling based on the region type information, spatial location information, text content, and character-level coordinate information of each layout unit to generate a region adjacency graph for the form layout, including: Extract the vertical, horizontal, alignment, spacing, and containment relationships between layout units; Extract the semantic features of fields in the text content corresponding to the layout unit; Based on the region type information, the spatial location information, and the field semantic features, an adjacency relationship is established between layout units to generate a region adjacency relationship graph that represents the structural relationship of the form.

5. The method of claim 1, wherein, The step of dynamically generating field parsing paths based on the region adjacency graph and combined with preset rule constraints to perform key-value pair matching between field name regions and field value regions without relying on a fixed template includes: Establish a rule constraint library, which includes at least the relative position constraints between the field name area and the field value area, the same row priority constraint, the proximity priority constraint, the area type constraint, and the text semantic consistency constraint. Based on the layout structure detected in the form to be parsed, candidate matching paths are dynamically selected from the region adjacency graph; Based on the rule constraint library, candidate matching paths are filtered and sorted to determine the target matching relationship between the field name area and the field value area.

6. The method of claim 1, wherein, The parsing of cross-region fields or multi-row fields includes: When field values ​​are distributed across multiple adjacent regions or multiple lines of text, multiple text fragments are merged based on character-level coordinate continuity, region adjacency, and text semantic integrity. When a table area is detected, the row and column relationships of the cell text within the table area are analyzed, and the table content is output as an independent structured sub-result. When a descriptive text area is detected, the content corresponding to the descriptive text area is marked as descriptive information to distinguish it from the parsing results of the field name area and the field value area.

7. The method of claim 1, wherein, The step of dynamically generating field parsing paths based on the region adjacency graph and combined with preset rule constraints to perform key-value pair matching between field name regions and field value regions without relying on a fixed template includes: Based on the character-level coordinate information of the characters within the field name area, extract the character baseline information, text centroid information, and tail guide symbol position information of the field name text; Based on the coordinate information of the first character or the first text fragment in the candidate field value region, a semantic connection baseline is constructed between the field name region and the candidate field value region; Based on the continuity between the semantic connection baseline and the character baseline information, the target matching direction between the field name region and the candidate field value region is determined to correct key-value mismatches in visual pseudo-alignment scenarios.

8. An adaptive form layout parsing system based on YOLO and OCR, characterized in that, include: The detection unit is used to input the form image into the layout detection model to detect the layout area in the form image and output multiple layout units and the area type information corresponding to each layout unit. The area type information includes at least one of the following: title area, field name area, field value area, table area and explanatory text area. The recognition unit is used to call the text recognition module to perform text recognition for each of the detected layout units in order to obtain the text content and character-level coordinate information of the corresponding layout unit. The fusion unit is used to perform fusion modeling based on the region type information, spatial location information, text content and character-level coordinate information of each layout unit to generate a region adjacency relationship graph of the form layout; The generation unit is used to dynamically generate field parsing paths based on the region adjacency graph and in combination with preset rule constraints, so as to perform key-value pair matching on field name regions and field value regions without relying on fixed templates, parse cross-region fields or multi-line fields, and generate structured results of the form.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program stored in the memory, implements the steps of the adaptive form layout parsing method based on YOLO and OCR as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the adaptive form layout parsing method based on YOLO and OCR as described in any one of claims 1-7.

Images