Certificate information extraction method and system fusing spatial layout and text semantics

Abstract

The invention discloses a certificate information extraction method and system fusing spatial layout and text semantics, and the method comprises the steps: inputting an original certificate image into an image segmentation correction model for segmentation and correction, and outputting a standardized certificate image; inputting the standardized image into a target detection model, and detecting and outputting bounding box coordinates of a text region of a preset target language; intercepting image blocks based on the bounding box coordinates, inputting the image blocks into an optical character recognition model, and outputting a character sequence and a corresponding character coordinate sequence; inputting the character sequence and the character coordinate sequence into a sequence labeling model, mapping the character sequence and the character coordinate sequence into a text semantic feature vector through a first embedded network, and mapping the character sequence and the character coordinate sequence into a spatial layout feature vector through a second embedded network; inputting the two feature vectors into a fusion network for modeling and fusion, and generating a multi-modal feature vector; and finally, decoding the structured information label corresponding to each character through a sequence labeling decoding layer. According to the invention, the accuracy and robustness of certificate information extraction are effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and automatic processing of certificate information, and in particular to a certificate information extraction method and system fusing spatial layout and text semantics. BACKGROUND

[0002] In the field of computer vision and information processing, automatic extraction of identity certificate information is a key technology. The current mainstream technology follows the serial processing paradigm of "image preprocessing → text detection → character recognition → information structuring". Specifically, the existing scheme usually first performs geometric deformation processing such as tilt correction on the certificate image, then uses a target detection model to locate the text region, then converts the text in the image into a character sequence through an optical character recognition model, and finally integrates the character sequence according to a preset rule to output structured information.

[0003] Although this paradigm has achieved success under certain conditions, it has inherent and difficult-to-overcome technical defects. The core problem is that the entire process is essentially a process of "flattening" the two-dimensional information contained in the image into one-dimensional text. In the optical character recognition stage, the text region image is converted into a pure text string, and the key spatial layout information (such as relative position, row and column relationship) between characters is completely discarded. Once there is slight deformation, damage or compact text arrangement in the original image, the recognition order of the text line or the field division by the OCR engine will be wrong, and the subsequent extraction method relying only on one-dimensional text will not be able to correct due to the lack of spatial context, resulting in catastrophic recognition errors.

[0004] In addition, each processing module (such as correction, detection, and recognition) of the existing technology is usually optimized and connected independently, forming an "information island". The process is one-way, lacking effective feedback and collaborative mechanisms. Small errors produced by the previous modules (such as text detection) are passed on and amplified without correction to the subsequent modules, and the overall fault tolerance and robustness of the system are poor.

[0005] Therefore, the existing technology faces serious challenges in terms of accuracy and stability when dealing with complex and variable certificate images in the real world. There is an urgent need for a new certificate information extraction method that can effectively utilize both text semantics and spatial layout information and achieve deeper collaboration between modules. SUMMARY

[0006] In view of the above problems, the present application provides a certificate information extraction method and system fusing spatial layout and text semantics to solve the technical problems of low accuracy and poor robustness of certificate information extraction caused by the loss of text spatial layout information and the one-way isolation of the process in the existing methods.

[0007] To achieve the above object, in a first aspect, the application provides a document information extraction method fusing spatial layout and text semantics, comprising the following steps:

[0008] S1: inputting an inputted document original image into a deep learning-based image segmentation correction model, performing pixel-level semantic segmentation on a document main body region, performing perspective transformation and affine transformation based on vertex coordinates of an inscribed polygon of the segmented document main body region, and outputting a standardized document image with a fixed aspect ratio;

[0009] S2: inputting the standardized document image into a target detection model trained by target language region enhancement, to detect and output only the bounding box coordinates of the text region belonging to a preset target language in the standardized document image, and filter out the character region of non-pre-set target language;

[0010] S3: based on the bounding box coordinates of the text region, cutting the corresponding image block from the standardized document image, and inputting the image block into an optical character recognition model to output a character sequence and a corresponding character coordinate sequence, wherein the character sequence contains the character recognition result in each image block, and the character coordinate sequence contains the normalized coordinates of each character in the standardized document image;

[0011] S4: inputting the character sequence and the corresponding character coordinate sequence into a sequence labeling model to obtain the structured information label corresponding to each character, specifically comprising:

[0012] mapping the character sequence into a text semantic feature vector through a first embedding network;

[0013] mapping the character coordinate sequence into a spatial layout feature vector through a second embedding network;

[0014] inputting the text semantic feature vector and the spatial layout feature vector into a trainable fusion network, modeling and fusing the interaction relationship between the two kinds of feature vectors through the fusion network to generate a fused multi-modal feature vector;

[0015] inputting the multi-modal feature vector into a sequence labeling decoding layer to decode the structured information label corresponding to each character for identifying the document field type.

[0016] In a second aspect, the application provides a document information extraction system fusing spatial layout and text semantics, which is used to execute the method of the first aspect of the application, and comprises:

[0017] An image normalization module is configured to input an input original document image into a deep learning-based image segmentation correction model, perform pixel-level semantic segmentation on a document body region, perform perspective transformation and affine transformation based on vertex coordinates of an inscribed polygon of the segmented document body region, and output a normalized document image with a fixed aspect ratio.

[0018] A target text detection module is configured to input the normalized document image into a target detection model trained based on a target language region to detect and output only a bounding box coordinate of a text region in a preset target language in the normalized document image, and filter out a character region in a non-preset target language.

[0019] A character recognition and positioning module is configured to extract an image block corresponding to the text region from the normalized document image based on the bounding box coordinate of the text region, input the image block into an optical character recognition model, and output a character sequence and a corresponding character coordinate sequence, where the character sequence includes a character recognition result in each image block, and the character coordinate sequence includes a normalized coordinate of each character in the normalized document image.

[0020] A multi-modal information structuring module is configured to input the character sequence and the corresponding character coordinate sequence into a sequence labeling model to obtain a structured information label corresponding to each character. The multi-modal information structuring module includes:

[0021] A text semantic embedding unit is configured to map the character sequence into a text semantic feature vector through a first embedding network.

[0022] A spatial layout embedding unit is configured to map the character coordinate sequence into a spatial layout feature vector through a second embedding network.

[0023] A feature fusion unit is configured to input the text semantic feature vector and the spatial layout feature vector into a trainable fusion network, model and fuse an interaction relationship between the two feature vectors through the fusion network, and generate a fused multi-modal feature vector.

[0024] A sequence labeling unit is configured to input the multi-modal feature vector into a sequence labeling decoding layer to decode a structured information label corresponding to each character to identify a document field type.

[0025] Distinguish from the prior art, the technical scheme relates to a kind of certificate information extraction methods of fusing space layout and text semantics, the method includes: the original image of certificate is input to image segmentation correction model and is segmented and corrected, and the standardized certificate image is output;Standardized image is input to target detection model, and the boundary box coordinates of text area of preset target language are detected and output;Image block is intercepted based on boundary box coordinates, and is input to optical character recognition model, and the character sequence and corresponding character coordinate sequence are output;Character sequence and character coordinate sequence are input to sequence labeling model, are mapped into text semantic feature vector by first embedding network, and are mapped into spatial layout feature vector by second embedding network;Two kinds of feature vectors are input to the modeling and fusion of trainable fusion network, and generate multi-modal feature vector;Finally, the structured information label corresponding to each character is decoded by sequence labeling decoding layer.This application effectively improves the accuracy and robustness of certificate information structured extraction by deeply fusing text semantic and spatial layout information.

[0026] The above invention content related record is only the summary of the technical scheme of the present application, in order to let the ordinary skilled in the art can more clearly understand the technical scheme of the present application, then can be implemented according to the content of the description and the drawing record, and in order to let the above-mentioned purpose of the present application and other purposes, features and advantages can be more easily understood, the following combining the specific embodiment of the present application and the drawing are described. BRIEF DESCRIPTION OF DRAWINGS

[0027] The drawings are only used to show the principle, implementation, application, characteristics and effect of the specific embodiment of the present application and other related contents, and cannot be considered as the limitation of the present application.

[0028] In the drawings of the specification:

[0029] Figure 1 The flow chart of the certificate information extraction method of fusing space layout and text semantics described for the first exemplary embodiment of the present application;

[0030] Figure 2 The flow chart of the certificate information extraction method of fusing space layout and text semantics described for the second exemplary embodiment of the present application;

[0031] Figure 3 The flow chart of the certificate information extraction method of fusing space layout and text semantics described for the third exemplary embodiment of the present application;

[0032] Figure 4 The flow chart of the certificate information extraction method of fusing space layout and text semantics described for the fourth exemplary embodiment of the present application;

[0033] Figure 5A flowchart of the document information extraction method fusing spatial layout and text semantics according to the fifth exemplary embodiment of the present application is shown in the following figure;

[0034] Figure 6 A flowchart of the document information extraction method fusing spatial layout and text semantics according to the sixth exemplary embodiment of the present application is shown in the following figure;

[0035] Figure 7 A flowchart of the document information extraction method fusing spatial layout and text semantics according to the seventh exemplary embodiment of the present application is shown in the following figure;

[0036] Figure 8 A module schematic diagram of the document information extraction system according to an exemplary embodiment of the present application is shown in the following figure;

[0037] The reference signs involved in the above figures are explained as follows:

[0038] 10, document information extraction system; 101, image standardization module; 102, target text detection module; 103, character recognition and positioning module; 104, multi-modal information structuring module; 1041, text semantic embedding unit; 1042, spatial layout embedding unit; 1043, feature fusion unit; 1044, sequence labeling unit. DETAILED DESCRIPTION

[0039] To explain the possible application scenarios, technical principles, specific schemes that can be implemented, and the purposes and effects that can be achieved of the present application in detail, the embodiments described herein are explained in detail below in combination with the specific embodiments listed and the accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of the present application, and therefore cannot be used to limit the protection scope of the present application.

[0040] As shown in the following figure, Figure 1 In a first aspect, the present application provides a document information extraction method fusing spatial layout and text semantics, which comprises:

[0041] S1: input the input document original image into a deep learning-based image segmentation correction model, perform pixel-level semantic segmentation on the document main body region, and perform perspective transformation and affine transformation based on the vertex coordinates of the circumscribed polygon of the segmented document main body region, and output a standardized document image with a fixed aspect ratio;

[0042] S2: input the standardized document image into a target detection model trained by target language region enhancement, to detect and output only the bounding box coordinates of the text region belonging to the preset target language in the standardized document image, and filter out the text regions of non-pre-set target languages;

[0043] S3: based on the bounding box coordinates of the text region, an image block corresponding to the standardized certificate image is intercepted, and the image block is input into an optical character recognition model to output a character sequence and a corresponding character coordinate sequence, the character sequence containing the character recognition result in each image block, and the character coordinate sequence containing the normalized coordinates of each character in the standardized certificate image;

[0044] S4: inputting the character sequence and the corresponding character coordinate sequence into a sequence labeling model to obtain a structured information label corresponding to each character.

[0045] As shown in Figure 2 , step S4 specifically includes:

[0046] S41: mapping the character sequence into a text semantic feature vector through a first embedding network;

[0047] S42: mapping the character coordinate sequence into a spatial layout feature vector through a second embedding network;

[0048] S43: inputting the text semantic feature vector and the spatial layout feature vector into a trainable fusion network, modeling and fusing the interaction relationship between the two feature vectors through the fusion network to generate a fused multi-modal feature vector;

[0049] S44: inputting the multi-modal feature vector into a sequence labeling decoding layer to decode a structured information label corresponding to each character for identifying the certificate field type.

[0050] In the embodiment, the deep learning-based image segmentation correction model refers to a model with pixel-level semantic segmentation and geometric transformation capabilities, which can accurately separate the certificate main body region from the complex background and eliminate image distortion through perspective transformation and affine transformation to output a standardized certificate image, providing a clean and unified input basis for subsequent text processing.

[0051] The preset target language refers to a language of text to be extracted, which is preset according to the certificate information extraction requirement, such as Chinese characters in the identity card extraction scenario, for filtering non-target languages (such as minority languages and English) to interfere with subsequent recognition.

[0052] The optical character recognition model refers to a model with character recognition and position positioning functions, which can process the input text image block and simultaneously output the character recognition result (character sequence) and the position information of each character in the standardized certificate image (character coordinate sequence).

[0053] The sequence labeling model refers to a model for labeling the field type of a character sequence. The core is to judge the category of each character belonging to the certificate field (such as "surname", "name", "province", "city") by fusing the text semantic and spatial layout features, and output the structured information label.

[0054] The first embedding network is a network structure for mapping a character sequence into a text semantic feature vector (such as a word embedding layer), which can convert the semantic information of the character into a high-dimensional vector form that can be processed by the model, and capture the semantic association between characters.

[0055] The second embedding network is a network structure for mapping a character coordinate sequence into a spatial layout feature vector (such as a small multi-layer perceptron MLP), which can convert the physical position information of the character into a high-dimensional vector, reflecting the spatial distribution relationship of the character in the certificate layout.

[0056] The trainable fusion network refers to a network that has the ability to dynamically learn the interaction relationship between text semantic features and spatial layout features. By training, the fusion weights of the two features are automatically adjusted to generate a multi-modal feature vector containing both semantic and spatial information, providing comprehensive feature support for sequence labeling.

[0057] The structured information label is used to identify the type of the character belonging to the certificate field (such as "name-surname" and "address-province"), so that the recognition result conforms to the structured format of the certificate information, facilitating subsequent storage, analysis and application.

[0058] The purpose of step S1 is to separate the certificate main body from the original certificate image containing complex background, eliminate problems such as tilt and perspective deformation, and output a standardized certificate image with a fixed aspect ratio, providing a unified and interference-free input for subsequent text detection. In step S1, the image segmentation correction model based on deep learning processes the original image through two key sub-steps:

[0059] Pixel-level semantic segmentation: The image segmentation correction model performs pixel-by-pixel analysis on the original certificate image, using the feature extraction capability of a deep learning network (such as a convolutional neural network) to accurately identify the pixel differences between the certificate main body area and the background area, output a segmentation mask of the certificate main body area (in the mask, "1" corresponds to the certificate pixels, and "0" corresponds to the background pixels), and completely separate the certificate main body from the background.

[0060] Geometric transformation correction: Extract the bounding polygon of the certificate main body area from the segmentation mask, and obtain the vertex coordinates of the polygon. Based on the geometric relationship between the vertex coordinates and the standard certificate template, first correct the perspective deformation of the certificate caused by the shooting angle (such as the trapezoidal distortion of the certificate edge caused by oblique shooting) through perspective transformation, and then scale the certificate main body to a preset fixed size (such as matching the standard aspect ratio of an ID card) through affine transformation, and finally output the standardized certificate image.

[0061] Step S2 is used to accurately position the text area of the preset target language in the standardized certificate image, filter the non-target language text area, reduce invalid recognition tasks, and improve the accuracy and efficiency of subsequent character recognition. In step S2, the target detection model trained by the target language area enhancement recognizes and distinguishes the visual features (such as stroke structure, character spacing, and differences between minority scripts) of the target language and non-target language text through the feature learning ability of deep learning, and only outputs the bounding box coordinates of the target language text area, actively excluding the interference of non-target language areas.

[0062] The specific processing process of S2 is as follows: input the standardized certificate image output by S1 into the target detection model, the target detection model scans the standardized certificate image globally, and detects all text areas in the standardized certificate image; through the target language features learned in the training process, the text area belonging to the preset target language (such as Chinese characters) is selected, and the bounding box coordinates (such as the left upper corner x coordinate, the left upper corner y coordinate, the right lower corner x coordinate, and the right lower corner y coordinate) of each target language text area are output, while ignoring the text area of non-target language (such as minority scripts in ID card), ensuring that only valid text is processed subsequently.

[0063] Step S3 is used to perform character recognition on the target language text area detected by S2, and simultaneously obtain the recognition result and position information of each character in the standardized image, providing "semantic + spatial" two-dimensional data for subsequent multi-modal feature fusion.

[0064] The specific processing process of S3 is as follows: the optical character recognition model first extracts visual features (such as capturing stroke and contour features of characters through convolutional neural network) from the text image block cut from the standardized certificate image, and then processes the sequential relationship between characters through sequence modeling (such as recurrent neural network) to achieve high-precision character recognition; at the same time, the optical character recognition model also determines the bounding box coordinates of each character in the standardized certificate image through positioning algorithm, and forms a character coordinate sequence after normalization processing, ensuring that the coordinate information is not affected by the image size.

[0065] For example, according to the bounding box coordinates of the text region output by S2, the corresponding text image block (such as the image block containing "Name: Zhang San") is accurately cropped from the standardized certificate image; the image block is input into the optical character recognition model, and the optical character recognition model outputs two key results: one is the character sequence (such as ['surname', 'name', ':', 'Zhang', 'San']), which contains all the character recognition results in the image block; the second is the character coordinate sequence, which contains the normalized coordinates of each character in the standardized image (such as dividing the horizontal coordinate of the center point of the character bounding box by the width of the standardized image and the vertical coordinate by the height of the standardized image to obtain normalized coordinates with a value range of [0, 1]), ensuring the consistency of coordinates output by standardized images of different sizes.

[0066] Step S4 is used to fuse the text semantic features and spatial layout features of the characters, and output the structured information label corresponding to each character through a sequence labeling model, so as to realize structured extraction of certificate information and solve the problem that traditional single text semantic processing cannot cope with out-of-order text. The sequence labeling model realizes structured extraction through "feature mapping-feature fusion-sequence decoding" in three steps, and the specific process is as follows:

[0067] Feature mapping: map each character in the character sequence to a text semantic feature vector through a first embedding network to capture the semantic meaning of the character (such as the semantic difference between "surname" and "name"); map each coordinate in the character coordinate sequence to a spatial layout feature vector through a second embedding network to reflect the physical position relationship of the character (such as the spatial relationship of "surname" on the left side of "name").

[0068] Feature fusion: input the text semantic feature vector and the spatial layout feature vector into a trainable fusion network, and the fusion network learns the interaction relationship between the two features (such as the semantic features of the "name" field are usually associated with the spatial features of the upper region of the certificate) through training, dynamically adjusts the fusion weight, and generates a multi-modal feature vector that contains both semantic and spatial information, providing comprehensive feature support for sequence labeling.

[0069] Sequence decoding: input the multi-modal feature vector into the sequence labeling decoding layer, and the decoding layer performs field type labeling on each character according to the structured rules of the certificate field (such as the "name" field is composed of "surname" and "name", and the "address" field contains the province / city / district / street hierarchy), and outputs the structured information label (such as the label "name-surname" corresponding to "surname", and the label "name-name" corresponding to "Zhang").

[0070] Specifically, in the implementation of step S4, the character sequence output by S3 is first input into the sequence labeling model together with the character coordinate sequence. The sequence labeling model first generates a text semantic feature vector and a spatial layout feature vector through two embedding networks respectively; a fusion network fuses the two vectors to generate a multi-modal feature vector; and finally, a structured information label of each character is output through a sequence labeling decoding layer to form structured certificate information (such as name: Zhang San, address: XX province XX city...). Even if the character sequence output by S3 is in disordered order (such as “Zhang” being placed before “surname”), the fusion network can still determine the correct field attribution of the character through the spatial layout feature, ensuring the accuracy of the structured extraction result.

[0071] The above scheme retains the two-dimensional layout information of the certificate by fusing the spatial layout feature, avoids the information loss caused by the traditional technology of “flattening” the image into one-dimensional text, can effectively handle the text disorder scenario, and improves the robustness in complex scenarios. By detecting and filtering the non-target language region, the subsequent OCR model focuses on processing the target language, reduces invalid calculation and recognition errors, and improves the character recognition accuracy.

[0072] In some embodiments, the target detection model is trained in the following manner: a mixed sample set containing target language and non-target language text is constructed for training, and during the training process, the annotation box corresponding to all non-target language character regions in the mixed sample set is forcibly set as a background category, thereby achieving negative sample reinforcement learning.

[0073] In this embodiment, the mixed sample set contains a sample set of target language text and non-target language text in the certificate image, and the samples cover different certificate types, shooting environments, and language mixed arrangement scenarios (such as an ID card image containing Chinese characters and minority nationality characters), ensuring that the model training can access diversified real scene data.

[0074] Negative sample reinforcement learning performs special annotation and training constraints on non-target samples (here, non-target language character regions) during the training process, strengthens the recognition and filtering ability of the model for non-target samples, and enables the model to actively exclude non-target sample interference during inference. The learning method is to improve the detection specificity of the model for target samples through negative sample constraints.

[0075] The annotation box refers to a rectangular box used to frame the text region in the sample image, which contains the coordinate information of the text region and is the core supervision signal for training the target detection model. For non-target language character regions, the annotation box needs to be clearly marked as a background category, which is clearly distinguished from the target category of the target language text region, guiding the model to learn the difference between the two types of regions.

[0076] In determining the base model, the YOLO series target detection model (such as YOLOv11) can be selected as the base model, which has high feature extraction capability and real-time detection performance, and is suitable for rapid detection of certificate text regions. The backbone network of YOLOv11 (such as CSPDarknet) can effectively extract the detailed features of the text region, and the detection head can accurately locate the text region bounding box. In the loss function design, on the basis of the original loss function (bounding box regression loss, class classification loss, confidence loss) of YOLOv11, a non-target language region suppression loss term is added to strengthen the model's recognition ability of the background class (non-target language region): when the target detection model predicts the non-target language region bounding box as the background class, the loss term has a small value, and the model loss decreases; when the target detection model mistakenly predicts the non-target language region as the target language text class, the loss term value increases significantly, and through loss penalty, the model is forced to correct the prediction deviation and learn to distinguish the features of target and non-target language regions.

[0077] Target detection model iteration training process:

[0078] Initial training stage (1-50 rounds): set a low learning rate (such as 1×10 -4 ), train with YOLOv11 pre-training weights as initial weights, so that the target detection model can preliminarily learn the features and bounding box regression rules of the target language text. At this time, the non-target language region suppression loss term weight can be set to 0.3 to avoid excessive punishment in the initial stage, which may cause difficulty in model convergence.

[0079] Negative sample strengthening stage (51-150 rounds): gradually increase the learning rate to 3×10 -4 , and increase the non-target language region suppression loss term weight to 0.7 to force the model to pay attention to the feature differences of the non-target language region (such as the stroke structure, character arrangement, and difference from Chinese characters of minority ethnic characters), and learn to actively filter the non-target language region.

[0080] Fine-tuning optimization stage (151-200 rounds): reduce the learning rate to 1×10 -5 , adjust the non-target language region suppression loss term weight to 0.5, fine-tune the target detection model parameters, and use the validation set to evaluate the model performance (target language region detection accuracy, non-target language region filtering rate) in real time. According to the validation results, adjust the hyperparameters (such as learning rate, loss term weight) to ensure that the target language detection accuracy of the model on the validation set is ≥95%, and the non-target language filtering rate is ≥98%.

[0081] The above scheme labels a non-target language region as a background category and designs a targeted loss function, so that the model can not only identify the target language region, but also actively suppress the detection output of the non-target language region, avoid misjudgment of the non-target language region as the target region, and significantly improve the detection accuracy. For the common mixed language arrangement in certificates (such as the minority language on the back of an ID card and English in foreign-related certificates), the target detection model has clear language distinguishing ability through negative sample reinforcement learning, and can be stably applied to multi-ethnic areas, foreign-related business and other scenes, and has stronger generalization ability.

[0082] In some embodiments, the fusion network is a gated fusion network; the fusion network models and fuses the interaction relationship between the two feature vectors to generate a fused multi-modal feature vector, specifically including: for each current character in the character sequence, the gated fusion network dynamically generates a fusion weight vector based on the context semantics of the current character and its corresponding graph attention encoding, and uses the fusion weight vector to weight and fuse the text semantic feature vector and the spatial layout feature vector of the current character to generate the multi-modal feature vector of the character;

[0083] As shown in Figure 3 The graph attention encoding is obtained by the following way:

[0084] S441: taking each recognized character as a graph node and taking the normalized coordinate value of the character as the initial node feature of the graph node, a undirected graph is constructed; wherein, if the Euclidean distance between two character nodes in the normalized coordinate space is less than a preset distance threshold, an edge is established between the two nodes;

[0085] S442: input the constructed undirected graph and the initial node features of all nodes of the undirected graph into a graph attention network; the graph attention network iteratively processes the undirected graph through multiple layers of graph attention layers; in each layer of graph attention layers, each node updates its node feature by calculating the attention weight with all neighbor nodes and weighting and summing the features of neighbor nodes;

[0086] S443: obtain the graph attention encoding of the corresponding character through the node feature vectors output by the last layer of graph attention layers of the graph attention network.

[0087] In this embodiment, the attention encoding is a feature vector obtained by encoding the spatial and semantic association relationship of the character nodes based on the graph attention network (GAT), which can represent the interaction information of the characters and adjacent characters (such as the adjacent relationship of “surname” and “name”, and the association relationship of “province” and “city”), and provide context dependency for the gated fusion network, helping the network to more accurately determine the fusion weight.

[0088] In step S441, when constructing the character undirected graph, an undirected graph is constructed with characters as nodes and spatial proximity relationships as edges, intuitively presenting the spatial distribution and association of characters in the certificate page, and specifically comprising:

[0089] Node definition: each character in the character sequence output in step S3 is taken as a node of the undirected graph, and the initial feature of each node is set as the normalized coordinate value of the character (for example, the initial feature of the node "surname" is the normalized center point coordinate surname surname, ensuring that the initial feature can directly reflect the spatial position information of the character.

[0090] Edge construction rule: the Euclidean distance between each two character nodes in the normalized coordinate space is calculated, and if the distance is less than a preset distance threshold (for example, 0.1, which is set according to the size of the certificate and the distance between characters, ensuring that adjacent characters in the same field can be determined as adjacent), it is considered that the two characters are adjacent in space, and an undirected edge is established between the corresponding two nodes; if the distance is greater than or equal to the preset distance threshold, no edge is established to avoid incorrect association of characters with a long spatial distance (such as characters in the "name" field and the "address" field).

[0091] Specific operation: assuming that the character sequence output in S3 is ['surname', 'name', ':', 'Zhang', 'three','sex', 'gender', ':','male'], and the corresponding normalized coordinates are (0.1, 0.2), (0.15, 0.2), (0.2, 0.2), (0.25, 0.2), (0.3, 0.2), (0.1, 0.3), (0.15, 0.3), (0.2, 0.3), (0.25, 0.3); the Euclidean distance of any two characters is calculated, such as the distance between "surname" and "name" is 0.05 (less than 0.1), an edge is established; the distance between "three" and "sex" is 0.141 (greater than 0.1), no edge is established; in the finally constructed undirected graph, "surname-name-:-Zhang-three" forms a connected subgraph, and "sex-gender-:-male" forms another connected subgraph, accurately reflecting the spatial distribution of the two fields.

[0092] In S442, through the multi-layer iterative processing of the graph attention network, the initial features (spatial coordinates) of the character nodes and the features of the neighbor nodes are fused to generate node features containing spatial association information, specifically comprising:

[0093] The features of all nodes of the current layer are linearly transformed, and the attention coefficients of each node and all neighbor nodes (determined by the edges of the undirected graph) are calculated to measure the importance of the nodes to each other. The attention coefficients of the nodes and all neighbor nodes are normalized by a Softmax function to obtain normalized attention weights, ensuring that the sum of the weights is 1. According to the normalized attention weights, the transformed features of all neighbor nodes of the node are weighted and summed to obtain the updated features of the node, which integrates the spatial correlation information of the neighbor nodes. For example, the updated features of the node "name" include the spatial features of "surname" and ":".

[0094] In a specific implementation, a graph attention network including three layers of graph attention layers can be constructed, and the feature dimension of each layer is set to 64. The undirected graph constructed in step S441 and the initial node features are input into the network, and after being processed by the first layer of graph attention layers, the node features are changed from 2-dimensional coordinates to 64-dimensional features, and include information of direct neighbors. When processed by the second layer of graph attention layers, the node features further integrate the information of indirect neighbors (for example, the features of "surname" include the information of "Zhang"). After being processed by the third layer of graph attention layers, the obtained node features can comprehensively reflect the spatial correlation of characters in the entire undirected graph.

[0095] In step S443, after the graph attention network is processed by multiple layers of iterations, the node features output by the last layer have sufficiently integrated the spatial features of the characters themselves and the correlation information of the neighbor nodes, and can comprehensively represent the spatial context relationship of the characters in the certificate page. The feature vector is directly used as the graph attention encoding. The node feature vector output by the last layer of the graph attention network, such as the graph attention encoding of the node "Zhang", is a 64-dimensional vector that includes not only the coordinate information of "Zhang" itself, but also the spatial correlation information of "surname", "name", ":", and "three". The graph attention encodings of all characters are arranged in the order of the character sequence to form a graph attention encoding sequence, which corresponds to the character sequence, the text semantic feature vector sequence, and the spatial layout feature vector sequence one by one.

[0096] Then, based on the context semantics and graph attention encoding of the characters, a fusion weight is dynamically generated to adaptively fuse the text semantic feature vector and the spatial layout feature vector, and a multi-modal feature vector is generated.

[0097] In acquiring context semantics, the text semantic feature vector sequence generated in step S4 is processed by a bidirectional long short-term memory network (BiLSTM). The forward LSTM of the BiLSTM captures the forward context semantics of the character (for example, the forward context of "Zhang" includes "San"), and the backward LSTM captures the reverse context semantics of the character (for example, the reverse context of "Zhang" includes "surname", "name", and ":"). The forward and backward outputs are spliced to obtain the context semantic feature vector of each character, which has the same dimension as the text semantic feature vector (for example, 256 dimensions).

[0098] In generating the fusion weight, the context semantic feature vector of each character is spliced with the graph attention encoding (with a dimension of 256+64=320), and a gating layer (composed of a fully connected layer and a Sigmoid activation function) is input. The gating layer outputs a fusion weight vector (w), where w∈[0,1], and the vector dimension is consistent with the text semantic feature vector (for example, 256 dimensions). If the context semantics and the graph attention encoding indicate that the semantic information is more important in the current scenario (for example, the character semantic is clear and the spatial position has a slight deviation), the weight of the corresponding semantic feature in w is increased; if the spatial information is more reliable (for example, the semantic is ambiguous but the spatial position conforms to the layout of the certificate field), the weight of the corresponding spatial feature in w is increased.

[0099] Then, the text semantic feature vector and the spatial layout feature vector are weighted and fused using the fusion weight vector (w) to generate a multi-modal feature vector (C). For example, when the value of a certain dimension in w is 0.7, it indicates that the contribution of the semantic feature in that dimension accounts for 70%, and the contribution of the spatial feature accounts for 30%, realizing adaptive fusion of the two types of features.

[0100] The above scheme constructs a character undirected graph and uses a graph attention network to process, so that the graph attention encoding contains the spatial context information of the characters, solving the problem that traditional feature fusion only focuses on the features of a single character itself and ignores the correlation between characters, and improving the model's understanding of the layout of the certificate, such as accurately identifying the adjacent relationship of characters in the name field. The gating fusion network dynamically generates a fusion weight based on context semantics and spatial correlation, avoiding the defect that fixed weight fusion cannot adapt to different scenarios (such as focusing on semantic features when the semantics are clear, and focusing on spatial features when the spatial information is reliable), significantly improving the expression ability of multi-modal features, and enabling the model to flexibly cope with certificate images of different qualities. The graph attention encoding reflects the spatial proximity of characters. Even if the text sequence is out of order due to OCR recognition (for example, "Zhang" is placed before "surname"), the model can still determine the correct attribution of the character (for example, "Zhang" is spatially adjacent to "surname" and "name", and is determined as the "name" field) through spatial correlation, further improving the robustness of the system to complex scenarios.

[0101] In some embodiments, in step S442, the undirected graph is iteratively processed, and in each layer of the graph attention layer, each node updates its node feature by calculating the attention weight with all its neighbor nodes and weighting and summing the features of the neighbor nodes.

[0102] As shown in Figure 4 , a field-aware adjacency matrix is constructed to aggregate information in the graph attention layer; the construction of the field-aware adjacency matrix includes:

[0103] S4421: Based on the character coordinate sequence, the Euclidean distance between each two character nodes in the normalized coordinate space is calculated, and a spatial adjacency matrix is constructed based on a first distance threshold;

[0104] S4422: Based on the text semantic feature vector obtained by the first embedding network based on the character sequence, the correlation degree between each two character node features is calculated, and a semantic correlation matrix is constructed based on a correlation degree threshold; the correlation degree is calculated using cosine similarity;

[0105] S4423: The spatial adjacency matrix and the semantic correlation matrix are fused to generate a field-aware adjacency matrix; for each target node in the undirected graph, the graph attention layer calculates the attention coefficient of the target node and all neighbor nodes defined by the field-aware adjacency matrix and normalizes it to obtain a normalized attention weight, and uses the normalized attention weight to weight and sum the features of the neighbor nodes to obtain the updated node feature corresponding to the target node.

[0106] In this embodiment, the field-aware adjacency matrix refers to a matrix that fuses the spatial adjacency relationship between characters and the semantic correlation relationship, which is used to accurately define the neighbor range and correlation strength of nodes in the graph attention layer, so that the information aggregation process not only conforms to the spatial layout rules of the certificate page (such as the spatial proximity of characters in the same field), but also follows the semantic logic of the text (such as the semantic correlation between “surname” and “name”), avoiding cross-field information interference.

[0107] The spatial adjacency matrix refers to a binary matrix constructed based on the Euclidean distance between characters, which is used to represent the spatial proximity relationship between characters, and the matrix element value is 1 indicating that the corresponding two characters are spatially adjacent, and 0 indicating that they are not adjacent, which intuitively presents the spatial distribution correlation of characters.

[0108] The semantic correlation matrix refers to a matrix constructed based on the cosine similarity of character semantic feature vectors, which is used to represent the semantic correlation degree between characters, and the larger the matrix element value is, the stronger the semantic correlation between the corresponding two characters is, quantifying the semantic correlation relationship between characters.

[0109] Suppose the character sequence of the certificate to be processed is [‘surname’, ‘name’, ‘:’, ‘Zhang’, ‘three’, ‘sex’, ‘:’, ‘male’], a total of 9 characters (N=9), each of which has a corresponding normalized coordinate in the standardized certificate image, and the respective text semantic feature vector has been obtained through the embedding network.

[0110] When constructing the spatial adjacency matrix, first define the matrix as a 9x9 matrix, and the elements in the matrix correspond to the spatial adjacency relationship of two characters. Set the first distance threshold to 0.08, and determine whether the characters are adjacent by calculating the Euclidean distance of the normalized coordinates: for example, the distance between “surname” and “name”, “name” and “:” is less than 0.08, and the corresponding matrix element is set to 1; “three” and “sex” belong to different fields, the distance is 0.141, which is greater than 0.08, and the element is set to 0; at the same time, the diagonal elements are all set to 1, ensuring that the character itself is included in the subsequent processing, and finally forming a clear spatial adjacency matrix A spatial .

[0111] When constructing the semantic association matrix, the matrix dimension is consistent with the spatial adjacency matrix, and the semantic association degree is quantified by cosine similarity. Set the association threshold to 0.6, and the semantic similarity between “name” and “Zhang”, “Zhang” and “three” in the same field is higher than 0.6, and the corresponding matrix element retains the specific similarity value (such as 0.65, 0.7); the similarity between “name” and “other”, “Zhang” and “sex” is lower than 0.6, and the element is set to 0, forming a semantic association strength quantization matrix A semantic .

[0112] When generating the field-aware adjacency matrix, the element multiplication method is used to integrate A spatial and A semantic . Only when the spatial adjacency element at a certain position is 1 and the semantic association element is greater than 0, the fused element will retain the semantic similarity value, such as A field [1][2]=1×0.7=0.7; if either condition is not met, the fused element is 0, such as A field [5][6]=0×0.1=0. The final generated A field matrix accurately selects the character pairs that are “spatially adjacent and semantically associated”, clearly defines the internal association of fields such as “name” and “sex”, and lays the foundation for subsequent graph attention layers to only fuse information within the same field and exclude cross-field interference.

[0113] The above scheme constructs a field-aware adjacency matrix by fusing spatial adjacency and semantic association, excludes cross-field interference, makes the graph attention layer only focus on the character association within the same field, solves the cross-field information pollution problem caused by the traditional definition of neighbors based on spatial distance (such as the fusion of “name” field characters with “gender” field character information), and improves the recognition ability of the model to the boundaries of the certificate field. Through attention coefficient combination and semantic association strength calculation, characters with stronger semantic association within the same field (such as “north” and “jing”) obtain higher attention weight, feature aggregation is more accurate, and the representation ability of multi-modal features to field attributes is further improved, such as the features of “address” field characters can more accurately reflect the address hierarchy.

[0114] The node feature update only incorporates the same field information, so that the subsequent sequence labeling decoding layer can more clearly identify the field type to which the character belongs, reduce the interference of cross-field characters on the labeling result (such as avoiding mislabeling “city” in the “address” field as part of the “name” field), and significantly improve the accuracy of structured information extraction.

[0115] In some embodiments, the deep learning-based image segmentation correction model is a task model that shares an encoder and has two parallel decoding heads, including a first decoding head and a second decoding head, which respectively perform the following two tasks:

[0116] The first decoding head performs pixel-level semantic segmentation on the certificate main body region, and outputs a certificate main body region segmentation mask;

[0117] The second decoding head parallelly regresses a heat map of one or more preset key points in the original certificate image, the preset key points corresponding to corner points or field center points with fixed layout features on the original certificate image;

[0118] The perspective transformation and the affine transformation calculate the transformation parameters according to the following steps:

[0119] S11: Extract the circumscribed polygon of the certificate main body region from the certificate main body region segmentation mask to obtain a first set of candidate corner point coordinate sets ;

[0120] S12: Extract the coordinates of each key point from the key point heat map by a peak finding algorithm to obtain a second set of candidate corner point coordinate sets ;

[0121] S13: According to the pre-defined geometric relationship of the certificate template, match and fuse the candidate points in and to form a final target corner point set ;

[0122] S14: Based on the final target corner point set The transformation matrix for the perspective transformation and the affine transformation is calculated by solving a least square problem with the corner point coordinates of the standard certificate template .

[0123] In this embodiment, the image segmentation correction model adopts a "shared encoder + parallel decoding head" architecture, with ResNet50 as the shared encoder. The original certificate image of 640x640 pixels is input, and multi-scale features of 160x160, 80x80, 40x40, etc. are extracted through 5 convolution stages. The first decoding head uses a "transpose convolution + convolution" structure to restore the feature map resolution to 640x640, and outputs a binary segmentation mask through a 1x1 convolution, realizing pixel-level segmentation of the certificate main body and background. The second decoding head receives the 40x40 high-level features, and outputs a 640x640 pixel heat map of 4 certificate corner points through 3 convolution layers, global average pooling, and a fully connected layer. The model is trained using a multi-task joint loss function, and outputs the segmentation mask and corner point heat map after inputting the original image.

[0124] Then the segmentation mask is subjected to contour detection, and the vertices of the circumscribed polygon are extracted to form P seg The key point coordinates of the heat map are extracted using a peak searching algorithm to form P heat The two sets of corner points are matched according to the geometric relationship of the standard template, and a weighted average method (a=0.5) is used to fuse them to generate P final The perspective and affine transformation matrix is obtained by least square solution with the standard template corner points as the target, the deformation is corrected by applying the matrix, and the standardized certificate image is output by scaling to the standard size.

[0125] The above scheme obtains segmentation corner points and key point corner points in parallel through a multi-task model, and has the advantages of global contour and local precision after fusion, greatly improving the corner point positioning accuracy and avoiding the positioning deviation of a single method. The least square solution ensures the optimality of the transformation matrix, effectively resists noise interference, and enhances the correction robustness in complex scenes such as uneven illumination and partial occlusion. The shared encoder design reuses features, reduces repeated calculations, improves inference speed, and the features between tasks are complementary, making the overall performance of the model better than that of a single task design, providing high-quality input for subsequent text detection and recognition.

[0126] In some embodiments, the fusion network is a candidate field-aware fusion network, and the text semantic feature vector and the spatial layout feature vector are input into the fusion network. The fusion network models and fuses the interaction relationship between the two types of feature vectors to generate a fused multi-modal feature vector, which includes:

[0127] Based on the text semantic feature vector, a field candidate generator generates a candidate list containing k most possible certificate field types , wherein, denotes the k-th candidate field type, denotes a predicted probability based on text semantics;

[0128] For a predefined field layout knowledge base, according to each candidate field type T in the candidate list L k , query a corresponding standard space layout descriptor , the standard space layout descriptor is a normalized spatial feature extracted from the standard certificate template for characterizing the field type , the standard space layout descriptor dimension is consistent with the spatial layout feature vector;

[0129] The fusion network receives the standard space layout descriptors of all candidate field types obtained by querying the spatial layout feature vector and their corresponding predicted probabilities ;

[0130] For each candidate field type , by calculating the similarity of the spatial layout feature vector and the standard space layout descriptor in the feature space, a layout matching degree score is obtained;

[0131] Based on the layout matching degree scores of all candidate field types and their predicted probabilities , a trainable decision network outputs a modality selection preference signal , wherein , is used to represent the contribution proportion of the spatial layout feature relative to the text semantic feature in the final decision in the current context;

[0132] According to the modality selection preference signal , the text semantic feature vector and the spatial layout feature vector are fused according to the following formula to generate the multi-modal feature vector :

[0133]

[0134] In this embodiment, the field layout knowledge base refers to a database that stores the standardized spatial features corresponding to each field type in the standard document template. Each field type is associated with a standard spatial layout descriptor, which is a spatial feature vector (such as the average position of the field, character spacing, line height ratio, etc.) statistically extracted from a large number of standard document samples, and its dimension is consistent with the spatial layout feature vector.

[0135] A standard spatial layout descriptor is a high-dimensional vector that represents the spatial attributes of a specific field type. For example, the descriptor of the "name" field contains information such as its normalized average y-coordinate (e.g., 0.2) and the normalized value of the character spacing (e.g., 0.05) in a standard document, which is used to perform similarity matching with the spatial layout feature vector of the current character.

[0136] The field candidate generator consists of two fully connected layers and a Softmax activation function. The input is the global average pooling result of the text semantic feature vector sequence generated in step S4 (converting the sequence features into a single vector with the same dimension as the text semantic feature vector, such as 256 dimensions). The output is the probability distribution of all preset document field types (e.g., "name", "gender", "address", "ID number", etc., a total of 8 fields). The output probability distributions are then sorted in descending order of probability value, and the top K (K=3, determined experimentally to balance candidate coverage and computational efficiency) field types with the highest probabilities are selected to form the candidate list L.

[0137] Specifically, assuming the current character sequence is ['surname', 'given name', ':', 'Zhang', 'San'], the global average pooling result of its text semantic feature vector is input into the field candidate generator, and the output field probability P1=0.92 for "name", field probability P2=0.05 for "alias", and field probability P3=0.02 for "former name" is generated. Then the candidate list L={('name', 0.92), ('alias', 0.05), ('former name', 0.02)} accurately identifies the high-probability field types.

[0138] When building the field layout knowledge base, a large number of standard document template images (such as 1000 clear standard ID card photos) are collected. For each preset field type (such as "name"), its spatial features in the template are extracted. The extracted content includes the following information:

[0139] The normalized average position of the field (the average x and y coordinates of the center points of all characters in the field).

[0140] Normalized average of character spacing (average Euclidean distance between the centers of adjacent characters within the field).

[0141] Normalized width and height of the field (width and height of the field bounding box divided by the image size).

[0142] The space features are combined into a high-dimensional vector (the dimension is consistent with the space layout feature vector, such as 256 dimensions) as a standard space layout descriptor of the field type, and are stored in the field layout knowledge base.

[0143] Then, according to each candidate field type in the candidate list, an exact match query is performed in the field layout knowledge base to obtain the corresponding standard space layout descriptor; if a candidate field type has no corresponding descriptor (such as a rare field), the candidate field is discarded.

[0144] Then, the matching degree of the current space layout feature and the standard descriptor of each candidate field is calculated, the modal selection preference signal a is dynamically generated by combining the prediction probability, and the fusion proportion of the two features is determined. Then, according to the modal selection preference signal a, a multi-modal feature vector is generated according to a linear fusion formula, and the intelligent complementation of the two features is realized.

[0145] In the embodiment, the adjustment rule of a value in different scenarios is learned by the decision network through training as follows:

[0146] If the P k and S k of a certain candidate field are both high (such as P1=0.92, S1=0.95), it indicates that the text semantics and the space layout are highly consistent, and the a value is taken as a middle value (such as 0.5), and the two features are balanced.

[0147] If the P k is high but the S k is low (such as the semantics is clear but the space position is offset), the a value is reduced (such as 0.3), and the text semantic feature is emphasized.

[0148] If the P k is low but the S k is high (such as the semantics is ambiguous but the space layout conforms to the standard), the a value is increased (such as 0.7), and the space layout feature is emphasized.

[0149] The above scheme dynamically adjusts the a value based on the candidate prediction and layout matching degree of the field type, so that the fusion proportion adapts to different scenarios, avoids the limitation of fixed weight fusion, and improves the pertinence of multi-modal features. The standard descriptor provided by the field layout knowledge base provides a clear benchmark for space feature matching, helping the model to more accurately determine the field type to which the character belongs, especially in the scene where the semantics is ambiguous or the character is incomplete (such as the "surname" character is missing, but the "name" field can still be determined through space layout matching). By narrowing down the matching range through candidate field screening, the interference of irrelevant fields is reduced, and the intelligent complementation of the two features makes the model more accurate in judging the field boundary and the field type, such as effectively distinguishing between "name" and "alias" fields with similar semantics but different space layouts.

[0150] In some embodiments, such as Figure 5As shown, the perspective transformation and affine transformation based on the vertex coordinates of the circumscribed polygon of the segmented document body region specifically include:

[0151] S501: Detect the smallest bounding rectangle of the polygon surrounding the main body area of ​​the document;

[0152] S502: Calculate the perspective transformation matrix based on the correspondence between the four vertices of the minimum bounding rectangle and the four corner points of the standardized document image;

[0153] S503: Apply the perspective transformation matrix to transform the original image of the document to correct the perspective distortion of the main body area of ​​the document;

[0154] S504: Perform an affine transformation on the main body area of ​​the document after perspective transformation, scale the main body area of ​​the document to a preset fixed size, and output the standardized document image with a fixed aspect ratio.

[0155] The above scheme effectively eliminates the "nearer is larger, farther is smaller" distortion of the document caused by the shooting angle by combining the minimum bounding rectangle and the perspective transformation matrix. It solves the problem of poor handling of perspective distortion by traditional correction methods (such as Hough transform), restoring the main body of the document to a regular rectangle. The affine transformation is strictly scaled according to a preset fixed size to ensure that the aspect ratio of the standardized image output from different original images (such as ID photos with different shooting distances and resolutions) is consistent, avoiding the offset of the coordinates of the subsequent text detection box due to size differences and improving the processing stability of subsequent modules. First, the irregular bounding polygon is transformed into a regular rectangle by using the minimum bounding rectangle, and then the perspective transformation is performed, which greatly simplifies the solution of the transformation matrix, improves the transformation efficiency, and ensures the real-time performance of model inference.

[0156] In some embodiments, such as Figure 6 As shown, inputting the image block into the optical character recognition model and outputting the character sequence and corresponding character coordinate sequence specifically includes:

[0157] S601: The image block is input into an optical character recognition model based on a convolutional recurrent neural network structure. The optical character recognition model extracts visual features from the image block through its internal convolutional neural network and performs sequence modeling on the extracted features through its internal recurrent neural network. It simultaneously outputs the recognition result of the character sequence contained in the image block and the bounding box coordinates of each character in the character sequence in the standardized document image coordinate system.

[0158] S602: Process the obtained boundary box coordinates of each character, calculate the center point coordinates of each character boundary box; divide the horizontal coordinates of the center point coordinates by the width of the standardized certificate image, and divide the vertical coordinates by the height of the standardized certificate image to obtain a normalized character coordinate sequence.

[0159] For example, a text image block (32x128 pixels) containing "Name: Zhang San" is input into the CNN network, and after 4 layers of convolution and 2 layers of pooling processing, an 8x32x128 feature map is output, which clearly represents the visual features of each character such as "surname", "name", ":", "Zhang", and "San" (such as the left and right structure of "Zhang" and the three horizontal structure of "San"). Then

[0160] The 8x32x128 feature map is serialized into 32 1024-dimensional feature vectors, which are input into a 2-layer BiLSTM network (512 hidden units per layer) to output 32 1024-dimensional sequence features; a fully connected layer and a Softmax output the character probability distribution at each position, and finally the character sequence ['surname', 'name', ':', 'Zhang', 'San'] is recognized.

[0161] Before CNN feature extraction and RNN sequence modeling, a coordinate prediction branch (composed of 2 layers of convolution layers and 1 layer of fully connected layers) is added, which receives the 8x32x128 feature map output by CNN, extracts position features through convolution operation, and then outputs the character boundary box coordinates (x1, y1, x2, y2) corresponding to each sequence position through the fully connected layer. The coordinate values are pixel values in the standardized certificate image coordinate system.

[0162] In the above scheme, the convolutional recurrent neural network structure integrates character recognition and coordinate positioning in one network, improving the consistency and accuracy of character recognition and coordinate positioning. The high-level visual features extracted by CNN and the sequence dependency captured by RNN are combined, effectively solving the confusion problem of similar characters (such as "Ren" and "Ru", "1" and "7"), especially suitable for scenarios with uniform font and dense characters in certificates. Through boundary box coordinate prediction and normalization processing, the obtained character coordinate sequence is not affected by image size, providing standardized position information for subsequent spatial layout feature extraction and multi-modal fusion, ensuring consistent spatial feature representation for standardized images of different sizes.

[0163] In some embodiments, as shown in FIG. 6, after the step S3, a boundary box iterative optimization step is further included: Figure 7

[0164] S701: Calculate the overall or local recognition confidence score of the character sequence output in step S3;

[0165] ​S702: If the recognition confidence score is lower than a preset score threshold, an extended image region containing the context outside the bounding box of the text region is cropped from the standardized certificate image based on the bounding box coordinates of the text region, and the extended image region and the recognized low-confidence character sequence are spliced together as optimization input information;

[0166] S703: The optimization input information is input into a pre-trained bounding box optimization network, which is mainly composed of a lightweight convolutional neural network, which directly outputs the fine-tuning amount of the bounding box coordinates of the original text region through regression;

[0167] S704: The fine-tuning amount is added to the bounding box coordinates of the original text region to generate optimized bounding box coordinates;

[0168] S705: Based on the optimized bounding box coordinates, the image block is re-cropped from the standardized certificate image, and the optical character recognition model is re-inputted to perform step S3 to obtain an updated character sequence and character coordinate sequence.

[0169] In this embodiment, the recognition confidence score refers to a quantitative indicator output by the optical character recognition model to represent the reliability of the character recognition result, with a value range of [0, 1]. The closer the score is to 1, the more reliable the recognition result is, and the closer the score is to 0, the more likely the recognition result is incorrect (e.g. recognition deviation caused by character blur or occlusion).

[0170] The overall recognition confidence refers to the average value of all character recognition confidence scores in the character sequence, which is used to evaluate the recognition reliability of the entire text image block. If the overall score is lower than a preset score threshold (e.g. 0.8), it indicates that the recognition result of the image block has a high risk of error.

[0171] The local recognition confidence refers to the average value of the recognition confidence scores of a plurality of consecutive characters (e.g. 3) in the character sequence, which is used to locate the local area with unreliable recognition result (e.g. low confidence due to occlusion of individual characters in a certain field).

[0172] The extended image region refers to an image region obtained by extending a preset pixel range (e.g. 5 pixels up, down, left and right) outside the original text region bounding box, which is used to supplement the character edge information that may be missed by the original bounding box (e.g. character strokes truncated by the bounding box), and improve the accuracy of re-recognition.

[0173] The bounding box optimization network structure uses a lightweight CNN (such as MobileNetV2) as the main network. The input is the concatenated features of the extended image region and the low-confidence character sequence (after the extended image region is feature-extracted by the CNN and concatenated with the word embedding features of the character sequence), and the output is the fine-tuning amount (Δx, Δy, Δw, Δh) of the original bounding box. Among them, Δx and Δy adjust the position of the bounding box, and Δw and Δh adjust the size of the bounding box.

[0174] Suppose the character sequence is ['姓', '名', ':', '张', '三'], and the corresponding confidence scores are [0.95, 0.93, 0.88, 0.62, 0.90]. The overall confidence is calculated as 0.856 by taking the average value. However, the average confidence of the local window k = 3 (characters [':', '张', '三']) is 0.8, so it is determined that the recognition of this local area is unreliable and the corresponding bounding box needs to be optimized. Then, an extended image region of 155×95 to 245×145 can be intercepted from the standardized document image. This extended image region completely contains all the strokes of the three characters ":", "张", and "三"; the extended image region is concatenated with the low-confidence character sequence [':', '张', '三'] to form the optimized input information. Then, the optimized input information is input into the bounding box optimization network, and the output fine-tuning amounts are Δx = 2, Δy = 1, Δw = 3, Δh = 2; the original bounding box is optimized to obtain; based on the re-intercepted image patch, it is input into the OCR model for recognition, and the updated character sequence [':', '张', '三'] is obtained, and the corresponding confidence scores are [0.88, 0.94, 0.91]. The overall confidence 0.91 > 0.8 (the preset score threshold), which meets the requirements, and the iteration stops.

[0175] The above solution supplements the character edge information through the extended image region, combines the bounding box optimization network to adjust the position and size of the bounding box, solves the problems of incomplete character recognition and low confidence caused by inaccurate original bounding boxes (such as recognition errors caused by truncated character strokes), and improves the accuracy of the character sequence. For complex scenarios such as image blur, character occlusion, and bounding box offset, the recognition effect is gradually improved through the iterative optimization mechanism, avoiding subsequent module processing failures caused by single recognition errors, and enhancing the adaptability of the system to low-quality document images. Only the low-confidence recognition results are optimized, and there is no need to perform an iterative process on all character sequences. While improving the recognition accuracy, the computational overhead is controlled to ensure that the overall processing efficiency of the system is not significantly affected.

[0176] In a second aspect, the present application provides a document information extraction system that integrates spatial layout and text semantics. The document information extraction system is used to execute the method as described in the first aspect of the present application, as Figure 8 shown, the document information extraction system 10 includes:

[0177] The image standardization module 101 is configured to input an inputted original document image into a deep learning-based image segmentation correction model, perform pixel-level semantic segmentation on a document body region, perform perspective transformation and affine transformation based on vertex coordinates of an inscribed polygon of the segmented document body region, and output a standardized document image with a fixed aspect ratio.

[0178] The target text detection module 102 is configured to input the standardized document image into a target detection model trained based on a target language region, to detect and output only a bounding box coordinate of a text region belonging to a preset target language in the standardized document image, and filter out a character region of a non-pre-set target language.

[0179] The character recognition and positioning module 103 is configured to cut an image block corresponding to the text region from the standardized document image based on the bounding box coordinate of the text region, and input the image block into an optical character recognition model to output a character sequence and a corresponding character coordinate sequence, wherein the character sequence includes a character recognition result in each image block, and the character coordinate sequence includes a normalized coordinate of each character in the standardized document image.

[0180] The multi-modal information structuring module 104 is configured to input the character sequence and the corresponding character coordinate sequence into a sequence labeling model to obtain a structured information label corresponding to each character. The multi-modal information structuring module includes:

[0181] The text semantic embedding unit 1041 is configured to map the character sequence into a text semantic feature vector through a first embedding network.

[0182] The spatial layout embedding unit 1042 is configured to map the character coordinate sequence into a spatial layout feature vector through a second embedding network.

[0183] The feature fusion unit 1043 is configured to input the text semantic feature vector and the spatial layout feature vector into a trainable fusion network, model and fuse an interaction relationship between the two feature vectors through the fusion network, and generate a fused multi-modal feature vector.

[0184] The sequence labeling unit 1044 is configured to input the multi-modal feature vector into a sequence labeling decoding layer to decode a structured information label corresponding to each character for identifying a document field type.

[0185] This application discloses a method and system for extracting document information by integrating spatial layout and textual semantics. The method includes: inputting the original document image into an image segmentation and correction model for segmentation and correction, and outputting a standardized document image; inputting the standardized image into a target detection model to detect and output the bounding box coordinates of a text region in a preset target language; cropping image blocks based on the bounding box coordinates, inputting them into an optical character recognition model, and outputting a character sequence and its corresponding character coordinate sequence; inputting the character sequence and character coordinate sequence into a sequence labeling model, mapping them to textual semantic feature vectors through a first embedding network, and mapping them to spatial layout feature vectors through a second embedding network; inputting the two feature vectors into a fusion network for modeling and fusion to generate a multimodal feature vector; and finally decoding the structured information label corresponding to each character through a sequence labeling decoding layer. This invention effectively improves the accuracy and robustness of document information extraction.

[0186] Finally, it should be noted that although the above embodiments have been described in the text and drawings of this application, this should not limit the scope of patent protection of this application. Any technical solutions that are based on the essential concept of this application and utilize the content described in the text and drawings of this application, resulting in equivalent structural or procedural substitutions or modifications, as well as the direct or indirect application of the technical solutions of the above embodiments to other related technical fields, are all included within the scope of patent protection of this application.

Claims

1. A method for extracting document information that integrates spatial layout and textual semantics, characterized in that, The method includes: S1: Input the original image of the document into the image segmentation and correction model based on deep learning, perform pixel-level semantic segmentation on the main body region of the document, and perform perspective transformation and affine transformation based on the vertex coordinates of the bounding polygon of the segmented main body region of the document, and output a standardized document image with a fixed aspect ratio. S2: Input the standardized document image into the target detection model trained with target language region enhancement, so as to detect and output only the bounding box coordinates of the text regions belonging to the preset target language in the standardized document image, and filter out the text regions of non-preset target language. S3: Based on the bounding box coordinates of the text region, extract the corresponding image block from the standardized document image, and input the image block into the optical character recognition model to output a character sequence and a corresponding character coordinate sequence. The character sequence contains the character recognition result in each image block, and the character coordinate sequence contains the normalized coordinates of each character in the standardized document image. S4: Input the character sequence and the corresponding character coordinate sequence into the sequence labeling model to obtain the structured information label corresponding to each character, specifically including: The character sequence is mapped into a text semantic feature vector through a first embedding network; The character coordinate sequence is mapped into a spatial layout feature vector through a second embedding network; Text semantic feature vectors and spatial layout feature vectors are input into a trainable fusion network. The interaction between the two feature vectors is modeled and fused through the fusion network to generate a fused multimodal feature vector. The multimodal feature vector is input into a sequence label decoding layer to decode the structured information label corresponding to each character, which is used to identify the type of document field.

2. The document information extraction method integrating spatial layout and textual semantics as described in claim 1, characterized in that, The object detection model is trained in the following way: a mixed sample set containing target language and non-target language text is constructed for training, and during the training process, the bounding boxes corresponding to all non-target language text regions in the mixed sample set are forcibly set as background category, thereby realizing negative sample reinforcement learning.

3. The method for extracting document information by integrating spatial layout and textual semantics as described in claim 1, characterized in that, The fusion network is a gated fusion network; The interaction relationship between the two feature vectors is modeled and fused using the fusion network to generate a fused multimodal feature vector. This process includes the following steps: For each current character in the character sequence, the gated fusion network dynamically generates a fusion weight vector based on the contextual semantics of the current character and its corresponding graph attention encoding. The fusion weight vector is then used to weight and fuse the textual semantic feature vector and spatial layout feature vector of the current character to generate the multimodal feature vector of the character. The graph attention encoding is obtained in the following way: S441: Take each identified character as a graph node, and use the normalized coordinate value of the character as its initial node feature to construct an undirected graph; wherein, if the Euclidean distance between two character nodes in the normalized coordinate space is less than a preset distance threshold, then an edge is established between the two nodes. S442: The constructed undirected graph and the initial node features of all its nodes are input into a graph attention network. The graph attention network iteratively processes the undirected graph through multiple graph attention layers. In each graph attention layer, each node updates its own node features by calculating the attention weights of all its neighboring nodes and performing a weighted summation of the features of its neighboring nodes. S443: The graph attention encoding of the corresponding character is obtained by using the feature vectors of each node output by the last graph attention layer of the graph attention network.

4. The method for extracting document information by integrating spatial layout and textual semantics as described in claim 1, characterized in that, In step S442, the undirected graph is iteratively processed. In each graph attention layer, each node updates its own node features by calculating the attention weights of all its neighboring nodes and performing a weighted sum of the features of its neighboring nodes. Specifically, this includes: Constructing a field-aware adjacency matrix to aggregate information at the graph attention layer; the construction of the field-aware adjacency matrix includes: S4421: Based on the character coordinate sequence, calculate the Euclidean distance between every two character nodes in the normalized coordinate space, and construct a spatial adjacency matrix based on the first distance threshold; S4422: Based on the text semantic feature vector obtained by the character sequence through the first embedding network, calculate the correlation degree between the features of every two character nodes, and construct a semantic correlation matrix based on the correlation degree threshold. The correlation degree is calculated using cosine similarity. S4423: The spatial adjacency matrix and the semantic association matrix are fused to generate a field-aware adjacency matrix; the graph attention layer, based on the field-aware adjacency matrix, calculates and normalizes the attention coefficients of the target node and all neighboring nodes defined by the field-aware adjacency matrix for each target node in the undirected graph, obtaining normalized attention weights, and uses the normalized attention weights to perform a weighted summation of the features of the neighboring nodes to obtain the updated node features corresponding to the target node.

5. The method for extracting document information by integrating spatial layout and textual semantics as described in claim 1, characterized in that, The deep learning-based image segmentation and correction model is a multi-task model that shares a single encoder and has two parallel decoding heads, including a first decoding head and a second decoding head, which perform the following two tasks respectively: The first decoding head performs pixel-level semantic segmentation on the main body area of ​​the document and outputs a segmentation mask for the main body area of ​​the document. The second decoding head regresses heatmaps of one or more preset key points in the original image of the document in parallel. The preset key points correspond to corner points or field center points with fixed layout features on the original image of the document. The perspective transformation and affine transformation are calculated according to the following steps: S11: Extract the bounding polygon of the document body region from the document body region segmentation mask to obtain the first set of candidate corner coordinates. ; S12: Extract the coordinates of each key point from the key point heatmap using a peak-finding algorithm to obtain a second set of candidate corner point coordinates. ; S13: Based on the predefined geometric relationships of the document template, and The candidate points are matched and merged to form a final set of target corner points. ; S14: Based on the final target corner point set Using the corner coordinates of the standard document template, a least-squares problem is solved to calculate the transformation matrix used for the perspective and affine transformations. .

6. The method for extracting document information by integrating spatial layout and textual semantics as described in claim 1, characterized in that, The fusion network is a candidate field-aware fusion network. Text semantic feature vectors and spatial layout feature vectors are input into the fusion network. The fusion network models and fuses the interaction relationship between the two feature vectors to generate a fused multimodal feature vector, including: Based on the text semantic feature vector, a field candidate generator is used to generate a candidate list containing the k most likely document field types. ,in, Indicates the type of the k-th candidate field. express Predictive probabilities based on text semantics; For a predefined field layout knowledge base, based on each candidate field type T in the candidate list L k Query Corresponding standard spatial layout descriptor The standard spatial layout descriptor It is extracted from the standard document template and used to characterize this field type. Standardized spatial characteristics, standard spatial layout descriptor The dimension is consistent with the spatial layout feature vector; The fusion network receives standard spatial layout descriptors of all candidate field types obtained from the spatial layout feature vector query. and its corresponding prediction probability ; For each candidate field type By calculating the spatial layout feature vector and the standard spatial layout descriptor The similarity in the feature space is used to obtain the layout matching score. ; Layout matching score based on all candidate field types and its prediction probability It outputs a modality selection preference signal through a trainable decision network. ,in, , Used to characterize the proportion of contribution that spatial layout features should have to the final decision relative to textual semantic features in the current context; Based on the modality selection preference signal The text semantic feature vector is processed according to the following formula. With the spatial layout feature vector The feature vectors are fused to generate the multimodal feature vectors. :

7. The method for extracting document information by integrating spatial layout and textual semantics as described in claim 1, characterized in that, The perspective and affine transformations based on the vertex coordinates of the bounding polygon of the segmented document body region specifically include: Detect the smallest bounding rectangle of the polygon surrounding the main body area of ​​the document; Based on the correspondence between the four vertices of the minimum bounding rectangle and the four corner points of the standardized document image, the perspective transformation matrix is ​​calculated. The original image of the document is transformed using the perspective transformation matrix to correct the perspective distortion of the main body area of ​​the document; An affine transformation is performed on the main body area of ​​the document after perspective transformation, scaling the main body area of ​​the document to a preset fixed size, and outputting a standardized document image with a fixed aspect ratio.

8. The method for extracting document information by integrating spatial layout and textual semantics as described in claim 1, characterized in that, The image block is input into the optical character recognition model, and the output character sequence and corresponding character coordinate sequence specifically include: The image block is input into an optical character recognition model based on a convolutional recurrent neural network structure. The optical character recognition model extracts visual features from the image block through its internal convolutional neural network and performs sequence modeling on the extracted features through its internal recurrent neural network. It simultaneously outputs the recognition results of the character sequence contained in the image block and the bounding box coordinates of each character in the character sequence in the standardized document image coordinate system. The bounding box coordinates of each character are processed to calculate the center point coordinates of each character's bounding box; the x-coordinate of the center point coordinates is divided by the width of the standardized ID image, and the y-coordinate is divided by the height of the standardized ID image to obtain the normalized character coordinate sequence.

9. The method for extracting document information by integrating spatial layout and textual semantics as described in claim 1, characterized in that, Following step S3, an iterative optimization step for the bounding box is also included: For the character sequence output in step S3, calculate its overall or partial recognition confidence score; If the recognition confidence score is lower than a preset score threshold, then based on the bounding box coordinates of the text region, an extended image region containing the outer context of the bounding box is extracted from the standardized document image, and the extended image region is concatenated with the recognized low-confidence character sequence to serve as optimized input information. The optimized input information is fed into a pre-trained bounding box optimization network, which is based on a lightweight convolutional neural network and directly outputs the fine-tuning amount of the bounding box coordinates of the original text region through regression. The fine-tuning amount is added to the original bounding box coordinates of the text region to generate the optimized bounding box coordinates; Based on the optimized bounding box coordinates, image blocks are re-extracted from the standardized document image and input again into the optical character recognition model to execute step S3, so as to obtain the updated character sequence and character coordinate sequence.

10. A document information extraction system integrating spatial layout and textual semantics, characterized in that, The document information extraction system is used to perform the method as described in any one of claims 1-9, wherein the document information extraction system comprises: The image standardization module is used to input the original image of the document into the deep learning-based image segmentation and correction model, perform pixel-level semantic segmentation on the main body region of the document, and perform perspective transformation and affine transformation based on the vertex coordinates of the bounding polygon of the segmented main body region of the document, and output a standardized document image with a fixed aspect ratio. The target text detection module is used to input the standardized document image into the target detection model trained with target language region enhancement, so as to detect and output only the bounding box coordinates of text regions belonging to the preset target language in the standardized document image, and filter out text regions that are not in the preset target language. The character recognition and localization module is used to extract corresponding image blocks from the standardized document image based on the bounding box coordinates of the text region, input the image blocks into the optical character recognition model, and output a character sequence and a corresponding character coordinate sequence. The character sequence contains the character recognition result in each image block, and the character coordinate sequence contains the normalized coordinates of each character in the standardized document image. A multimodal information structuring module is used to input the character sequence and the corresponding character coordinate sequence into the sequence labeling model to obtain a structured information label for each character; the multimodal information structuring module includes: A text semantic embedding unit is used to map the character sequence into a text semantic feature vector through a first embedding network; A spatial layout embedding unit is used to map the character coordinate sequence into a spatial layout feature vector through a second embedding network; The feature fusion unit is used to input the text semantic feature vector and the spatial layout feature vector into a trainable fusion network. The fusion network models and fuses the interaction relationship between the two feature vectors to generate a fused multimodal feature vector. The sequence labeling unit is used to input the multimodal feature vector into a sequence labeling decoding layer to decode the structured information label corresponding to each character for identifying the document field type.

Images