Intelligent character recognition and structured processing method based on multi-modal deep learning

Abstract

The invention discloses an intelligent character recognition and structured processing method based on multi-modal deep learning, and belongs to the technical field of character recognition, and the method comprises the following steps: 1, receiving an image uploaded by a user, carrying out the detection of a character main body region, and extracting a main body region image containing characters; step 2, preprocessing the image; 3, performing character target detection on the preprocessed image by adopting a YOLOv8 model, and positioning a text region; step 4, identifying the detected character area by using a CRNN model, and outputting text content; 5, performing layout analysis on the recognition result, and determining a reading sequence and a logic structure of the text; and step 6, using a natural language model, combining the text content and the coordinate information, and carrying out structured output on an identification result. Multiple tasks such as business licenses and licenses are integrated into one model, the problem of multiple models in the prior art is solved, the resource utilization rate is high, and the recognition precision is high.

Description

Technical Field

[0001] This invention belongs to the field of text recognition technology, specifically relating to an intelligent text recognition and structured processing method based on multimodal deep learning. Background Technology

[0002] Currently, business license and permit recognition technologies primarily employ a single-license-single-model approach, leading to the need to deploy multiple independent models for multi-license recognition, resulting in low resource utilization. Furthermore, existing recognition methods simply identify certain fixed fields, making it difficult to handle multiple tasks within the same image simultaneously. Moreover, recognition accuracy is limited when facing complex scenarios (such as text occlusion, uneven lighting, and deformation). In addition, existing technologies lack an understanding of text semantics and layout logic, frequently resulting in logical errors in the recognition results. Summary of the Invention

[0003] The purpose of this invention is to provide an intelligent text recognition and structured processing method based on multimodal deep learning, which integrates the recognition of business licenses, permits, etc. into a single model, solving the problem of multiple models in the prior art and achieving high recognition accuracy in complex scenarios.

[0004] To achieve the above objectives, the present invention provides the following technical solution: an intelligent text recognition and structured processing method based on multimodal deep learning, characterized by comprising the following steps:

[0005] Step 1: Receive the image uploaded by the user, perform text subject region detection, and extract the image of the subject region containing the text;

[0006] Step 2: Preprocess the image;

[0007] Step 3: Use the YOLOv8 model to detect text targets in the preprocessed image and locate the text region;

[0008] Step 4: Use the CRNN model to identify the detected text regions and output the text content;

[0009] Step 5: Analyze the layout of the recognition results to determine the reading order and logical structure of the text;

[0010] Step 6: Using a natural language model, the recognition results are output in a structured manner by combining the text content and coordinate information.

[0011] Furthermore, step 1, the detection of the main text region, includes the following steps:

[0012] Step 1.1: Use U2Net to detect text regions and obtain a saliency map of the text regions in the image;

[0013] Step 1.2: Generate a binary mask for the text region based on the saliency map output by U2Net;

[0014] Step 1.3: Use the binary mask of the generated text region to perform a masking operation on the original image and extract the main region containing the text.

[0015] Furthermore, step 2, image preprocessing, includes the following steps:

[0016] Step 2.1: Crop and perform affine transformation on the extracted text main area to correct the image tilt and distortion;

[0017] Step 2.2: Perform document distortion restoration on the corrected image;

[0018] Step 2.3: Perform super-resolution processing on the restored image.

[0019] Furthermore, in step 2.2, the DocUNet model is used to learn the geometric deformation features of the document image through a combination of convolutional neural network (CNN) and deconvolution network, and to predict the restored planar document image.

[0020] Furthermore, in step 2.3, the restored image is super-resolution processed using an improved SRGAN model. In the improved SRGAN model, Text-SSIM is used as part of the loss function, which is:

[0021] L total =λ1L adv +λ2L perceptual +λ3L Text-SSIM

[0022] Where: L adv Adversarial loss, used to improve the visual quality of images, L perceptual It is a perceptual loss used to preserve high-level semantic information of an image, L Text-SSIM λ1, λ2, and λ3 are the weight coefficients of adversarial loss, perceptual loss, and text loss, respectively, used to control the proportion of different losses.

[0023] Furthermore, in step 3, the YOLOv8 model is used to detect text targets in the preprocessed image and locate text regions, including the following steps:

[0024] Step 3.1: Use instance segmentation technology to segment the text region at the pixel level and generate an accurate text region mask;

[0025] Step 3.2: While detecting the text rectangle regions, predict the coordinates of the center point of each text rectangle region;

[0026] Step 3.3: Generate the bounding box of the text region based on the segmentation results and center point information;

[0027] Step 3.4: Output the bounding box and center point coordinates of each text region as input for subsequent text recognition.

[0028] Furthermore, in step 4, a CRNN model is used to identify the detected text regions and output the text content, including the following steps:

[0029] Step 4.1: Preprocess the text region image detected in Step 3, including grayscale conversion, normalization, and size adjustment;

[0030] Step 4.2, Convolutional Neural Network (CNN) Feature Extraction: MobileNet is used to extract features from text images;

[0031] Step 4.3: Serialize the feature map output by the CNN in the width direction to generate a feature sequence;

[0032] Step 4.4, Recurrent Neural Network (RNN) sequence modeling, using bidirectional LSTM (Long Short-Term Memory) to model the feature sequence and capture the contextual dependencies of the text sequence;

[0033] Step 4.5: Transcribe the RNN output using the CTC (Connectionist Temporal Classification) loss function to map the sequence into the final text content;

[0034] Step 4.6: Post-process the recognition results, including removing duplicate characters and correcting spaces.

[0035] Furthermore, step 5 involves analyzing the layout of the recognition results to determine the reading order and logical structure of the text, including the following steps:

[0036] Step 5.1: Preprocess the text region image detected in Step 5, including normalization and size adjustment operations;

[0037] Step 5.2: Use the YOLO model to detect layout elements in the document image and identify text blocks, titles, tables, images, and logo layout elements;

[0038] Step 5.3: Determine the reading order of the text based on the bounding box coordinates and category labels of the layout elements;

[0039] Step 5.4: Generate the logical structure of the document based on the reading order and the category tags of the layout elements;

[0040] Step 5.5: Output the reading order and logical structure of the document as the basis for subsequent structured output.

[0041] Furthermore, step 6 uses a natural language model to combine text content and coordinate information to output a structured version of the recognition results, including the following steps:

[0042] Step 6.1: Encode the text content using a language model and extract semantic features;

[0043] Step 6.2: Encode the coordinate information using an encoder to extract spatial features;

[0044] Step 6.3: Fuse the outputs of the text encoder and the coordinate encoder to generate joint features that combine text content and coordinate information;

[0045] Step 6.4: Design a task-specific layer based on the specific task and output task-specific results.

[0046] Compared with the prior art, the beneficial effects of the present invention are:

[0047] (1) It integrates multiple tasks such as business licenses and permits into one model, which solves the problem of multiple models in the existing technology, has high resource utilization, and high recognition accuracy when facing complex scenarios;

[0048] (2) Image quality was improved through document distortion recovery and affine transformation;

[0049] (3) Page layout analysis and reading order algorithms have been added to enhance the understanding of the text's logical structure;

[0050] (4) The introduction of natural language models enhances semantic understanding capabilities and reduces logical errors in recognition results;

[0051] (5) Coordinate information is added when training the natural language model, which enhances the robustness of the algorithm against interference such as text occlusion, lighting, and deformation. Attached Figure Description

[0052] Figure 1 System architecture diagram;

[0053] Figure 2 Flowchart of main body segmentation and geometric correction;

[0054] Figure 3 This is a schematic diagram of the YOLO text detection algorithm structure;

[0055] Figure 4 Example diagram of structured output results. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] Please see Figures 1-4 This invention provides a technical solution for intelligent text recognition and structured processing based on multimodal deep learning.

[0058] A method for intelligent text recognition and structured processing based on multimodal deep learning includes the following steps:

[0059] Step 1: Receive the image uploaded by the user, perform text subject region detection, and extract the image of the subject region containing the text;

[0060] Step 2: Preprocess the image;

[0061] Step 3: Use the YOLOv8 model to detect text targets in the preprocessed image and locate the text region;

[0062] Step 4: Use the CRNN model to identify the detected text regions and output the text content;

[0063] Step 5: Analyze the layout of the recognition results to determine the reading order and logical structure of the text;

[0064] Step 6: Using a natural language model, the recognition results are output in a structured manner by combining the text content and coordinate information.

[0065] This method is applicable to the automated recognition and structured output of text in scenarios such as business licenses, permits, and food packaging. It integrates the recognition of business licenses, permits, etc., into a single model, solving the problem of multiple models in existing technologies. It also achieves high recognition accuracy in complex scenarios.

[0066] Step 1, the detection of the main text region, includes the following steps:

[0067] Step 1.1: Use U2Net to detect text regions and obtain the saliency map of the text regions in the image. U2Net is a deep learning-based saliency detection model that can effectively detect text regions in an image. The saliency map of the text regions in the image can be obtained through the output of U2Net.

[0068] Step 1.2: Based on the saliency map output by U2Net, generate a binary mask for the text region. By setting an appropriate threshold (r>0.5), mark the highlighted areas (i.e., text regions) in the saliency map as the foreground and the rest as the background. Further optimize the extracted main text region, including removing small noise regions and merging adjacent text regions, to ensure that the extracted text region is complete and accurate.

[0069] Step 1.3: Using the binary mask of the generated text region, perform a masking operation on the original image to extract the main region containing the text. This step can remove irrelevant backgrounds from the image and retain the text portion.

[0070] Step 2, image preprocessing, includes the following steps:

[0071] Step 2.1: Crop and perform affine transformation on the extracted text main area to correct the image tilt and deformation. By detecting the edges or corners of the text area, calculate the tilt angle of the image and perform rotation and affine transformation to restore the text area to a normal rectangular shape.

[0072] Step 2.2: Perform document distortion restoration on the corrected image;

[0073] Step 2.3: Perform super-resolution processing on the restored image.

[0074] In step 2.2, the DocUNet model, through a combination of convolutional neural networks (CNNs) and deconvolutional networks, learns the geometric deformation features of the document image and predicts the restored planar document image. Specifically, to adapt to the data, a custom-generated image is provided: given image I, an MxN grid is placed on it, grid points are used as control points, randomly generating bending or folding deformations, and finally, a background is added.

[0075] Advantages of using DocUNet:

[0076] High-precision restoration: DocUNet technology can handle complex document deformations, including perspective deformation and surface deformation, and its restoration effect is better than traditional geometric transformation methods.

[0077] Wide applicability: Suitable for various scenarios, such as shooting tilted documents, curved book pages, folded paper, etc.

[0078] Automated processing: By combining deep learning with geometric modeling, DoCUNet technology can automatically recover documents without human intervention.

[0079] In step 2.3, an improved SRGAN model is used to perform super-resolution processing on the restored image. In the improved SRGAN model, Text-SSIM is used as part of the loss function, which is:

[0080] L total =λ1L adv +λ2L perceptual +λ3L Text-SSIM

[0081] Where: L adv Adversarial loss, used to improve the visual quality of images, L perceptual It is a perceptual loss used to preserve high-level semantic information of an image, L Text-SSIM λ1, λ2, and λ3 are the weight coefficients of adversarial loss, perceptual loss, and text loss, respectively, used to control the proportion of different losses.

[0082] SRGAN is a super-resolution technique based on Generative Adversarial Networks (GANs) that can restore low-resolution images to high-resolution images while preserving details and texture information. To further improve the quality of document images, this invention innovatively modifies SRGAN to make it more suitable for super-resolution processing of document images.

[0083] The improved SRGAN model incorporates the following innovations based on the original SRGAN:

[0084] Multi-scale feature extraction:

[0085] Introducing multi-scale convolutional layers into the generator can simultaneously capture low-frequency and high-frequency information of the image, improving the edge clarity of the text in the document.

[0086] Text region attention mechanism:

[0087] By adding a text region attention module to the discriminator, the model can focus more on the quality of the text region and avoid over-optimization of the background region.

[0088] Perceptual loss function optimization:

[0089] Text structure similarity (Text-SSIM) loss is introduced into the loss function to ensure the integrity and readability of the text structure. Different weights are assigned to each local block based on the importance of the text region. For example, text edge regions have higher weights, while background regions have lower weights.

[0090] Advantages of the improved SRGAN:

[0091] By introducing improved SRGAN technology, this invention can more efficiently improve the resolution of document images, significantly enhance the readability and recognition accuracy of text, and is suitable for various application scenarios such as document digitization and image text extraction.

[0092] YOLOv8 is a high-efficiency object detection model capable of detecting objects in images and generating bounding boxes in real time. To further improve the accuracy of text detection, especially to prevent segmentation errors in text regions that are too close together, this invention improves YOLOv8 by incorporating the center point information of the text rectangular region.

[0093] Step 3 uses the YOLOv8 model to detect text targets in the preprocessed image and locate text regions, including the following steps:

[0094] Step 3.1: Use instance segmentation technology to segment the text region at the pixel level and generate an accurate text region mask;

[0095] Step 3.2: While detecting the text rectangle regions, predict the center point coordinates of each text rectangle region. The center point information is used to assist in the accurate positioning of the text region and prevent segmentation errors of text regions that are too close.

[0096] Step 3.3: Based on the segmentation results and center point information, generate bounding boxes for the text region. Optimize the position of the bounding boxes using the center point information to ensure that the bounding boxes accurately cover the text region.

[0097] Step 3.4: Output the bounding box and center point coordinates of each text region as input for subsequent text recognition.

[0098] Advantages of improving YOLOv8:

[0099] High-precision detection: Through instance segmentation and center point prediction, text regions can be located more accurately, especially for complex backgrounds or overlapping text regions.

[0100] Preventing segmentation errors in text regions that are too close: Center point information can effectively prevent segmentation errors in text regions that are too close, thus improving detection accuracy.

[0101] Real-time performance: YOLOv8 has a high inference speed and can process documents and images in real time.

[0102] Wide applicability: Suitable for a variety of document images, including low-resolution, blurry, or noisy images.

[0103] Step 4 uses a CRNN model to identify the detected text regions and output the text content, including the following steps:

[0104] Step 4.1: Preprocess the text region image detected in Step 3, including grayscale conversion, normalization, and size adjustment (uniformly adjusted to a fixed height, such as 32 pixels, and the width scaled proportionally), and perform data augmentation on the image, such as random rotation, translation, and scaling, to improve the robustness of the model.

[0105] Step 4.2, Convolutional Neural Network (CNN) feature extraction: MobileNet is used to extract features from text images. The CNN part consists of multiple convolutional layers, pooling layers and activation functions (such as ReLU) to capture local features and global context information of text images. The output feature map has a size of H×W×C, where H is the height, W is the width and C is the number of channels.

[0106] Step 4.3 involves serializing the feature maps output by the CNN along the width direction to generate a feature sequence. The feature vector dimension at each time step is C, and the sequence length is W. This step converts the 2D feature maps into 1D feature sequences, facilitating subsequent processing by the recurrent neural network.

[0107] Step 4.4: Recurrent Neural Network (RNN) sequence modeling. A bidirectional LSTM (Long Short-Term Memory) is used to model the feature sequence, capturing the contextual dependencies of the text sequence. The bidirectional RNN can simultaneously consider forward and backward contextual information, improving the recognition ability of complex text. The hidden state at each time step is output, with dimensions T×D, where T is the sequence length and D is the dimension of the hidden state.

[0108] Step 4.5: Transcribe the RNN output using the CTC (Connectionist Temporal Classification) loss function to map the sequence to the final text content. The CTC loss function can handle the problem of inconsistent lengths between the input and output sequences, making it particularly suitable for text recognition tasks.

[0109] Step 4.6 involves post-processing the recognition results, including removing duplicate characters and correcting spaces, to improve the accuracy of the recognition results.

[0110] Advantages of CRNN models:

[0111] End-to-end training: The CRNN model can complete text detection and recognition tasks end-to-end without the need for additional character segmentation steps, simplifying the process.

[0112] Strong context modeling capability: The bidirectional LSTM model can capture the contextual dependencies of text sequences, improving the ability to recognize complex text.

[0113] Highly adaptable: The model can handle texts of different lengths, fonts, and languages, and has high versatility.

[0114] High efficiency: Using a lightweight CNN backbone (such as MobileNet), the model has high inference speed while ensuring recognition accuracy.

[0115] Step 5 involves analyzing the layout of the recognition results to determine the reading order and logical structure of the text, including the following steps:

[0116] Step 5.1: Preprocess the text region image detected in Step 5, including normalization and size adjustment, to ensure that the input image meets the input requirements of the YOLO model.

[0117] Step 5.2: Use the YOLO model to detect layout elements in the document image and identify text blocks, titles, tables, images, and logo layout elements. Each detected layout element contains a bounding box and a category label (such as "title", "text", "table", etc.).

[0118] Step 5.3: Determine the reading order of the text based on the bounding box coordinates and category labels of the layout elements. The specific method is as follows:

[0119] Horizontal priority: For languages ​​written from left to right (such as English and Chinese), text blocks are arranged horizontally first.

[0120] Vertical priority: For languages ​​written from top to bottom (such as traditional Chinese), text blocks are arranged vertically first.

[0121] Logical relationship inference: Infer the logical structure of the text based on the category labels of layout elements (such as "heading" usually being placed before "body");

[0122] Step 5.4: Generate the logical structure of the document based on the reading order and the category tags of the layout elements, for example: title → text → table → image;

[0123] Step 5.5: Output the reading order and logical structure of the document as the basis for subsequent structured output.

[0124] Advantages of improving the YOLO model:

[0125] High-precision detection: Through multi-task learning and context-aware mechanisms, it can more accurately detect page elements and predict reading order.

[0126] Real-time performance: The YOLO model has a high inference speed and can process documents and images in real time.

[0127] Highly adaptable: The model can handle multiple document formats and languages, and has high versatility.

[0128] Automated processing: The model can automatically perform page layout analysis and reading order prediction without human intervention.

[0129] By improving the YOLO model, this invention can efficiently and accurately analyze document layout and determine the reading order and logical structure of text. Combining innovations such as multi-task learning, context awareness, and adaptive anchor boxes, the model exhibits significant advantages in layout element detection, reading order prediction, and logical structure generation, making it suitable for various application scenarios such as business licenses and food packaging text extraction.

[0130] Step 6 uses a natural language model to combine text content and coordinate information to output a structured version of the recognition results. The coordinate-information-enhanced NLP structured processing includes the following steps:

[0131] Step 6.1: Encode the text content using a pre-trained language model (such as BERT or RoBERTa) to extract semantic features. Input: Text content (e.g., "document title"), Output: Semantic feature vector of the text;

[0132] Step 6.2: Encode the coordinate information using an encoder (such as a fully connected network or Transformer) to extract spatial features. Input: Coordinate information of the text region; Output: Spatial feature vector of the coordinates.

[0133] Step 6.3: Fuse the outputs of the text encoder and coordinate encoder to generate joint features that combine text content and coordinate information. Fusion methods: Concatenation: Concatenate the text features and coordinate features together. Attention mechanism: Dynamically adjust the weights of the text features and coordinate features using an attention mechanism.

[0134] Step 6.4: Design task-specific layers based on specific tasks (such as text classification, named entity recognition, and relation extraction), and output task-specific results (such as classification labels, entity labels, and relation labels).

[0135] Loss function:

[0136] L total =λ1L class +λ2L crf +λ3L bce +λ4L coord

[0137] Among them, L class This is a text classification task, using cross-entropy loss.crf Named entity recognition task, using CRF loss, L bce This is a relation extraction task, using binary cross-entropy loss, L coord The coordinate information is used to assist in the loss. λ1λ2λ3λ4 are weighting coefficients used to balance the task's loss.

[0138] Advantages of location-based NLP architecture:

[0139] Using coordinate information as labels: Inputting text coordinate information as labels into the NLP model helps the model better understand the spatial relationships and logical structure of sentences.

[0140] Spatial relationship modeling: Calculate the spatial relationships (such as vertical and horizontal relationships) between text blocks using coordinate information, and input them into the model as auxiliary labels.

[0141] Multimodal feature fusion: This method combines text features and coordinate features to generate joint features that combine text content and spatial information.

[0142] By inputting text coordinate information as labels into an NLP model, this invention can more accurately understand the spatial relationships and logical structure of sentences, thus solving the problem of text misinterpretation. Combining innovative features such as coordinate-assisted loss, multimodal feature fusion, and dynamic weight adjustment, the model exhibits significant advantages in text parsing and spatial relationship modeling, making it suitable for text extraction tasks in complex scenarios.

[0143] Advantages of this invention:

[0144] 1. Innovatively, it integrates multiple tasks such as business licenses and permits, food packaging information, etc., into one model, solving the problem of multiple models in existing technologies.

[0145] 2. Image quality was improved through document distortion recovery and affine transformation.

[0146] 3. Page layout analysis and reading order algorithms have been added to enhance the understanding of the text's logical structure.

[0147] 4. The introduction of natural language models enhances semantic understanding capabilities and reduces logical errors in recognition results.

[0148] 5. Adding coordinate information during the training of the natural language model enhances the algorithm's robustness to interferences such as text occlusion, lighting, and deformation.

[0149] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for intelligent text recognition and structured processing based on multimodal deep learning, characterized in that, Includes the following steps: Step 1: Receive the image uploaded by the user, perform text subject region detection, and extract the image of the subject region containing the text; Step 2: Preprocess the image; Step 3: Use the YOLOv8 model to detect text targets in the preprocessed image and locate the text region; Step 4: Use the CRNN model to identify the detected text regions and output the text content; Step 5: Analyze the layout of the recognition results to determine the reading order and logical structure of the text; Step 6: Using a natural language model, the recognition results are output in a structured manner by combining the text content and coordinate information.

2. The one according to claim 1, characterized in that, Step 1, the detection of the main text region, includes the following steps: Step 1.1: Use U2Net to detect text regions and obtain a saliency map of the text regions in the image; Step 1.2: Generate a binary mask for the text region based on the saliency map output by U2Net; Step 1.3: Use the binary mask of the generated text region to perform a masking operation on the original image and extract the main region containing the text.

3. The one according to claim 2, characterized in that, Step 2, image preprocessing, includes the following steps: Step 2.1: Crop and perform affine transformation on the extracted text main area to correct the image tilt and distortion; Step 2.2: Perform document distortion restoration on the corrected image; Step 2.3: Perform super-resolution processing on the restored image.

4. The one according to claim 3, characterized in that, In step 2.2, the DocUNet model is used to learn the geometric deformation features of the document image through a combination of convolutional neural network (CNN) and deconvolution network, and to predict the restored planar document image.

5. The one according to claim 3, characterized in that, In step 2.3, an improved SRGAN model is used to perform super-resolution processing on the restored image. In the improved SRGAN model, Text-SSIM is used as part of the loss function, which is: L total =λ1L adv +λ2L perceptual +λ3L Text-SSIM Where: L adv Adversarial loss, used to improve the visual quality of images, L perceptual It is a perceptual loss used to preserve high-level semantic information of an image, L Text-SSIM λ1, λ2, and λ3 are the weight coefficients of adversarial loss, perceptual loss, and text loss, respectively, used to control the proportion of different losses.

6. The one according to claim 1, characterized in that, Step 3 uses the YOLOv8 model to detect text targets in the preprocessed image and locate text regions, including the following steps: Step 3.1: Use instance segmentation technology to segment the text region at the pixel level and generate an accurate text region mask; Step 3.2: While detecting the text rectangle regions, predict the coordinates of the center point of each text rectangle region; Step 3.3: Generate the bounding box of the text region based on the segmentation results and center point information; Step 3.4: Output the bounding box and center point coordinates of each text region as input for subsequent text recognition.

7. The one according to claim 1, characterized in that, Step 4 uses a CRNN model to identify the detected text regions and output the text content, including the following steps: Step 4.1: Preprocess the text region image detected in Step 3, including grayscale conversion, normalization, and size adjustment; Step 4.2, Convolutional Neural Network (CNN) Feature Extraction: MobileNet is used to extract features from text images; Step 4.3: Serialize the feature map output by the CNN in the width direction to generate a feature sequence; Step 4.4, Recurrent Neural Network (RNN) sequence modeling, using bidirectional LSTM (Long Short-Term Memory) to model the feature sequence and capture the contextual dependencies of the text sequence; Step 4.5: Transcribe the RNN output using the CTC (Connectionist Temporal Classification) loss function to map the sequence into the final text content; Step 4.6: Post-process the recognition results, including removing duplicate characters and correcting spaces.

8. The one according to claim 1, characterized in that, Step 5 involves analyzing the layout of the recognition results to determine the reading order and logical structure of the text, including the following steps: Step 5.1: Preprocess the text region image detected in Step 5, including normalization and size adjustment operations; Step 5.2: Use the YOLO model to detect layout elements in the document image and identify text blocks, titles, tables, images, and logo layout elements; Step 5.3: Determine the reading order of the text based on the bounding box coordinates and category labels of the layout elements; Step 5.4: Generate the logical structure of the document based on the reading order and the category tags of the layout elements; Step 5.5: Output the reading order and logical structure of the document as the basis for subsequent structured output.

9. The one according to claim 1, characterized in that, Step 6 uses a natural language model to combine text content and coordinate information to output a structured version of the recognition results, including the following steps: Step 6.1: Encode the text content using a language model and extract semantic features; Step 6.2: Encode the coordinate information using an encoder to extract spatial features; Step 6.3: Fuse the outputs of the text encoder and the coordinate encoder to generate joint features that combine text content and coordinate information; Step 6.4: Design a task-specific layer based on the specific task and output task-specific results.

Images