Document image processing method, electronic equipment and storage medium

Abstract

The invention provides a document image processing method, electronic equipment and a storage medium, and the method comprises the steps: inputting a document image into a trained unified model, and obtaining a plurality of different types of recognition results corresponding to each target region in the document image; for each target area, generating a prompt text corresponding to the target area according to a plurality of different types of recognition results corresponding to the target area; and inputting the prompt text of each target area and the received user question into a trained multi-modal model to obtain a document understanding result corresponding to the user question output by the multi-modal model. The method and the device are used for fully fusing information of multiple modes, performing targeted understanding on complex document images and improving the accuracy, the flexibility and the practicability of document image processing.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a document image processing method, electronic device, storage medium, and computer program product. Background Art

[0002] With the continuous development of digital technology, document image processing has been widely applied in various fields. Document images contain not only text information but also various elements such as charts, tables, flowcharts, and images. These elements together constitute the complex and rich content of documents. Understanding document images requires comprehensive consideration of multiple information modalities.

[0003] Existing technologies can be mainly divided into two categories: deep learning optical character recognition (OCR) and systems based on multimodal large model understanding. Figure 1 A schematic diagram of the document image recognition architecture of the prior art is shown in FIG. Figure 1 As shown, the first method in the existing technology only inputs image data and outputs simple text information. However, its disadvantage is that it can only recognize discrete text and requires subsequent post-processing to provide the corresponding answer to the question, which affects the overall effect. The second method inputs image data and question text, and the output result can obtain a corresponding answer based on the question and image. However, when processing complex document images, it cannot fully integrate information from multiple modalities, nor can it provide targeted understanding results based on the user's specific questions. This limits the flexibility and practicality of document image processing technology in practical applications and cannot meet users' needs for in-depth understanding of complex document images. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a document image processing method, electronic device, storage medium and computer program product to fully integrate information from multiple modalities, conduct targeted understanding of complex document images, and improve the accuracy, flexibility and practicality of document image processing.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of the present application are as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for processing a document image, comprising: inputting a document image into a trained unified model to obtain recognition results of multiple different categories corresponding to each target area in the document image; for each target area, generating a prompt text corresponding to the target area based on the recognition results of multiple different categories corresponding to the target area; inputting the prompt text of each target area and the received user question into a trained multimodal model to obtain a document understanding result corresponding to the user question output by the multimodal model.

[0007] In an embodiment, the unified model comprises a target detection model, a text recognition model and a classification model, and the recognition results of the plurality of different categories comprise a position of the target region, text content and a chart type.

[0008] In an embodiment, the unified model further comprises a residual module, an output of the target detection model serving as an input of the residual module, and an output of the residual module serving as an input of the text recognition model and the classification model.

[0009] In an embodiment, the method further comprises, before the inputting of the document image into the trained unified model, obtaining a sample image and label information of a plurality of different categories corresponding to the sample image, inputting the sample image into an initial model to obtain prediction results of a plurality of different categories output by the initial model, performing model loss calculation according to the label information of a plurality of different categories and the prediction results of a plurality of different categories, and adjusting parameters of the initial model according to the calculated model loss until the model loss satisfies a preset condition to obtain the unified model trained by the initial model.

[0010] In an embodiment, the method further comprises, before the inputting of the document image into the trained unified model, obtaining a sample image and label information of a plurality of different categories corresponding to the sample image, inputting the sample image into an initial model to obtain prediction results of a plurality of different categories output by the initial model, performing model loss calculation according to the label information of a plurality of different categories and the prediction results of a plurality of different categories, and adjusting parameters of the initial model according to the calculated model loss until the model loss satisfies a preset condition to obtain the unified model trained by the initial model.

[0011] In an embodiment, the model loss calculation according to the label information of a plurality of different categories and the prediction results of a plurality of different categories comprises, for each category, determining a loss value between the label information corresponding to the category and the prediction results corresponding to the category, and determining an overall loss by weighted summation according to a weight corresponding to each category and the loss value corresponding to each category.

[0012] In an embodiment, the method further comprises dynamically adjusting the weight corresponding to each category to minimize the overall loss.

[0013] According to a second aspect of the present application, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the document image processing method described in the above embodiments.

[0014] According to a third aspect of the present application, a storage medium is provided, which stores a computer program executable by a processor to complete the document image processing method described in the above embodiments.

[0015] According to a fourth aspect of the present application, a computer program product is provided, comprising computer programs / instructions which, when executed by a processor, implement the document image processing method described in the above embodiments.

[0016] The document image processing method, electronic device, storage medium and computer program product provided by the above embodiments utilize a unified model to perform identification of multiple different categories for each target region in a document image, and generate prompt text, thereby providing more abundant information for a multi-modal model, which helps to improve the understanding accuracy of the overall content of the document image. By combining the prompt text of the target region with a user question, the multi-modal model can output more accurate and targeted document understanding results for the specific question of the user, thereby meeting the user's demand for specific information. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0018] Figure 1 A schematic diagram of the document image recognition architecture of the prior art;

[0019] Figure 2 A logical diagram of the document image processing method provided by the embodiments of the present application;

[0020] Figure 3 A flowchart of the document processing method provided by the embodiments of the present application;

[0021] Figure 4 A style diagram of the document image to be processed provided by the embodiments of the present application;

[0022] Figure 5 A flowchart of the generation of prompt text provided by the embodiments of the present application;

[0023] Figure 6A unified model training and inference process schematic diagram provided for an embodiment of the present application is shown in FIG. 1.

[0024] Figure 7 A unified model training flowchart provided for an embodiment of the present application is shown in FIG. 2.

[0025] Figure 8 A unified model training process schematic diagram provided for an embodiment of the present application is shown in FIG. 3.

[0026] Figure 9 A document image preprocessing process schematic diagram provided for an embodiment of the present application is shown in FIG. 4.

[0027] Figure 10 A document image annotation schematic diagram provided for an embodiment of the present application is shown in FIG. 5.

[0028] Figure 11 A chart category classification schematic diagram provided for an embodiment of the present application is shown in FIG. 6.

[0029] Figure 12 A model loss calculation process schematic diagram provided for an embodiment of the present application is shown in FIG. 7.

[0030] Figure 13 An implementation code schematic diagram of a SE Layer module provided for an embodiment of the present application is shown in FIG. 8.

[0031] Figure 14 An implementation code schematic diagram of a SimAttention layer provided for an embodiment of the present application is shown in FIG. 9.

[0032] Figure 15 A structure schematic diagram of an electronic device provided for an embodiment of the present application is shown in FIG. 10.

[0033] Icon: 001-electronic device; 002-processor; 003-memory. DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

[0035] Figure 2 A logic schematic diagram of a document image processing method provided for an embodiment of the present application is shown in FIG. 11. Figure 3 A flowchart of a document processing method provided for an embodiment of the present application is shown in FIG. 12, which shows a document image processing method that can be executed by a processor, and the method includes the following steps S1-S3: Figure 2 Figure 3 The method includes the following steps S1-S3:

[0036] In step S1, a document image is input into a trained unified model to obtain recognition results of multiple different categories corresponding to each target region in the document image.

[0037] ​The document image is an image to be analyzed and processed, and the document image can be converted from a word document or a pdf document. Exemplarily, Figure 4 A schematic diagram of the style of the document image to be processed provided by the embodiment of the present application. The specific style of the document image includes but is not limited to the style as shown in Figure 4 For example, the document image can include an important trend chart as shown in Figure 4 a, a relationship chart as shown in Figure 4 b, a table chart as shown in Figure 4 c, etc.

[0038] The unified model can be a pre-trained deep learning model, or can be trained in the manner described in the following embodiments. Details are not repeated here.

[0039] Specifically, the data of the document image can be first read by using the cv2.imread function, and the channel can be converted from BGR to RGB by using the cv2.cvtColor function. At this time, the dimension of the data is the original dimension number [B, C, H, W], where C = 3 (indicating three channels of RGB). Next, the original RGB pixel value of the image is normalized. For example, after the original RGB value [230, 100, 20] is normalized, [0.0784, 0.0824, 0.0980] is obtained by dividing by 255, then the average value [0.485, 0.456, 0.406] is subtracted in the order of RGB, and then divided by the standard deviation [0.229, 0.224, 0.225], and finally [-1.7754, -1.6681, -1.3687] is obtained. The purpose of this step is to make all the processed values have a mean of 0 and a unit variance. The normalized features are then input into the unified model. It should be noted that during the training of the unified model, the normalized features are also input into the unified model, which can provide faster convergence and more stable training.

[0040] The unified model can identify different target regions in the document image and output recognition results of multiple different categories corresponding to each target region. The target region refers to a region in the document image that has a specific content or function, such as a text paragraph region, a table region, a chart region, etc. Different categories refer to the types of these target regions, such as different categories that can include text paragraphs, tables, flowcharts, bar charts, line charts, pie charts, etc. The recognition result refers to the analysis result of the unified model for each target region, and the recognition result can include the position of the target region, the text content, and the chart type, etc. The recognition result is obtained by processing the document image through multiple functional modules of the unified model.

[0041] By identifying different types of elements such as text paragraphs, tables, charts, etc. in the document image, detailed identification information is provided for subsequent processing, enabling a comprehensive understanding of the structure and content of the document image. In this way, multiple elements in the document can be efficiently identified, avoiding the limitations of single modal identification, and improving the accuracy and completeness of identification.

[0042] In an embodiment, the unified model can include a target detection model, a text recognition model, and a classification model, and the multiple different categories of recognition results include the positions of target regions, text content, and chart types.

[0043] Specifically, the unified model integrates multiple functional models and can simultaneously process multiple types of information in the document image, including a target detection model, a text recognition model, and a classification model. Among them, the target detection model is used to identify different target regions in the document image and determine the positions of these regions. It can detect the position information of various elements (such as text paragraphs, tables, charts, etc.) in the document, usually in the form of a bounding box (bounding box). For example, it can detect the specific position of a table in the image, including the coordinates of its upper left corner and lower right corner. Assuming that the document image contains a table, the target detection model can output the position information of the table, such as [x0, y0, x1, y1], indicating that the upper left corner coordinates of the table are (x0, y0) and the lower right corner coordinates are (x1, y1).

[0044] The text recognition model (usually based on OCR technology) is used to recognize the text content in the target region. It can convert the text in the image into editable text format for subsequent processing. For example, it can recognize the text content in a table or a text paragraph in the document. If the target region is a text paragraph, the text recognition model can output the text content of the paragraph, such as "This is an example of text."

[0045] The classification model is responsible for classifying the target region and determining its category. It can identify whether the target region is a text paragraph, a table, a chart (such as a bar chart, a line chart, etc.), or other types of elements. For example, it can classify a region containing data as a table or a region containing trend information as a line chart. Assuming that the target region is a chart, the classification model can output its category, such as "bar chart" or "line chart".

[0046] Through these sub-models, the unified model can comprehensively identify different target regions in the document image and output multiple information such as position, text content, and chart type, providing a rich information foundation for subsequent document understanding, enabling more accurate understanding of the content of the document image.

[0047] In an embodiment, the unified model further comprises a residual module, the output of the target detection model is input into the residual module, and the output of the residual module is input into the text recognition model and the classification model. The step S1 specifically comprises: inputting the document image into the target detection model to obtain the position of the target region output by the target detection model, obtaining the text content output by the text recognition model, and obtaining the chart type output by the classification model. The output of the target detection model is output to the text recognition model and the classification model through the residual module, so as to ensure that the unified model has accurate performance in the text recognition and classification tasks.

[0048] Step S2, for each target region, generating prompt text corresponding to the target region according to the recognition results of multiple different categories corresponding to the target region.

[0049] The prompt text is a natural language text that summarizes and describes the recognition results of the target region, which is used to provide more intuitive information for subsequent document understanding. For example, if the target region is identified as a table, the prompt text can be structured data containing row titles, column titles and specific data. The role of this step is to convert complex recognition results into natural language form that is easy to understand, providing auxiliary information for subsequent multi-modal understanding, helping the model better understand the content of the document image.

[0050] Step S3, inputting the prompt text of each target region and the received user question into the trained multi-modal model to obtain the document understanding result corresponding to the user question output by the multi-modal model.

[0051] The multi-modal model is a deep learning model that can process multiple different types of data simultaneously. It combines visual information (such as the position of the target detection frame, the chart type, etc.) and language information (such as the prompt text, the user question, etc.) to generate more accurate and comprehensive document understanding results. The user question is a query or request that the user makes about the content of the document, usually expressed in natural language. The user question can involve specific content in the document, including but not limited to table data, chart trends, text paragraphs, etc. For example, the user question can be: "What is the content of Table 2?" "What trends are shown?" "Summarize the key information in the document." Figure 3 The document understanding result is the output of the multi-modal model generated according to the user question and the prompt text, aiming to answer the user's question and provide detailed explanations of the content of the document. For example: if the user question is "What is the content of Table 2?", combined with the content of the prompt text, the output of the document understanding result may be "Table 2 shows the sales data of each quarter in 2024, the specific data is as follows: the first quarter sales is 100 million yuan, the second quarter sales is 120 million yuan, the third quarter sales is 130 million yuan, and the fourth quarter sales is 150 million yuan."

[0052] By performing multiple different types of recognition on each target region in the document image through the unified model and generating prompt text, more rich information is provided for the multi-modal model, which helps to improve the accuracy of understanding the overall content of the document image. By combining the prompt text of the target region with the user question, the multi-modal model can output more accurate and targeted document understanding results for the specific question of the user, not only improving the accuracy and flexibility of document image processing, but also enhancing the interactivity of the system, so that the user can obtain personalized document understanding content according to his own needs.

[0053] Figure 5 is a detailed flowchart of step S2 in the above embodiment. In an embodiment, as shown in Figure 5 , step S2 includes the following steps S21-S22.

[0054] Step S21, for each target region, post-processing the recognition results of multiple different types corresponding to the target region to obtain the result intermediate.

[0055] Specifically, Figure 6 The unified model training and inference process diagram provided by the embodiments of the present application, combined with Figure 2 and Figure 6 As shown, in the model inference stage, the recognition results (such as the coordinates of the target detection frame, the category information, the text content, etc.) output by the unified model need to be post-processed. Post-processing refers to a series of conversion and optimization operations on the original recognition results output by the unified model, in order to convert them into a format that is easier to understand and use. The process of post-processing includes restoring the normalized coordinate information (such as the center point coordinates, width and height) to the real size, mapping the category ID to the specific category name (such as "table", "flowchart", "column chart", etc.), mapping the encoding of the OCR text to the actual text content, etc. These processed results are integrated into the result intermediate, which serves as the basis for generating prompt text subsequently.

[0056] In the model inference stage, the original results of the unified model output are as follows (taking a single document screenshot as an example): results = [[cx, cy, h, w, cls_det, cls_img, ocr_text],...]. Among them, the content of results may contain multiple members, and each member represents all the results in a target detection box. Specifically, [cx, cy, h, w, cls_det] represents the normalized center point coordinates of the target detection box and the height and width of the box. These normalized coordinates need to be converted to real dimensions, and the dimensions of the original picture are H x W. Finally, the coordinate information of the coordinate box is [cx * W, cy * H, h * H, w * W], which indicates that the model inference information has been restored to the real image. For cls_img, after passing through the last fully connected layer (FC) of the classification model, the class number of the document small picture can be obtained through the softmax operation. Similarly, ocr_text will be mapped to the dictionary table after processing. For example, a result [13, 345, 13, 78, 12, 45] can be converted to actual text such as "I love my motherland" by referring to the dictionary table ID.

[0057] Step S22, combine the result middleware with the context information of the target area to obtain the prompt text corresponding to the target area.

[0058] Specifically, the context information of the target area refers to the text content related to the target area in the document, usually located near the target area. These text paragraphs provide background information or explanations of the target area. For example, if the target area is a chart, the context information may be the chart title or the explanatory text below the chart. The result middleware refers to the recognition result of the target area after post-processing, which includes the position information, category information and text content of the target area. The prompt text refers to the natural language text that summarizes and describes the recognition result of the target area and the context information. By combining the context information, a more targeted and accurate prompt text can be generated. For example, assume that the target area is a table, its category is "table", and the OCR text content is "Table 2: Sales Data". The context information is the text paragraph near the table, such as "Table 2 is the sales data of each quarter in 2024". Combining these information, the generated prompt text can be: "Table 2 shows the sales data of each quarter in 2024, the specific data is as follows: … …".

[0059] In an embodiment, in order to better process different types of document information (such as text, table, chart, etc.), different prompt templates (prompt) are designed for each type. These templates define the description method and task requirements of different types of data, so that the generated prompt text can more accurately reflect the content and purpose of the target area. Among them, the prompt template includes but is not limited to:

[0060] 'Chart: Description: Organizes data in rows and columns, suitable for presenting structured numerical information with clear column headers and row data, allowing for clear comparison of different data items. Task: Identify and describe the chart type, its components, and the information it conveys.

[0061] 'Flowchart: Description: Represents steps, decisions, flow directions, and logical relationships through graphical symbols (e.g., rectangles, diamonds), emphasizing the sequence of steps, conditions, and decision points. Task: Identify and describe the flowchart's components, the logic it represents, and the information it conveys.

[0062] 'Bar Chart: Description: Represents data for different categories using the length of rectangular bars, suitable for displaying the size comparison of categories, either horizontally or vertically, making it easy to compare numerical differences. Task: Identify and describe the bar chart's components, the categories it represents, and the information it conveys.

[0063] 'Line Chart: Description: Connects data points with lines, suitable for showing trends in continuous changes, emphasizing time series or the process of quantity changes. Task: Identify and describe the line chart's components, the trend it represents, and the information it conveys.

[0064] 'Pie Chart: Description: Divides the whole data into several parts, showing the proportion of each part in the whole, suitable for displaying proportional relationships. Task: Identify and describe the pie chart's components, the proportions it represents, and the information it conveys.

[0065] 'Scatter Plot: Description: Represents the relationship between two variables through the distribution of points, revealing patterns such as correlation, trends, and clustering between variables. Task: Identify and describe the scatter plot's components, the relationship it represents, and the information it conveys.

[0066] 'Code Diagram: Description: Shows the structure, syntax, or logic of code, suitable for demonstrating the functionality, implementation method, or algorithm of a program. Task: Identify and describe the code diagram's components, the code it represents, and the information it conveys.

[0067] Next, the corresponding prompt template is selected based on the category of the target region, and the final prompt text is generated by combining the context information. Specifically, the text paragraph type data is used to extract the text content related to the target region (such as the chart), which is usually located near the chart as context information. These context information (denoted as context) are added to the pre-defined prompt template to generate a complete prompt text. The generated prompt text will finally be input into the multi-modal model together with the user question to generate more accurate document understanding results. For example, the user may ask "What are the future trends like?", which will generate a detailed answer by combining the prompt text and the user question. Figure 3

[0068] Through post-processing and the combination of context information, the generated prompt text can more accurately reflect the content and background of the target region, thereby improving the multi-modal model's understanding of document images and enabling the system to better handle different types of document images and user questions, meeting diverse application needs.​

[0069] Figure 7 A training flowchart of the unified model provided for the embodiments of the present application is shown in Figure 8 A training flowchart of the unified model provided for the embodiments of the present application is shown in Figure 6 、 Figure 7 and Figure 8 The training process of the unified model is as follows:

[0070] In an embodiment, before inputting the document image into the trained unified model, the method further comprises the following steps S11-S14:

[0071] Step S11, obtaining a sample image and label information corresponding to the sample image in multiple different categories.

[0072] Wherein, obtaining the sample image and the label information corresponding to the sample image in multiple different categories comprises pre-processing the sample image, wherein the pre-processing content includes data preparation, data labeling and data input feature conversion.

[0073] Figure 9 A sample image pre-processing flowchart provided for the embodiments of the present application is shown in Figure 9 Data preparation is the first step of pre-processing, mainly involving the collection of sample image data. For PDF data, a large number of PDF papers can be downloaded from public websites, and internal PDF and Word documents can also be used as resources. After preliminary processing, the documents form the original document data, providing a basis for subsequent labeling and model training.

[0074] Figure 10 A sample image labeling diagram provided for the embodiments of the present application is shown in Figure 10 Next, for the target detection task, different objects in the sample image need to be recognized, including text paragraphs, tables, flowcharts, trend charts, formulas and pure pictures. Therefore, a target detection labeling tool is used for labeling. The specific process is as follows: after loading the picture, the specific object is boxed according to the above-mentioned categories, and the labeling result is saved. The original label data is stored in VOC format, which contains the coordinates and category information of the target box. This part of data is recorded as target detection GT (target detection ground truth).

[0075] Subsequently, the OCR text data is labeled. First, the data labeled in the target detection task is cropped into single small pictures according to the bounding box, which is used for the OCR text task. Since the OCR technology is relatively mature, the cropped pictures can be pre-labeled using a public model to obtain the OCR original text label. Then, manual verification is performed to check the accuracy of the text recognition result. If missing or errors are found, the text content needs to be corrected. Although this process still requires human intervention, it greatly reduces the workload compared to starting from scratch. This part of data is recorded as OCR text recognition GT.

[0076] Finally, the chart type classification task is performed. The classification of this task is more detailed and mainly serves the subsequent plug-in middleware, involving categories such as tables, flowcharts, bar charts, line charts, pie charts, scatter charts, radar charts, tree charts, 3D charts, formula charts, code charts, explanation charts, and pure pictures, etc. Figure 11 The chart category classification diagram provided by the embodiments of the present application is shown in Figure 11 as shown, the cropped single small pictures also need to be manually labeled by humans, and these labeled data are recorded as classification GT.

[0077] After the above steps, the data and labels required for all tasks have been manually labeled and pre-labeled, and have been verified by humans. Next, the original pictures are converted into original features. For the original features of the pictures, including the original large pictures and single small pictures, the original pictures are read and the RGB values are normalized to 0-1. For the label GT, the VOC format label of the target detection GT needs to be converted to [cx, cy, h, w, cls] format. Among them, the box information in the VOC format is x0, y0, x1, y1, and the category label. The calculation formula is: w = x1-x0, h = y1-y0, cx = x0+w / 2, cy = y0+h / 2, and cls is the category position index (for example, when the category is “flowchart”, cls is 2, the index starts from 0), which is recorded as target detection GT_DET. The OCR text recognition GT is encoded through a public dictionary table, and each word corresponds to an ID. For example, the OCR text “this is a test text” is encoded as [10, 23, 675, 987, 54, 1034, 8765, 87] after dictionary encoding, which is recorded as OCR text recognition GT_OCR. The classification GT needs to be converted into the corresponding ID. For example, when the category is “3D chart”, the converted cls is 8, which is recorded as classification GT_CLS.

[0078] At this point, the preprocessing work of the data part is completed, and the sample images and the label information corresponding to the sample images of multiple different categories have been obtained, providing a necessary data basis for model training.

[0079] Step S12, taking the sample image as the input of the initial model, obtaining the prediction results of multiple different categories of the initial model output.

[0080] The initial model refers to a model that has not been trained and optimized before the training begins, and usually includes target detection modules, text recognition modules, and classification modules, etc. The initial model will try to identify the target region in the sample image and output the prediction results of the corresponding categories through each module. The prediction result refers to the analysis result of the initial model about different target regions in the input sample image after processing, including the predicted position of the target region, the predicted text content, and the predicted chart type, etc.

[0081] Step S13, performing model loss calculation according to the label information of multiple different categories and the prediction results of multiple different categories.

[0082] The loss value reflects the difference between the model prediction result and the true value, and through loss value calculation, the performance of the model on different tasks can be comprehensively evaluated, and the adjustment of the model parameters can be guided.

[0083] In an embodiment, Figure 12 The flowchart of model loss calculation provided by the embodiment of the present application is shown in Figure 8 and Figure 12 As shown, step S13, performing model loss calculation according to the label information of multiple different categories and the prediction results of multiple different categories, includes the following steps S131-S132:

[0084] Step S131, for each category, determining the loss value between the label information corresponding to the category and the prediction result corresponding to the category.

[0085] In the model training process, for each different category (such as target detection, text recognition, classification, etc.), the loss value between the label information corresponding to each category and the prediction result corresponding to each category is calculated respectively, including the IoU loss (iou_loss) of the target detection module, the target confidence loss (obj_loss) and the classification loss (cls_loss_1), the loss of the classification feature head (cls_loss_2), the CTC loss (ctc_loss) of the OCR feature head, and the MSE loss (mse_loss) of the distillation branch of the OCR task. Among them, the CTC loss solves the problem of inconsistent length between input sequence and output sequence by allowing blank labels and flexibility of alignment, while the MSE loss is used to optimize the OCR feature head parameters.

[0086] Step S132, according to the weight corresponding to each category and the loss value corresponding to each category, determining the overall loss by weighted summation.

[0087] Wherein, after obtaining the loss value of each category, a corresponding weight is assigned to it according to the importance and complexity of each category. The weight reflects the importance of different tasks in the overall model performance. For example, the target detection task may have a greater impact on the overall performance of the model, so it is given a higher weight; while the text recognition task is also important, but may be relatively secondary in some application scenarios, so the weight is slightly lower. Multiply the loss value of each category by its corresponding weight, and then sum all the weighted loss values of all categories to obtain an overall loss. The lower the overall loss, the better the comprehensive performance of the model, and the more accurate the prediction of different categories. In this way, the system can balance the importance of different tasks, optimize the model parameters, and make the model achieve the best performance on multiple tasks.

[0088] By assigning weights to each category, the contribution of each task in the total loss can be dynamically adjusted according to its importance and complexity, making the model training focus more on key tasks. The overall loss obtained by weighted summation can comprehensively reflect the performance of the model on multiple tasks, and by minimizing the overall loss, the model parameters are jointly optimized, so that a better performance balance is achieved on multiple tasks.

[0089] In an embodiment, step S13 further comprises step S133, dynamically adjusting the weight corresponding to each category to minimize the overall loss.

[0090] Specifically, the dynamic adjustment of the weight corresponding to each category is realized by introducing an adaptive weight parameter ada_weights. These weights are learnable parameters, initialized as a tensor (e.g. torch.ones(6)) and set to require gradient update (requires_grad=True). During model training, the loss value of each category is multiplied by its corresponding weight, and then all the weighted loss values are added to obtain the total loss. Through backpropagation, the gradient of the total loss with respect to each weight is calculated, and the weights are updated according to these gradients, thereby dynamically adjusting the contribution of each category in the total loss. This adaptive adjustment mechanism enables the model to automatically balance the importance of different tasks according to the performance during training, in order to reduce the overall loss, and finally make the overall loss reach the minimum.

[0091] Step S14, according to the calculated model loss, adjust the parameters of the initial model until the model loss meets the preset condition, and obtain the unified model trained by the initial model.

[0092] Wherein, according to the calculated model loss, the parameters of the initial model are adjusted using the back propagation algorithm. By continuously iterating and optimizing the model parameters, the model loss value is gradually reduced until the model loss meets the preset conditions (such as the loss value is lower than a certain threshold or reaches the maximum number of iterations). After this training process, the initial model gradually learns how to more accurately identify and understand various target regions in the document image, and finally obtains the trained unified model.

[0093] In an embodiment, as shown in Figure 8 To improve the indicators of the unified model, several key modifications are made to the architecture of the traditional target detection model.

[0094] Firstly, the first convolutional layer of the target detection model is modified to the Focus layer. The Focus layer can more effectively extract low-level features of the input image through slicing operation, especially having obvious advantages in small object detection. Compared with the traditional convolutional layer, the Focus layer can significantly reduce the amount of calculation and improve the efficiency of feature extraction by reducing redundant calculation. This design enables the model to more accurately identify and locate small objects when processing document images containing small objects.

[0095] Figure 13 The implementation code diagram provided by the embodiment of the present application containing the SELayer module is shown in Figure 13 Next, the SELayer is introduced in the SPP layer of the target detection model. The SPP layer is a kind of spatial pyramid pooling layer, which is used in the backbone feature extraction network to improve the ability of feature extraction. The SELayer is a kind of attention mechanism module, which can better capture important feature information by adaptively adjusting the weight between channels, so as to improve the sensitivity of the model to key targets. After adding the SELayer, the model has stronger ability in feature selection, especially the performance in complex scenes is significantly improved. This mechanism enables the model to more effectively focus on important areas in the document image, improving the accuracy of identification.

[0096] Finally, the Upsample layer in the target detection model is replaced with a ConvTranspose2d layer. The Upsample layer is a simple up-sampling layer commonly used to increase the spatial resolution of feature maps. The ConvTranspose2d layer (also known as deconvolution or transpose convolution layer) is a more complex up-sampling layer that increases the spatial resolution of feature maps through a convolution transpose operation. ConvTranspose2d can more accurately restore the spatial resolution of images through the convolution transpose operation, avoiding the blurring phenomenon caused by traditional interpolation methods. Using the ConvTranspose2d layer not only effectively improves the quality of up-sampling, but also provides better performance in multi-scale detection tasks, especially when dealing with large size targets, the effect is more significant. This improvement makes the model maintain higher accuracy and robustness when dealing with targets of different sizes, while significantly reducing the consumption of computing resources. These improvements enable the model to more effectively recognize and understand various elements in the image when processing complex document images, providing more accurate foundation data for subsequent document understanding tasks.

[0097] In an embodiment, referring to Figure 8 As shown, the unified model further includes a residual module, which is a network structure component used to optimize the feature extraction capability of the model. By introducing residual connections, it allows signals in the network to bypass certain layers directly, thereby alleviating the gradient vanishing problem in deep network training and enhancing the training effect and stability of the model.

[0098] Specifically, the document image is first input into the target detection model, which is responsible for identifying the location information of different target regions in the document image and outputting the bounding box coordinates of these target regions. These coordinate information provides accurate region positioning for subsequent text recognition and classification tasks. Subsequently, these location features are passed to the residual module, which further optimizes and enhances the location features extracted by the target detection model through the introduction of residual connections. The optimized location features are then passed to the text recognition model and the classification model to obtain the text content output by the text recognition model and the chart type output by the classification model. The text recognition model (usually based on OCR technology) is responsible for recognizing the text content in the target region and converting the image text into editable text format. The classification model is responsible for classifying the target region and determining its category, such as tables, flowcharts, bar charts, etc.

[0099] Through this design, various information (position, text content, chart type) in the document image can be processed by a special model respectively, and the feature extraction process is optimized through the residual module to improve the accuracy of text recognition and classification. It can also output diversified recognition results such as the position of the target area, text content and chart type, providing a richer information foundation for subsequent document understanding. The introduction of the residual module makes the model more efficient when processing complex document images, reduces the risk of overfitting, and improves the overall performance of the model.

[0100] In an embodiment, since the input of the residual module is multiple output features in the target detection task, it requires a residual network architecture that is adaptable and can effectively process complex input data. Therefore, the residual module is a self-defined residual network backbone, which includes at least two residual blocks, each of which includes a convolutional layer, a lightweight attention mechanism layer and a residual connection, for enhancing the robustness and accuracy of feature extraction.

[0101] In an embodiment, Figure 14 The implementation code schematic diagram of the SimAttention layer provided by the embodiment of the present application is shown as follows: Figure 14 As shown in the figure, the residual module adopts a structure composed of three residual blocks, and each residual block embeds a lightweight attention mechanism SimAttention layer. The SimAttention layer optimizes the highlighting of important information in the feature map by adaptively weighting the input feature map, thereby effectively improving the feature extraction capability, especially when processing complex or diversified image data, which can significantly enhance the robustness and recognition accuracy of the network. This design enables the model to pay more attention to and utilize key features more effectively, thereby improving the overall performance.

[0102] In terms of network structure design, the first residual block serves as the input layer of the backbone network, undertaking the preliminary mapping task from the original feature to the network representation, laying the foundation for subsequent feature extraction. The second residual block serves as the intermediate residual block, further optimizing the feature extraction process. The third residual block serves as the output layer, connecting the classification feature head (i.e. classification model) and the OCR feature head (i.e. text recognition model), providing targeted feature representation for different tasks. The classification feature head and the OCR feature head adopt the same convolutional network structure to ensure the consistency and transferability of the network in different tasks. This design not only improves the universality of the model, but also reduces the additional training cost caused by task differences.

[0103] Each feature head is composed of three layers of convolutional operations, each including a Conv2D layer (responsible for feature extraction), a ReLU activation function (introducing nonlinearity and enhancing the model's expressive power), a max-pooling layer (reducing the spatial dimension of the feature map and reducing the computational load), and a DropBlock2D layer. The DropBlock2D layer further improves the regularization effect of the network by introducing the dropout of local features during training, reducing the risk of overfitting. This regularization strategy is particularly important for handling large-scale datasets and can effectively improve the generalization ability of the model.

[0104] After the convolutional layer, the feature map is finally converted into the output result required by the target task through the fully connected layer, ensuring that the network has accurate performance in classification and OCR tasks. The role of the fully connected layer is to integrate the local features extracted by the convolutional layer into global features, thereby providing a decision basis for the final classification or OCR task. This structural design not only ensures the adaptability of the model in different tasks, but also improves the overall performance and efficiency of the model through a reasonable hierarchical structure and attention mechanism.

[0105] The embodiments of the present application also provide an electronic device, Figure 15 As shown in the structural schematic diagram of the electronic device provided by the embodiments of the present application, Figure 15 The electronic device 001 can include a processor 002 and a memory 003 for storing processor-executable instructions, and the processor 002 is configured to implement the steps of the document image processing method of any of the above-mentioned embodiments.

[0106] The embodiments of the present application also provide a computer-readable storage medium, which stores a computer program, and the computer program is executable by a processor to complete the document image processing method of any of the above-mentioned embodiments.

[0107] The embodiments of the present application also provide a computer program product, which includes computer programs / instructions, and the computer programs / instructions are executed by a processor to implement the document image processing method of any of the above-mentioned embodiments.

[0108] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are only illustrative.

[0109] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the method of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0110] The above are only embodiments of the present application and are not used to limit the protection scope of the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application should be included in the protection scope of the present application.

[0111] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for processing a document image, characterized in that: include: Inputting the document image into the trained unified model to obtain recognition results of multiple different categories corresponding to each target area in the document image; For each target area, generating a prompt text corresponding to the target area according to the recognition results of multiple different categories corresponding to the target area; The prompt text of each target area and the received user question are input into the trained multimodal model to obtain the document understanding result corresponding to the user question output by the multimodal model.

2. The method according to claim 1, characterized in that The unified model includes a target detection model, a text recognition model and a classification model, and the recognition results of the multiple different categories include the location of the target area, text content and chart type.

3. The method according to claim 2, characterized in that The unified model further includes: a residual module, the output of the target detection model serves as the input of the residual module; the output of the residual module serves as the input of the text recognition model and the classification model; the document image is input into the trained unified model to obtain recognition results of multiple different categories corresponding to each target area in the document image, including: The document image is input into the target detection model, the position of the target area output by the target detection model is obtained, the text content output by the text recognition model and the chart type output by the classification model are obtained.

4. The method according to claim 1, wherein For each target area, generating a prompt text corresponding to the target area according to the recognition results of the multiple different categories corresponding to the target area includes: For each target area, post-process the recognition results of multiple different categories corresponding to the target area to obtain result middleware; The result middleware is combined with the context information of the target area to obtain a prompt text corresponding to the target area.

5. The method according to claim 1, wherein Before inputting the document image into the trained unified model, the method further includes: Obtaining sample images and label information of multiple different categories corresponding to the sample images; Using the sample image as input of an initial model to obtain prediction results of multiple different categories output by the initial model; Calculating the model loss based on the label information of the multiple different categories and the prediction results of the multiple different categories; According to the calculated model loss, the parameters of the initial model are adjusted until the model loss meets the preset conditions, thereby obtaining a unified model trained by the initial model.

6. The method according to claim 5, characterized in that The step of calculating the model loss based on the label information of the multiple different categories and the prediction results of the multiple different categories includes: For each category, determining the loss value between the label information corresponding to the category and the prediction result corresponding to the category; According to the weight corresponding to each category and the loss value corresponding to each category, the overall loss is determined by weighted summation.

7. The method according to claim 6, characterized in that Also includes: The weight corresponding to each category is dynamically adjusted to minimize the overall loss.

8. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to execute the document image processing method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and the computer program can be executed by a processor to complete the document image processing method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, implements the document image processing method according to any one of claims 1 to 7.

Images