Ticket recognition method and system for rpa office process automation system

Abstract

The present application belongs to the field of computer vision and natural language processing, and particularly relates to a bill identification method and system for an RPA office process automation system. The method disclosed in the present application realizes the series prediction of an OCR engine+SER+RE, and innovatively combines the advantages of an OCR model and a LayoutXLM model, thereby ensuring the prediction accuracy of each step, improving the automation degree of the process, and being capable of better meeting the task scenario of bill identification.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and natural language processing, and specifically relates to a method and system for document recognition in RPA office workflow automation systems. Background Technology

[0002] Natural Language Processing (NLP) is an interdisciplinary field encompassing computer science, artificial intelligence, and linguistics. It primarily studies how to enable computers to understand, process, generate, and simulate human language, thereby achieving the ability to engage in natural conversations with humans. NLP techniques enable various applications such as machine translation, question-answering systems, sentiment analysis, and text summarization. With the development of deep learning technology, artificial neural networks and other machine learning methods have made significant progress in the field of NLP. Future development directions include deeper semantic understanding, better dialogue systems, broader cross-language processing, and more powerful transfer learning techniques.

[0003] In recent years, with the rapid development of deep learning technology, both natural language processing and computer vision have gradually matured. However, neither can adequately address the needs of invoice recognition tasks when used alone. This is because invoice recognition requires not only identifying the position and content of text in a given image file but also analyzing the potential relationships between different text elements to extract key information. Therefore, combining the technical characteristics of both technologies to automate the invoice recognition process is a worthwhile research direction. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a document recognition method for RPA office workflow automation systems. By combining the accurate text detection and recognition capabilities of the OCR model with the text understanding capabilities of the LayoutXLM model, the document recognition process is comprehensively automated.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows: A method for document recognition in an RPA office workflow automation system includes the following steps: S1. Preparation phase: The RPA system receives the document file to be identified uploaded by the front end and processes it; S2. Text detection and text recognition stage: The CTPN model is used to detect text locations, and the CRNN model is used to recognize text content; S3. Semantic Entity Recognition Stage: Based on the results of text detection and recognition in step S2, the semantic entity recognition task scenario SER of the LayoutXLM text understanding model is used to further identify and classify the text entities, dividing the text entities into question type QUESTION and answer type ANSWER. S4. Relationship Extraction Stage: Using the LayoutXLM text understanding model, the Relationship Extraction (RE) task scenario is used to predict whether there is a relationship between text entities in the question and answer classes, and to perform matching. Finally, the results are returned to the front-end ticket recognition.

[0006] Preferably, in step S1, S11. Receive the invoice image or PDF to be processed; S12. Run the set process, encapsulate the request ticket file and send the corresponding request to the background RPA server; S13. If the image file is in PNG, JPG, JPEG, GIF, or BMP format, the system will temporarily save it. If the document file is in PDF format, the PDF file will be converted into an image first, and then temporarily saved to the system.

[0007] Preferably, in step S13, the specific method for converting the PDF file into an image is as follows: First, each page of the PDF file is read and converted into a pixel bitmap on a page-by-page basis. Then, the resolution and color space are specified and the image is rendered as a PIL bitmap for easy subsequent processing.

[0008] Preferably, step S2 includes image preprocessing, including background removal, tilt correction, and image enhancement; The steps to remove the background are as follows: 1) First, a background subtractor is obtained through the cv2.createBackgroundSubtractorMOG() function to separate the foreground and background of the image and obtain a binary image of the foreground part; 2) Using the morphological function cv2.morphologyEx(), with the cv2.MORPH_CLOSE parameter set, a closing operation is performed on the image to clearly separate the boundary between the foreground and background; 3) Overlay the processed foreground and background images to remove background parts unrelated to the text; The tilt correction steps are as follows: 1) Read the image in grayscale form and use the morphological function cv2.morphologyEx() to perform closing and opening operations on the image; 2) Use cv2.findContours to sort the image contours, obtain the bounding box of the largest contour, and obtain the four vertices through the largest bounding box; 3) Using vertex information, generate a perspective transformation matrix using the cv2.getPerspectiveTransform() function and perform perspective transformation using the cv2.warpPerspective() function to correct the image tilt and restore the text lines to horizontal. The image enhancement steps are as follows: 1) Calculate the grayscale histogram of the image using the hist function in the Matplotlib library, and calculate the cumulative histogram of the image; 2) The mapping relationship between input gray levels and output gray levels is defined as follows: (1); in For input pixels, To output pixels, The height of the image. The width of the image; 3) Output the gray level of each pixel in the image in a loop according to the mapping relationship.

[0009] Preferably, in step S2, the text position detection step is as follows: S2-11. Input image, pass through a backbone network consisting of five Conv stages of VGG16 for feature extraction, output dimension is... B represents the batch size, and C represents the number of channels. S2-12. Based on the feature map obtained in step S21, use... Using a sliding window to extract spatial features, the size of the feature map after the sliding window is still [missing information]. However, each pixel is blended into its surroundings. Range information; S2-13. Reshape the feature map obtained in step S22 into... The data is then input into a bidirectional LSTM network to extract the sequence features of each row, thereby obtaining the fusion of image features and sequence features. S2-14. Input the obtained features into the FC fully connected layer to obtain... The output is N, which is the number of samples in the input feature map. S2-15. After passing through the RPN network, it is divided into three prediction branches, one of which outputs... ,in This indicates the prediction for each anchor. , This is the anchor category information. It contains the anchor's size information; the second branch outputs... ,in , representing the score for the foreground and background, denoted as ,in It is the prospect score. This is the background score, the third branch output. ,in , indicates prediction The side-refinement of each anchor, i.e. , The calibration ratio is calculated using the following formula: (2); Indicates the left or right boundary of the text box. The x-coordinate of the anchor center. The anchor has a fixed width of 16 pixels, obtained through the above calculations. This refers to a scaling ratio, which allows for accurate stretching of the result; S2-16. The loss calculation method for the text location detection stage is defined as follows: (3); in To predict the classification loss for whether each anchor contains a text region, cross-entropy loss is used for calculation. The regression loss for the center y-coordinate and height h of each anchor in the text region is calculated using the Smooth L1 loss function; The regression loss for the center x-coordinates of the anchors on both sides of the text region is calculated using the Smooth L1 loss function. , , It is the impact factor.

[0010] Preferably, the specific method for text content recognition in step S2 is as follows: S2-21. Obtain the input image and scale it to a uniform size; S2-22. The image is input into the convolutional layer for feature extraction to obtain a feature map. The convolutional layer is a modification of the VGG network, with the MaxPooling kernel size of the second and third layers increased from... Changed to The CNN layer ultimately outputs a feature map; S2-23. Input consists of a recurrent layer composed of a bidirectional recurrent neural network (RNN), and the feature sequence obtained from the convolutional layer... Each component in Predict a label distribution ; S2-24. Input the transcription layer, select the symbol corresponding to the index with the highest probability in each component of the predicted sequence as the recognition result, and finally form the sequence as the final recognition sequence to obtain the recognition result.

[0011] Preferably, the specific method for semantic entity recognition in step S3 is as follows: The image, text, and location information obtained in step S2 are input into the LayoutXLM model. First, a semantic entity recognition task is performed. An encoder is used to concatenate the image vector sequence and the text vector sequence into a unified input sequence, and a layout vector is added at the corresponding position. Then, the fused three modal information is input into a Transformer network with a self-attention mechanism to perform semantic entity recognition and classification. Semantic entities are divided into QUESTION and ANSWER. The format of the three modal vector sequences is as follows: (4); in For image information, For image feature vectors, It is a one-dimensional position vector. This refers to the segmentation information of the entire text. The three segments are summed to obtain the final image modality vector. ; (5) Among them Word segmentation information, For word feature vectors, It is a one-dimensional position vector. For each word, segmented information. The words are segmented into vectors, and the three vectors are added together to obtain the text modality vector. ; (6); in and For two-dimensional location information, use the coordinates of the top left corner of the bounding box. and the coordinates of the bottom right corner And width and height, that is and They collectively represent a bounding box, and are ultimately summed to obtain a vector of the layout modalities. .

[0012] Preferably, the specific method for step S4, the relationship extraction stage, is as follows: S41. Using the LayoutXLM text understanding model for relation extraction, the results of semantic entity recognition are processed to extract key information needed by the user from the ticket; the key information extraction process is as follows: the text set to be processed... With entity category The relation extraction task involves connecting a dual affine attention classifier to the LayoutXLM network backbone to predict whether a relation exists between two entities, and further, what kind of relation exists. If the relation is a category, then the specific process of the relation extraction task is defined as follows: ,in ,in For entities that have a relationship, Define the relation classes that exist between them; stipulate that the set R contains only one relation class. The goal of LayoutXLM prediction is to determine whether two entities satisfy a relation pair in R, thus forming a match. right, , It is a set The two categories defined in the document; S42. Return the extracted key information to the front end in the format required by the user.

[0013] A document recognition system for RPA office workflow automation systems includes a document receiving module, a text detection and text recognition module, and a semantic entity recognition module. The invoice file receiving module is used to receive invoice files to be identified uploaded by the front end. If the file is an image file in PNG, JPG, JPEG, GIF, or BMP format, the system will directly save it temporarily. If the file is a PDF file, the PDF file will be converted into an image first, and then it will be temporarily saved to the system. Text detection and text recognition module: Uses CTPN model to detect text location and CRNN to recognize text content; Semantic Entity Recognition Module: Based on the results of text detection and recognition, the Semantic Entity Recognition (SER) task of the LayoutXLM text understanding model is used to further identify and classify text entities, dividing them into question entities (QUESTION) and answer entities (ANSWER). Relation Extraction Phase Module: Using the LayoutXLM text understanding model, the relation extraction (RE) task scenario predicts whether there is a relationship between text entities in the question and answer classes, performs matching, and finally returns the result to the front-end invoice recognition.

[0014] Compared with the prior art, the beneficial effects of this application are as follows: (1) This invention introduces the text detection model CTPN and the text recognition model CRNN to initially detect the text position and content in the invoice file.

[0015] (2) This invention introduces the multimodal text understanding model LayoutXLM, which comprehensively utilizes information from three modalities: image, text, and layout, to further identify text entities and extract relationships in the document to be tested, thereby extracting key information from the document.

[0016] (3) This invention combines the technical features of the above two models and realizes the automation of the ticket recognition process through the concatenation prediction of OCR+SER+RE.

[0017] (4) This invention provides users with a convenient function to upload invoice image files through the front-end RPA system page, and can return the results to the user in a timely and clear manner. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0019] Figure 1 This is a schematic diagram of an overall method for document recognition in an RPA office workflow automation system disclosed in an embodiment of the present invention.

[0020] Figure 2 This is a diagram of the CTPN model architecture used in the text detection stage of this invention.

[0021] Figure 3 This is a diagram of the CRNN model architecture used in the text recognition stage of this invention.

[0022] Figure 4 This is a diagram of the LayoutXLM model architecture used in the semantic entity recognition and relation extraction stage of this invention. Detailed Implementation

[0023] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0024] This invention provides a document recognition method for RPA office workflow automation systems, such as... Figure 1 As shown, this method can combine the technical characteristics of the two models to achieve cascaded prediction of OCR engine + SER + RE, thereby extracting key information from the invoice.

[0025] The specific implementation method is as follows: A method for document recognition in an RPA office workflow automation system includes the following steps: (1) Preparation stage: (1.1) Select the invoice recognition function module through the front-end RPA system; (1.2) Write a graphical or scripting language to design an automated process and upload images or PDFs of the invoices to be processed; (1.3) Run the pre-written process, encapsulate the request ticket file and send the corresponding request to the background RPA server; (1.4) If the image file is in PNG, JPG, JPEG, GIF, or BMP format, the system will temporarily store it for later processing. If the document is in PDF format, the PDF file will be converted into an image first, and then temporarily stored in the system. The specific method is as follows: First, each page of the PDF file is read and converted into a pixel bitmap on a page-by-page basis. Then, the resolution and color space are specified and the file is rendered as a PIL bitmap for later processing.

[0026] (2) Text detection and text recognition stage: (2.1) Image preprocessing: Due to the inconsistent shooting conditions of ticket images, a series of problems such as blurring and tilting may occur, which will affect the accuracy of text detection and recognition. Therefore, image preprocessing is required, mainly implemented through the Python OpenCV library, including the following steps: Background Removal: 1) First, a background subtractor is obtained using the `cv2.createBackgroundSubtractorMOG()` function to separate the foreground and background of the image, obtaining a binary image of the foreground portion. 2) Using the morphological function `cv2.morphologyEx()` with the `cv2.MORPH_CLOSE` parameter set, a closing operation is performed on the image to clearly distinguish the boundary between the foreground and background. 3) The processed foreground and background images are overlaid to remove background parts unrelated to the text, making the text stand out more and aiding in subsequent localization and recognition.

[0027] Tilt Correction: 1) First, read the image in grayscale format and perform closing and opening operations on the image using the morphological function `cv2.morphologyEx()`. 2) Sort the image contours using `cv2.findContours` to obtain the bounding box of the largest contour, and obtain the four vertices through the largest bounding box. 3) Using the vertex information, generate a perspective transformation matrix using the `cv2.getPerspectiveTransform()` function and perform perspective transformation using the `cv2.warpPerspective()` function to complete the tilt correction of the image, restoring the text lines to horizontal, facilitating accurate recognition in the subsequent process.

[0028] Image enhancement: 1) First, calculate the gray-level histogram of the image using the `hist` function in the Matplotlib library, and then calculate the cumulative histogram of the image. 2) Define the mapping relationship between the input gray levels and the output gray levels as follows: (1); in For input pixels, To output pixels, The height of the image. Given the image width, the above mapping relationship is equivalent to first obtaining the cumulative probability histogram of the grayscale histogram (i.e., ...). ), and then expand this range to between 0-255 to obtain the pixels of the output image. 3) Circularly output the grayscale level of each pixel in the image according to the mapping relationship. Through the above steps, histogram equalization is achieved, which solves the problem of low contrast caused by the small grayscale range of the image, making the text clearer, thereby further improving the recognition effect.

[0029] (2.2) Text location detection: The CTPN model is used for text location detection. 1) Input image: Features are extracted through a backbone network consisting of five Conv stages of VGG16. The output dimension is... ( 2) Based on the feature map obtained in step 1), use... Using a sliding window to extract spatial features, the size of the feature map after the sliding window is still [missing information]. However, each pixel is blended into its surroundings. 3) Reshape the feature map obtained in step 2) into... The data is then fed into a bidirectional LSTM (BI-LSTM) network, which fully utilizes contextual information to extract sequence features from each row, thereby achieving a fusion of image features and sequence features. 4) Input the number of samples in the input feature map. 5) Finally, it passes through a network similar to RPN, where it splits into three prediction branches, as shown in the appendix. Figure 2 As shown, each pixel corresponds to There are anchors, and one branch outputs... ,in This indicates the prediction for each anchor. , This is the anchor category information. It contains the anchor's size information; the second branch outputs... ,in , representing the score for the foreground and background, denoted as ,in It is the prospect score. This is the background score, the third branch output. ,in , indicates prediction The side-refinement of each anchor, i.e. Since the width of each anchor is fixed, inconsistencies may occur in the horizontal direction, requiring calibration. This refers to the calibration ratio, calculated using the following formula: (2); Indicates the left or right boundary of the text box. The x-coordinate of the anchor center. The anchor has a fixed width of 16 pixels, obtained through the above calculations. This refers to a scaling ratio, which allows for accurate stretching of the result. This is a marker for anchors and has no actual meaning.

[0030] The loss calculation method in the text location detection stage is defined as follows, where To predict the classification loss for whether each anchor contains a text region, cross-entropy loss is used for calculation. The regression loss for the center y-coordinate and height h of each anchor in the text region is calculated using the Smooth L1 loss function; The regression loss for the center x-coordinates of the anchors on both sides of the text region is calculated using the Smooth L1 loss function; , , The influencing factors for the above three types of losses are: (3).

[0031] (2.3) Text Content Recognition: The CRNN model is used to recognize text content. The CRNN model mainly consists of three parts: convolutional layers, recurrent layers, and transcription layers. The steps of the CRNN model for text recognition are as follows: 1) Obtain the input image and scale it to a uniform size. 2) Input the image into the convolutional layer for feature extraction to obtain the feature map. The convolutional layer is modified based on the VGG network, and the size of the MaxPooling convolutional kernel in the second and third layers is reduced from... Changed to The final output feature map size of the CNN layer is (512, 1, 40), meaning it outputs 512 features. 3) Input consists of a recurrent layer composed of a bidirectional recurrent neural network (RNN), and the feature sequence obtained from the convolutional layer. Each component in Predict a label distribution Ordinary RNNs suffer from vanishing and exploding gradients. Therefore, a Bidirectional Long Short-Term Memory (Bi-LSTM) network is used instead of a traditional RNN to fully capture contextual information and improve training efficiency. 4) The transcription layer is input, and the symbol corresponding to the index with the highest probability in each component of the predicted sequence is selected as the recognition result. The final sequence is then used as the final recognition sequence to obtain the recognition result. The architecture of the text recognition model CRNN is shown in the attached figure. Figure 3 As shown.

[0032] (3) Semantic entity recognition stage: (3.1) Modify the default training parameters and train the LayoutXLM text understanding model with sufficient ticket fields to ensure that the model has the ability to classify text entities into question type QUESTION and answer type ANSWER.

[0033] (3.2) Input the image, text, and location information obtained in step (2) into the LayoutXLM model. LayoutXLM is a multimodal text understanding model that can simultaneously accept information from three modalities—image, text, and layout—during the input stage and utilize deep networks to achieve deep fusion of multimodal information. First, perform the semantic entity recognition task. Use an encoder to concatenate the image vector sequence and text vector sequence into a unified input sequence and add layout vectors at corresponding positions. Then, input the fused information from the three modalities into a Transformer network with a self-attention mechanism to perform semantic entity recognition and classification, dividing semantic entities into QUESTION and ANSWER classes. The format of the three modal vector sequences is as follows: Image modality vector : (4); in For image information, For image feature vectors, It is a one-dimensional position vector. This refers to the segmentation information of the entire text. The three segments are summed to obtain the final image modality vector. .

[0034] Vector of text modality : (5); Among them Word segmentation information, For word feature vectors, It is a one-dimensional position vector. For each word, segmented information. The words are segmented into vectors, and the three vectors are added together to obtain the text modality vector. .

[0035] Layout modal vectors : (6); in and For two-dimensional location information, use the coordinates of the top left corner of the bounding box. and the coordinates of the bottom right corner And width and height, that is and They collectively represent a bounding box, and are ultimately summed to obtain a vector of the layout modalities. .

[0036] (4) Relationship extraction stage: (4.1) The relation extraction task scenario using the LayoutXLM text understanding model is used to process the results of semantic entity recognition, thereby extracting the key information required by the user from the ticket. The general process of key information extraction is as follows: the text set to be processed With entity category The relation extraction task involves connecting a dual affine attention classifier to the LayoutXLM network backbone to predict whether a relation exists between two entities, and further, what kind of relation exists. If the set is a collection of relation categories, then the specific process principle of the relation extraction task can be defined as follows: ,in ,in For entities that have a relationship, Let R be the class of relationships that exist between them. At this point, we stipulate that set R contains only one class of relationships. The model predicts whether two entities satisfy the relation pair in R, forming a match. right, , It is a set The two categories defined in [the document / document].

[0037] For example, the QUESTION entity "Invoice Amount" can be matched with the ANSWER entity "¥500".

[0038] (4.2) Return the key information extracted in the above steps to the front end in the format required by the user.

[0039] This method combines the text detection and recognition capabilities of the OCR engine with the text understanding capabilities of the LayoutXLM model. Through the OCR engine + SER + RE concatenated prediction method, it ultimately achieves the tasks of ticket recognition and key information extraction.

[0040] A document recognition system for RPA office workflow automation systems includes a document receiving module, a text detection and text recognition module, and a semantic entity recognition module. The invoice file receiving module is used to receive invoice files to be identified uploaded by the front end. If the file is an image file in PNG, JPG, JPEG, GIF, or BMP format, the system will directly save it temporarily. If the file is a PDF file, the PDF file will be converted into an image first, and then it will be temporarily saved to the system. Text detection and text recognition module: Uses CTPN model to detect text location and CRNN to recognize text content; Semantic Entity Recognition Module: Based on the results of text detection and recognition, the Semantic Entity Recognition (SER) task of the LayoutXLM text understanding model is used to further identify and classify text entities, dividing them into question entities (QUESTION) and answer entities (ANSWER). Relation Extraction Phase Module: Using the LayoutXLM text understanding model, the relation extraction (RE) task scenario predicts whether there is a relationship between text entities in the question and answer classes, performs matching, and finally returns the result to the front-end invoice recognition.

[0041] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for document recognition in an RPA office workflow automation system, characterized in that, Includes the following steps: S1. Preparation phase: The RPA system receives the document file to be identified uploaded by the front end and processes it; S2. Text detection and text recognition stage: The CTPN model is used to detect text locations, and the CRNN model is used to recognize text content; S3. Semantic Entity Recognition Stage: Based on the results of text detection and recognition in step S2, the semantic entity recognition task scenario SER of the LayoutXLM text understanding model is used to further identify and classify the text entities, dividing the text entities into question type QUESTION and answer type ANSWER. The image, text, and location information obtained in step S2 are input into the LayoutXLM model. First, a semantic entity recognition task is performed. An encoder concatenates the image vector sequence and text vector sequence into a unified input sequence, and adds layout vectors at corresponding positions. Then, the fused three modal information is input into a Transformer network with a self-attention mechanism for semantic entity recognition and classification. Semantic entities are categorized into QUESTION and ANSWER. The format of the three modal vector sequences is as follows: （4）； in For image information, For image feature vectors, It is a one-dimensional position vector. This refers to the segmentation information of the entire text. The three segments are summed to obtain the final image modality vector. ; （5） Among them Word segmentation information, For word feature vectors, It is a one-dimensional position vector. For each word, segmented information. The words are segmented into vectors, and the three vectors are added together to obtain the text modality vector. ; （6）； in and For two-dimensional location information, use the coordinates of the top left corner of the bounding box. and the coordinates of the bottom right corner And width and height, that is and They collectively represent a bounding box, and are ultimately summed to obtain a vector of the layout modalities. ; S4. Relationship Extraction Stage: Using the LayoutXLM text understanding model, the relationship extraction task (RE) is used to predict whether there is a relationship between text entities in the question class and the answer class, and to perform matching. Finally, the results are returned to the front-end ticket recognition. S41. Using the LayoutXLM text understanding model for relation extraction, the results of semantic entity recognition are processed to extract key information needed by the user from the ticket; the key information extraction process is as follows: the text set to be processed... With entity category The relation extraction task involves connecting a dual affine attention classifier to the LayoutXLM network backbone to predict whether a relation exists between two entities, and further, what kind of relation exists. If the relation is a category, then the specific process of the relation extraction task is defined as follows: ,in ,in For entities that have a relationship, Define the relation classes that exist between them; stipulate that the set R contains only one relation class. The goal of LayoutXLM prediction is to determine whether two entities satisfy a relation pair in R, thus forming a match. right, , It is a set The two categories defined in the document; S42. Return the extracted key information to the front end in the format required by the user.

2. The document recognition method for an RPA office workflow automation system according to claim 1, characterized in that, In step S1, S11. Receive the invoice image or PDF to be processed; S12. Run the set process, encapsulate the request ticket file and send the corresponding request to the background RPA server; S13. If the image file is in png, jpg, jpeg, gif, or bmp format, the system will temporarily save it. If the document is in PDF format, first convert the PDF file to an image, and then temporarily save it to the system.

3. The document recognition method for an RPA office workflow automation system according to claim 2, characterized in that, In step S13, the specific method for converting a PDF file into an image is as follows: First, each page of the PDF file is read and converted into a pixel bitmap on a page-by-page basis. Then, the resolution and color space are specified and the image is rendered as a PIL bitmap for easier subsequent processing.

4. The document recognition method for an RPA office workflow automation system according to claim 2, characterized in that, Step S2 includes image preprocessing, including background removal, tilt correction, and image enhancement; The steps to remove the background are as follows: 1) First, a background subtractor is obtained through the cv2.createBackgroundSubtractorMOG() function to separate the foreground and background of the image and obtain a binary image of the foreground part; 2) Using the morphological function cv2.morphologyEx(), with the cv2.MORPH_CLOSE parameter set, a closing operation is performed on the image to clearly separate the boundary between the foreground and background; 3) Overlay the processed foreground and background images to remove background parts unrelated to the text; The tilt correction steps are as follows: 1) Read the image in grayscale form and use the morphological function cv2.morphologyEx() to perform closing and opening operations on the image; 2) Use cv2.findContours to sort the image contours, obtain the bounding box of the largest contour, and obtain the four vertices through the largest bounding box; 3) Using vertex information, generate a perspective transformation matrix using the cv2.getPerspectiveTransform() function and perform perspective transformation using the cv2.warpPerspective() function to correct the image tilt and restore the text lines to horizontal. The image enhancement steps are as follows: 1) Calculate the grayscale histogram of the image using the hist function in the Matplotlib library, and calculate the cumulative histogram of the image; 2) The mapping relationship between input gray levels and output gray levels is defined as follows: （1）； in For input pixels, To output pixels, The height of the image. The width of the image; 3) Output the gray level of each pixel in the image in a loop according to the mapping relationship.

5. The document recognition method for an RPA office workflow automation system according to claim 1, characterized in that, In step S2, the text position detection steps are as follows: S2-11. Input image, pass through a backbone network consisting of five Conv stages of VGG16 for feature extraction, output dimension is... B represents the batch size, and C represents the number of channels. S2-12. Based on the feature map obtained in step S21, use... Using a sliding window to extract spatial features, the size of the feature map after the sliding window is still [missing information]. However, each pixel is blended into its surroundings. Range information; S2-13. Reshape the feature map obtained in step S22 into... The data is then input into a bidirectional LSTM network to extract the sequence features of each row, thereby obtaining the fusion of image features and sequence features. S2-14. Input the obtained features into the FC fully connected layer to obtain... The output is N, which is the number of samples in the input feature map. S2-15. After passing through the RPN network, it is divided into three prediction branches, one of which outputs... ,in This indicates the prediction for each anchor. , This is the anchor category information. It contains the anchor's size information; the second branch outputs... ,in , representing the score for the foreground and background, denoted as ,in It is the prospect score. This is the background score, the third branch output. ,in , indicates prediction The side-refinement of each anchor, i.e. , The calibration ratio is calculated using the following formula: （2）； Indicates the left or right boundary of the text box. The x-coordinate of the anchor center. The anchor has a fixed width of 16 pixels, obtained through the above calculations. This refers to a scaling ratio, which allows for accurate stretching of the result; S2-16. The loss calculation method for the text location detection stage is defined as follows: （3）； in To predict the classification loss for whether each anchor contains a text region, cross-entropy loss is used for calculation. The regression loss for the center y-coordinate and height h of each anchor in the text region is calculated using the Smooth L1 loss function; The regression loss for the center x-coordinates of the anchors on both sides of the text region is calculated using the Smooth L1 loss function. , , It is the impact factor.

6. The document recognition method for an RPA office workflow automation system according to claim 1, characterized in that, The specific method for text content recognition in step S2 is as follows: S2-21. Obtain the input image and scale it to a uniform size; S2-22. The image is input into the convolutional layer for feature extraction to obtain a feature map. The convolutional layer is a modification of the VGG network, with the MaxPooling kernel size of the second and third layers increased from... Changed to The CNN layer ultimately outputs a feature map; S2-23. Input consists of a recurrent layer composed of a bidirectional recurrent neural network (RNN), and the feature sequence obtained from the convolutional layer... Each component in Predict a label distribution ; S2-24. Input the transcription layer, select the symbol corresponding to the index with the highest probability in each component of the predicted sequence as the recognition result, and finally form the sequence as the final recognition sequence to obtain the recognition result.

7. A document recognition system for RPA office workflow automation systems, adapted to the method described in any one of claims 1-6, characterized in that, This includes modules for receiving invoices and documents, text detection and recognition, and semantic entity recognition. The invoice file receiving module is used to receive the invoice file to be identified uploaded by the front end. If it is an image file in png, jpg, jpeg, gif, or bmp format, the system will directly save it temporarily. If the document is in PDF format, first convert the PDF file to an image, and then temporarily save it to the system. Text detection and text recognition module: Uses CTPN model to detect text location and CRNN to recognize text content; Semantic Entity Recognition Module: Based on the results of text detection and recognition, the Semantic Entity Recognition (SER) task of the LayoutXLM text understanding model is used to further identify and classify text entities, dividing them into question entities (QUESTION) and answer entities (ANSWER). Relation Extraction Phase Module: Using the LayoutXLM text understanding model, the relation extraction (RE) task scenario predicts whether there is a relationship between text entities in the question and answer classes, performs matching, and finally returns the result to the front-end invoice recognition.

Images