Text recognition method, device, electronic device and storage medium combining RPA and AI

Abstract

This application proposes a text recognition method, device, electronic device and storage medium combining RPA and AI, which relates to the field of artificial intelligence technology. The specific disclosed technical solution is as follows: the AI ​​platform receives the image sent by the RPA robot, and the AI ​​platform obtains the table area image from the image; the AI ​​platform extracts each cell from the table area image; the AI ​​platform constructs a table according to the position information of each cell; the AI ​​platform extracts the text information in each cell from the image, and generates a text recognition result based on the text information and the table and sends it to the RPA robot. Thus, the technical problems of poor accuracy and low efficiency of text recognition in the prior art are solved. By first constructing a table in the image and then extracting the text information in the table from the image, the text recognition in the image can be completed quickly and accurately.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a text recognition method, device, electronic device and storage medium combining RPA and AI. Background Art

[0002] Robotic Process Automation (RPA), referred to as Robotic Process Automation, uses specific "robot software" to simulate human operations on computers and automatically execute process tasks according to rules.

[0003] Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. Artificial Intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes machine learning, speech recognition, image recognition, natural language processing and expert systems.

[0004] Generally, banks have a wide range of businesses and there are various bills with various formats. Tables, as structured information in the image content, often appear in various bank bill images. In related technologies, text detection technology is used to extract the text position, and then a heuristic method is used to infer the table information based on the text position. However, this method has technical problems such as poor accuracy and low efficiency. Summary of the invention

[0005] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0006] To this end, this application proposes a text recognition method that combines RPA and AI, which solves the technical problems of poor accuracy and low efficiency of text recognition methods in the prior art. By first constructing a table in the image and then extracting the text information in the table from the image, the text recognition in the image can be completed quickly and accurately.

[0007] This application proposes an AI platform.

[0008] This application proposes a text recognition system that combines RPA and AI.

[0009] The present application provides an electronic device.

[0010] The present application proposes a computer-readable storage medium.

[0011] In one aspect, an embodiment of the present application proposes a text recognition method combining RPA and AI, which is applied to an AI platform and includes:

[0012] The AI ​​platform receives the image sent by the RPA robot and obtains the table area image from the image;

[0013] The AI ​​platform extracts individual cells from the table area image;

[0014] The AI ​​platform constructs a table based on the location information of each cell;

[0015] The AI ​​platform extracts the text information in each cell from the image, generates text recognition results based on the text information and table, and sends them to the RPA robot.

[0016] Another aspect of the present application provides an AI platform, including:

[0017] A first acquisition module, used for acquiring an image;

[0018] A second acquisition module is used to acquire a table area image from the image;

[0019] An extraction module, used to extract individual cells from a table area image;

[0020] A construction module, used to construct a table according to the position information of each cell;

[0021] The generation module is used to extract the text information in each cell from the image and generate the text recognition result according to the text information and the table.

[0022] In another aspect, the present application provides a text recognition system combining RPA and AI, which is characterized by comprising: the AI ​​platform, business system and RPA robot described in the above-mentioned aspect;

[0023] The business system obtains images through camera capture or scanner scanning and sends them to the RPA robot.

[0024] The RPA robot sends the image to the AI ​​platform for processing, generates text recognition results and returns them to the RPA robot;

[0025] The RPA robot sends the text recognition results to the business system’s database for storage.

[0026] In yet another aspect of the present application, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the text recognition method combining RPA and AI as described in the aforementioned aspect of the embodiment is implemented.

[0027] In another aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the text recognition method combining RPA and AI as described in the aforementioned method embodiment is implemented.

[0028] The technical solution provided in the embodiments of the present application may have the following beneficial effects:

[0029] The AI ​​platform receives the image sent by the RPA robot, obtains the table area image from the image, extracts each cell from the table area image, constructs a table based on the location information of each cell, extracts the text information in each cell from the image, and generates a text recognition result based on the text information and the table and sends it to the RPA robot. This solves the technical problems of poor accuracy and low efficiency of text recognition in the prior art. By first constructing a table in the image and then extracting the text information in the table from the image, text recognition in the image can be completed quickly and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0031] Figure 1 A structural diagram of a text recognition system combining RPA and AI provided in an embodiment of the present application;

[0032] Figure 2 A flowchart of a text recognition method combining RPA and AI provided in an embodiment of the present application;

[0033] Figure 3 A flowchart of another text recognition method combining RPA and AI provided in an embodiment of the present application;

[0034] Figure 4 An example diagram of a text recognition system combining RPA and AI provided in an embodiment of the present application;

[0035] Figure 5a This is an example diagram of an image received by the RPA provided in an embodiment of the present application;

[0036] Figure 5b An example diagram of a table area image provided in an embodiment of the present application;

[0037] Figure 5c An example diagram of each cell provided in the embodiments of the present application;

[0038] Figure 5d An example diagram of table adjustment provided in an embodiment of the present application;

[0039] Figure 5e An example diagram of table adjustment provided in an embodiment of the present application;

[0040] Figure 5f An example diagram of table adjustment provided in an embodiment of the present application;

[0041] Figure 6 A structural diagram of a text recognition device combining RPA and AI provided in an embodiment of the present application;

[0042] Figure 7 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0043] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0044] The following describes the text recognition method, device, electronic device and storage medium combining RPA and AI in an embodiment of the present application with reference to the accompanying drawings.

[0045] Specifically, this application proposes a text recognition method that combines RPA and AI. RPA receives images and sends them to the AI ​​platform. The AI ​​platform first constructs a table in the image and then extracts the text information in the table from the image. It can quickly and accurately complete the text recognition in the image and then feed back to the RPA robot. The combination of RPA technology and OCR technology can help users extract table structured information in images more efficiently, greatly improving work efficiency and data accuracy.

[0046] In actual applications, the images received by the RPA robot come from many different businesses, most of which are open images, that is, there may be tables or no tables in the images, and the position, scale, shape, color, background, and format of the tables in the images are also different. The text recognition method combining RPA and AI proposed in this application can process the open images described above, receive the image sent by the RPA robot, obtain the table area image from the image, then extract each cell from the table area image, build a table according to the position information of each cell, extract the text information in each cell from the image, generate a text recognition result based on the text information and the table, and send it to the RPA robot. In addition, if the table area image cannot be obtained from the image, text recognition is performed directly, which can quickly and accurately complete the text recognition in the image.

[0047] In order to make the text recognition method combining RPA and AI described in the embodiment of the present application more clear to those skilled in the art, firstly, Figure 1 Describe a text recognition system that combines RPA and AI.

[0048] Figure 1 A structural diagram of a text recognition system combining RPA and AI provided in an embodiment of the present application.

[0049] like Figure 1 As shown, the system includes: a business system 100, an RPA robot 200 and an AI platform 300.

[0050] The business system 100 obtains an image by photographing it with a camera, or the business system 100 obtains an image by scanning it with a scanner and sends it to the RPA robot 200.

[0051] The RPA robot 200 sends the image to the AI ​​platform 300 for processing, generates a text recognition result and returns it to the RPA robot 200.

[0052] The RPA robot 200 sends the text recognition results to the database of the business system 100 for storage.

[0053] In the embodiment of the present application, the RPA robot 200 can provide professional and comprehensive process automation solutions for enterprises and individuals. The RPA robot can intelligently understand the existing applications of the enterprise through the user interface, and automate routine operations based on rules, such as automatically and repeatedly reading emails, reading office software components, operating databases, web pages and client software, etc., collecting data, performing tedious calculations, generating files and reports in large quantities, and completing boring file checks, etc. It can greatly reduce the investment in labor costs, effectively improve the existing office efficiency, and complete work accurately, stably and quickly.

[0054] As an example of a scenario, such as the digitization of bank bills, the RPA robot 200 receives various bill images collected by the business system 100, converts them into corresponding digital information through the AI ​​platform 300, and transcribes them to the corresponding databases to achieve digitization and de-humanization.

[0055] Among them, the bill image received by the RPA robot 200 is usually open data, coming from different application scenarios. This application implements a robust table recognition system by secondary extraction of table information in the image, and integrates the table recognition system with RPA and OCR (Optical Character Recognition) technology, so that the RPA robot 200 can process structured tables in various types of bank bill images, expand the capabilities of the RPA robot 200, and meet the needs of bank bill digitization. In addition, the RPA robot 200 sends the recognized structured data to the business system (such as the financial system) to realize the automatic entry of bank bills and improve the efficiency of bank bill processing.

[0056] In order to help those in this field understand the specific process of how the AI ​​platform performs text recognition, the following is a Figure 2 The text recognition method combining RPA and AI is described from the AI ​​platform side.

[0057] Figure 2 A flowchart of a text recognition method combining RPA and AI provided in an embodiment of the present application.

[0058] like Figure 2 As shown, the method comprises the following steps:

[0059] Step 101: The AI ​​platform receives an image sent by the RPA robot and obtains a table area image from the image.

[0060] In an embodiment of the present application, the RPA robot can send one or more received images to the AI ​​platform for text recognition processing according to the specific processing scenario, wherein the images received by the RPA robot are generally images of documents such as bills processed by the business system for the same or different business processes taken by a camera or scanned by a scanner.

[0061] In the embodiment of the present application, the table area image refers to an image that only includes the table portion captured from the received entire image.

[0062] In the embodiment of the present application, after receiving the image sent by the RPA robot, the AI ​​platform obtains the table area image from the image in many ways, as illustrated below:

[0063] In the first example, the AI ​​platform processes the image through an object detection algorithm to obtain multiple first arrays, determines the table category according to the category identifiers in the multiple first arrays, obtains the target array corresponding to the table category, and determines the table area image according to each value in the target array.

[0064] In the embodiment of the present application, the first array refers to a matrix that can determine a rectangular area and the category of objects included in the rectangular area; the target array refers to a matrix in which the category of objects included in the rectangular area is a table.

[0065] In the second example, the AI ​​platform uses OpenCV (a cross-platform computer vision and machine learning software library) to identify the straight lines in the table, determines the table area based on the straight lines, and directly captures the table area image from the image.

[0066] It should be noted that if the table area image cannot be obtained from the image, the text information extraction operation can be directly performed on the image, thereby improving the processing efficiency.

[0067] Step 102, the AI ​​platform extracts each cell from the table area image.

[0068] In an embodiment of the present application, after the AI ​​platform obtains the table area image, each cell is further extracted from the table area image. There are many ways to extract each cell from the table area image, and examples are as follows:

[0069] In the first example, the AI ​​platform processes the table area image through an instance segmentation detection algorithm to obtain multiple second arrays, and generates each cell according to each value in the multiple second arrays.

[0070] In the second example, the AI ​​platform recognizes the straight lines in the table through OpenCV (a cross-platform computer vision and machine learning software library) and directly generates each cell based on the straight lines.

[0071] Step 103, the AI ​​platform constructs a table based on the location information of each cell.

[0072] In an embodiment of the present application, after the AI ​​platform obtains each cell, the AI ​​platform can reconstruct a table based on each cell. As a possible implementation method, the AI ​​platform determines the minimum cell width and the minimum cell height in each cell based on the position information of each cell, determines the first step length based on the minimum cell width, and determines the second step length based on the minimum cell height, divides the total table area corresponding to each cell in a first direction according to the first step length, and divides the total table area in the first direction according to the second step length, generates a table network area, numbers each cell, maps each cell to the table network area according to the coordinate position, merges the table network area by columns in the first direction, and merges the table network area by rows in the second direction to generate a table.

[0073] In the embodiment of the present application, the first step length refers to the distance between two horizontal cuts of the table area; the second step length refers to the distance between two vertical cuts of the table area; the table network area refers to the table area that has been cut into a grid.

[0074] As another possible implementation method, the AI ​​platform directly calculates and merges the position information of each cell (i.e., each coordinate information) through a cell merging function, and generates a table based on the merged position information.

[0075] In step 104, the AI ​​platform extracts the text information in each cell from the image, generates a text recognition result based on the text information and the table, and sends it to the RPA robot.

[0076] In an embodiment of the present application, the AI ​​platform determines the positions of each cell in the image based on the position coordinate information of the table, extracts text information from each position in the image, and fills the text information into the corresponding cells and sends it to the RPA robot. In other words, the table position and the position of each cell are mapped back to the image sent by the RPA robot, thereby completing the structured recognition of the table information. The structured information of the table includes the overall position of the table in the original image (the pixel positions of the four vertices of the smallest quadrilateral surrounding the table), the total number of rows and columns of the cells in the table, the position of each cell in the table (the pixel positions of the four vertices of the smallest quadrilateral surrounding each cell), the starting and ending row and column numbers of each cell in the table, and finally, the OCR technology is combined to identify the text information in each cell, thereby completing the extraction of the overall table information, and generating a text recognition result based on the text information and the table and sending it to the RPA robot.

[0077] In the text recognition method combining RPA and AI in the embodiment of the present application, the AI ​​platform receives the image sent by the RPA robot, obtains the table area image from the image, extracts each cell from the table area image, constructs a table according to the position information of each cell, extracts the text information in each cell from the image, and generates a text recognition result based on the text information and the table and sends it to the RPA robot. Thus, the technical problems of poor accuracy and low efficiency of text recognition in the prior art are solved. By first constructing a table in the image and then extracting the text information in the table from the image, the text recognition in the image can be completed quickly and accurately.

[0078] Figure 3 A flowchart of another text recognition method combining RPA and AI provided in an embodiment of the present application.

[0079] like Figure 3 As shown, the method comprises the following steps:

[0080] In step 201, the AI ​​platform receives an image sent by the RPA robot. The AI ​​platform processes the image through an object detection algorithm to obtain multiple first arrays, determines the table category according to the category identifiers in the multiple first arrays, obtains the target array corresponding to the table category, and determines the table area image according to each value in the target array.

[0081] In the embodiment of the present application, Figure 4 Taking the text recognition system and bank notes combined with RPA and AI as an example, the business system includes a camera and a scanner, which can take pictures or scan bank notes to generate images, such as Figure 5a As shown in the image, the RPA robot includes a data collection module, an image upload module, a result write-back module, and a result export module. It can collect images generated by the business system, or directly upload images to the RPA robot through the upload module, and then send the images to the AI ​​platform for processing.

[0082] Furthermore, the AI ​​platform can use object detection and segmentation technology on the images received by the RPA robot to implement a table recall model and complete the table position detection (the table position is the pixel positions of the four vertices of the smallest quadrilateral surrounding the table). If the table position cannot be detected, it means that the image does not contain a table, and the text recognition information in the image is directly output, and the table area image is cut out in the image based on the table position.

[0083] In an embodiment of the present application, an image is processed by an object detection algorithm to obtain multiple first arrays, a table category is determined based on category identifiers in the multiple first arrays, a target array corresponding to the table category is obtained, and a table area image is determined based on each value in the target array.

[0084] Specifically, Figure 5b As shown, for a table, four small rectangles are used to represent the four corners of the table, and one large rectangle represents the table. That is, the recalled objects include two categories, the corner class and the table class. The table recall model adopts the object detection algorithm. After the table recall model is executed, there will be a return value, namely the first array, which can be organized as follows: [[xywh cls], [xywh cls]…[xywh cls]].

[0085] Among them, x, y represent a coordinate point in the image, w, h represent the width and height of a rectangle centered on the point, (x, y, w, h) can determine a rectangular area in the image, and cls is the category of the object contained in the rectangular area (corner point class or table class). If there is no cls in the array that is a table category, the figure does not contain a table; if there is cls in the array that is a table category, the figure contains a table, and the preliminary calculation area where the table is located is (x, y, w, h) corresponding to cls.

[0086] Specifically, for each table area, the following method can be used to accurately calculate it, that is, to correct it. For each corner point category, only the center point of the rectangle, that is, (x, y), is taken as Pcorner. For each table category, since it is a rectangle, it contains four vertices, which are set as Ptop_left, Ptop_right, Pdown_left, Pdown_right, that is, Ptop_left = (xw / 2, yh / 2), Ptop_right = (x+w / 2, yh / 2), Pdown_left = (xw / 2, y+h / 2), Pdown_right = (x+w / 2, y+h / 2).

[0087] Therefore, for each corner center point Pcorner, find the vertex of the rectangle closest to it, and for each vertex of the table, find the corner center point Pcorner closest to it. If the nearest vertex of the nearest center point of a vertex is itself, and the distance between the center point and the vertex is less than, for example, 0.3*w and 0.3*h, then replace the coordinates of the vertex with the coordinates of the center point. After the replacement, the table is a quadrilateral, and the area contained in the quadrilateral is the table area.

[0088] It should be noted that the rectangular area of ​​the table category is corrected. If there is no corner point category, the rectangular area returned by the table is the table area. Since images often have transformations such as tilt and distortion, the table area in the image is not necessarily a rectangle. If the rectangular area returned by the table is used to represent the real non-rectangular table area, it is not accurate enough and its vertices need to be corrected.

[0089] It should be noted that the scale range of the table in the image is relatively large, which brings certain difficulties to the positioning of the table area. Using corner points to correct the table area can alleviate the multi-scale problem to a certain extent.

[0090] In step 202, the AI ​​platform processes the table area image through an instance segmentation detection algorithm to obtain multiple second arrays, and generates each cell according to each value in the multiple second arrays.

[0091] In an embodiment of the present application, object detection and segmentation technology is used on the table area image to implement a table detection model to complete the detection of the position of each table cell (the position of the table cell is the pixel position of the four vertices of the smallest quadrilateral surrounding the table cell).

[0092] In the embodiments of the present application, Figure 5c As shown, for a table, each small rectangular box (such as the rectangular box where a1 is located) represents a cell. The table detection model adopts an instance segmentation algorithm. After the table detection model is executed, there will be a return value array, which is organized as follows: [[x1 y1 x2 y2], [x1 y1 x2 y2]…[x1 y1 x2 y2]].

[0093] Among them, x1, y1 represent the upper left corner vertex of the cell rectangle, x2, y2 represent the lower right corner vertex of the cell rectangle, and each [x1 y1 x2 y2] represents a cell, that is, the result returned by the table detection model is a large number of cell areas.

[0094] Step 203, detecting whether there is any overlapping of cell areas among the cells, and / or detecting whether the blank area between two cells is smaller than the area of ​​one cell, and / or detecting whether there is any unidentified cell among the cells.

[0095] Step 204: If there is overlapping of cell areas, and / or the area of ​​the blank area between two cells is smaller than the area of ​​one cell, the AI ​​platform adjusts the target cell according to the principle of minimum adjustment times, and / or if there are unidentified cells, the AI ​​platform completes the unidentified cells with maximized cells that meet the alignment principle until there are no unidentified cells.

[0096] In the embodiment of the present application, the alignment principle refers to alignment between upper and lower cells, and between left and right cells.

[0097] In an embodiment of the present application, the cell edges in the table are deduplicated / completed, and the cleaning adopts the minimization principle, that is, the edge duplicate areas and edge blank areas of the table cells are minimized to ensure the consistency of the horizontal and vertical positions of the table cell structure; the unrecognized cell areas in the table are completed, and the completion adopts the maximization principle, that is, the continuous and regular table areas with unrecognized cells are regarded as a single cell and their information is completed.

[0098] Specifically, Figure 5d As shown, in the cell results returned by the table detection, there will be Figure 5dIn the situation shown, the areas of cell 1 and cell 2 overlap, and the area of ​​cell 3 is incomplete, where the overlapping area can be understood as two adjacent cells with overlapping areas; the incomplete area can be understood as two adjacent cells with a blank area in between, and the width / height of the blank area is smaller than the smallest width / height of all cells in the table, that is, the blank area between the two cells is less than one cell, and the blank area is regarded as the omitted edge area of ​​a cell.

[0099] In the embodiment of the present application, the cell to the left of the cell to be adjusted and the cell above it are already aligned, and generally the minimum number of adjustments to the cells is used to make all detected cells aligned, for example Figure 5d As shown, if you want to adjust Table 1 and Table 2 so that they no longer overlap, in the first example, the right border of cell 1 can be retracted to no longer overlap with cell 2, but after this adjustment, cell 1 is still not aligned with cell 4 above it and cell 6 below it. In order to follow the alignment principle, cells 4 and 6 need to be aligned with cell 1, and cells 5 and 7 need to be strictly aligned with cell 2, which requires 5 adjustments in total; in the second example, the left border of cell 2 is retracted to no longer overlap with cell 1. A total of 5 adjustments are required, and the principle is the same as above; in the third example, cell 1 is adjusted to align with cell 4, and cell 2 is adjusted to align with cell 5. A total of 2 adjustments are required, and the third example is preferred for adjustment.

[0100] Specifically, Figure 5e As shown in the figure, the thick lines and unindicated areas are the detected cell areas, and the thin lines indicate the unrecognized cell areas. The maximized cells that meet the alignment principle are completed in the unrecognized areas until there are no unrecognized areas. Figure 5e As shown, complete the blue area into one cell; Figure 5f If the 2 shown is completed into one cell, the alignment principle is not met, so area 1 is first completed into one cell, area 2 is completed into one cell, and then area 3 is completed into one cell.

[0101] In step 205, the AI ​​platform determines the minimum cell width and the minimum cell height in each cell according to the position information of each cell, determines the first step length according to the minimum cell width, and determines the second step length according to the minimum cell height, divides the total table area corresponding to each cell in the first direction according to the first step length, and divides the total table area in the first direction according to the second step length to generate a table network area.

[0102] Step 206, the AI ​​platform numbers each cell, maps each cell to a table network area according to the coordinate position, merges the columns of the table network area in a first direction, and merges the rows of the table network area in a second direction to generate a table.

[0103] In an embodiment of the present application, based on the position of each table cell, the reconstruction of the table structure is completed, that is, the total number of rows and columns of cells in the table, the starting and ending row and column numbers of each cell are inferred, and the table position and the position of each cell are mapped back to the image received by the RPA robot, that is, the structured recognition of the table information is completed. The structured information of the table includes the overall position of the table in the original image (the pixel positions of the four vertices of the smallest quadrilateral surrounding the table), the total number of rows and columns of cells in the table, the position of each cell of the table (the pixel positions of the four vertices of the smallest quadrilateral surrounding each cell), and the starting and ending row and column numbers of each cell of the table. Combined with OCR technology, the text information in each cell is identified, and the extraction of the overall table information is completed.

[0104] In an embodiment of the present application, each cell is traversed to find the minimum cell width w and the minimum height h. For example, let w = max(1, w / 2), h = max(1, h / 2), and divide the table from left to right according to the step size w; divide the table from top to bottom according to the step size h. The divided table area is in a grid shape, and each grid is assigned a default value of -1.

[0105] Furthermore, each cell is numbered, that is, a number value is assigned to each cell, and each cell is mapped to the grid area according to the coordinate position, and the value assigned to the grid whose covered area exceeds 1 / 2 is changed to the value of the cell number.

[0106] In an embodiment of the present application, two adjacent columns are extracted from the table network area from left to right, and if the cell values ​​corresponding to the two columns are equal, either column is deleted; two adjacent rows are extracted from the table network area from top to bottom, and if the cell values ​​corresponding to the two rows are equal, either row is deleted.

[0107] Furthermore, from left to right, the columns of the table network area are merged. From left to right, two adjacent columns are extracted from the table network area and set as e[left] and e[right]. Let len ​​be the length of the column. For each i∈[0, len-1], if the following conditions are met, the two columns are merged, that is, the -1 element in e[left] is assigned to the value of the corresponding position of e[right], and e[right] is deleted: e[left][i]==e[right][i]||e[left][i]==-1||e[right]==-1. Repeat the above steps until there are no two columns that meet the conditions.

[0108] Furthermore, from top to bottom, the grid structure is merged into rows; the merging rules are as follows: from top to bottom, extract two adjacent rows from the table network area, set them as e[top] and e[down]; let len ​​be the length of the row, for each i∈[0, len-1], if the following conditions are met, then the two rows are merged, that is, the -1 element in e[top] is assigned to the value of the corresponding position of e[down], and e[dowm] is deleted: e[top][i]==e[down][i]||e[top][i]==-1||e[down]==-1, repeat the above steps until there are no two rows that meet the conditions.

[0109] In an embodiment of the present application, the AI ​​platform assigns the values ​​of the grid elements with a value of -1 remaining in the grid structure to the values ​​of their neighboring grid elements in accordance with the alignment principle. The number of rows of the grid structure after the merging is the number of cell rows in the table structure, and the number of columns of the grid structure is the number of cell columns in the table structure. For each cell, its position in the table structure is its position in the grid structure (cell row and column, starting and ending positions).

[0110] In step 207, the AI ​​platform determines the positions of each cell in the image based on the position coordinate information of the table, extracts text information from each position in the image, fills the text information into the corresponding cells, and sends it to the RPA robot.

[0111] In an embodiment of the present application, table detection is performed on the table area given by the table recall module, so it is necessary to map the coordinate results of the table detection back to the image, retrieve its position in the image, that is, the coordinate position of each cell in the image, and then extract text information based on the cell area.

[0112] Therefore, by adopting the secondary extraction method of table information, that is, first applying object detection and segmentation technology to implement the table recall model, completing the primary extraction of table location information, distinguishing between images with tables and images without tables, and for images with tables, applying object detection and segmentation technology to implement the table detection model, completing the secondary extraction of table structure information, both the stability of the table recognition system and the average speed of the overall RPA process can be guaranteed.

[0113] In order to implement the above embodiment, the present application also proposes a text recognition device combining RPA and AI.

[0114] Figure 6 A schematic diagram of the structure of an AI platform provided in an embodiment of the present application.

[0115] like Figure 6As shown, the AI ​​platform includes: a first acquisition module 601, a second acquisition module 602, an extraction module 603, a construction module 604 and a generation module 605.

[0116] The first acquisition module 601 is used to acquire an image.

[0117] The second acquisition module 602 is used to acquire a table area image from the image.

[0118] The extraction module 603 is used to extract each cell from the table area image.

[0119] The construction module 604 is used to construct a table according to the position information of each cell.

[0120] The generating module 605 is used to extract the text information in each cell from the image, and generate a text recognition result according to the text information and the table.

[0121] Furthermore, in a possible implementation of an embodiment of the present application, the second acquisition module 602 is specifically used to process the image through an object detection algorithm to obtain multiple first arrays; determine the table category according to the category identifiers in the multiple first arrays; obtain the target array corresponding to the table category, and determine the table area image according to each value in the target array.

[0122] Furthermore, in a possible implementation of the embodiment of the present application, the extraction module 603 is specifically used to process the table area image through an instance segmentation detection algorithm to obtain multiple second arrays; and generate each cell according to each value in the multiple second arrays.

[0123] Furthermore, in a possible implementation method of an embodiment of the present application, a construction module 604 is specifically used to determine the minimum cell width and the minimum cell height in each cell according to the position information of the each cell; determine a first step length according to the minimum cell width, and determine a second step length according to the minimum cell height; divide the total table area corresponding to the each cell in a first direction according to the first step length, and divide the total table area in the first direction according to the second step length to generate a table network area; number the each cell, and map the each cell to the table network area according to the coordinate position; merge the columns of the table network area according to the first direction, and merge the rows of the table network area according to the second direction to generate the table.

[0124] Furthermore, in a possible implementation method of the embodiment of the present application, the table is generated by merging columns of the table network area according to the first direction and merging rows of the table network area according to the second direction, including: extracting two adjacent columns from the table network area from left to right, and deleting either column if the cell values ​​corresponding to the two columns are equal; extracting two adjacent rows from the table network area from top to bottom, and deleting either row if the cell values ​​corresponding to the two rows are equal.

[0125] Furthermore, in a possible implementation method of an embodiment of the present application, the generation module 605 is specifically used to determine the positions of the cells in the image based on the position coordinate information of the table; extract text information from the positions in the image, and fill the text information into the corresponding cells and send it to the RPA robot.

[0126] Furthermore, in a possible implementation method of the embodiment of the present application, before constructing a table according to the position information of the individual cells, it also includes: an adjustment module, used to detect whether there is overlapping cell area coverage among the individual cells, and / or, whether the area of ​​the blank area between two cells is smaller than the area of ​​one cell, and / or, whether there are unidentified cells among the individual cells; if there is overlapping cell area coverage, and / or, the area of ​​the blank area between two cells is smaller than the area of ​​one cell, the target cell is adjusted according to the principle of minimum adjustment times, and / or, if there are unidentified cells, the maximized cells that meet the alignment principle are completed in the unidentified cells until there are no unidentified cells.

[0127] It should be noted that the above explanation of the method embodiment is also applicable to the device of this embodiment, and will not be repeated here.

[0128] In the text recognition device combining RPA and AI in the embodiment of the present application, by receiving the image sent by the RPA robot, the table area image is obtained from the image; each cell is extracted from the table area image; a table is constructed according to the position information of each cell; the text information in each cell is extracted from the image, and the text recognition result is generated according to the text information and the table and sent to the RPA robot. Thus, the technical problems of poor accuracy and low efficiency of text recognition in the prior art are solved. By first constructing a table in the image and then extracting the text information in the table from the image, the text recognition in the image can be completed quickly and accurately.

[0129] In order to implement the above embodiment, the present application also proposes an electronic device, Figure 7 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 7 As shown, a memory 21, a processor 22, and a computer program stored in the memory 21 and executable on the processor 22.

[0130] When the processor 22 executes the program, the text recognition method combining RPA and AI provided in the above embodiment is implemented.

[0131] Furthermore, the computer device also includes:

[0132] The communication interface 23 is used for communication between the memory 21 and the processor 22 .

[0133] The memory 21 is used to store computer programs that can be run on the processor 22 .

[0134] The memory 21 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0135] The processor 22 is used to implement the text recognition combining RPA and AI as described in the above embodiment when executing the program.

[0136] If the memory 21, the processor 22 and the communication interface 23 are implemented independently, the communication interface 21, the memory 21 and the processor 22 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0137] Optionally, in a specific implementation, if the memory 21, the processor 22 and the communication interface 23 are integrated on a chip, the memory 21, the processor 22 and the communication interface 23 can communicate with each other through an internal interface.

[0138] The processor 22 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0139] The electronic device of the embodiment of the present application receives an image sent by the RPA robot, obtains a table area image from the image; extracts each cell from the table area image; constructs a table according to the position information of each cell; extracts the text information in each cell from the image, generates a text recognition result according to the text information and the table, and sends it to the RPA robot. Thus, the technical problems of poor accuracy and low efficiency of text recognition in the prior art are solved. By first constructing a table in the image and then extracting the text information in the table from the image, the text recognition in the image can be completed quickly and accurately.

[0140] In order to implement the above-mentioned embodiment, the embodiment of the present application proposes a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the text recognition method combining RPA and AI described in the above-mentioned method embodiment is implemented.

[0141] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0142] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0143] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0144] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0145] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0146] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0147] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0148] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A text recognition method combining RPA and AI, characterized in that: The method comprises: The AI ​​platform receives the image sent by the RPA robot, and extracts an image including only the table portion from the image to obtain a table area image. The AI ​​platform extracts each cell from the table area image; The AI ​​platform constructs a table according to the position information of each cell; The AI ​​platform extracts text information in each cell from the image, generates a text recognition result according to the text information and the table, and sends it to the RPA robot; The AI ​​platform intercepts an image including only a table portion from the image to obtain a table area image, including: The AI ​​platform processes the image by an object detection algorithm to obtain a plurality of first arrays; The AI ​​platform determines the table category according to the category identifiers in the multiple first arrays; The AI ​​platform obtains a target array corresponding to the table category, and determines a table area image according to each value in the target array; The AI ​​platform extracts each cell from the table area image, including: The AI ​​platform processes the table area image by an instance segmentation detection algorithm to obtain a plurality of second arrays; The AI ​​platform generates each cell according to each value in the plurality of second arrays; The AI ​​platform constructs a table according to the position information of each cell, including: The AI ​​platform determines the minimum cell width and the minimum cell height of each cell according to the position information of each cell; The AI ​​platform determines a first step length according to the minimum cell width, and determines a second step length according to the minimum cell height; The AI ​​platform divides the total table area corresponding to each cell in a first direction according to the first step length, and divides the total table area in a second direction according to the second step length to generate a table network area; The AI ​​platform numbers each of the cells and maps each of the cells to the table network area according to the coordinate position; The AI ​​platform merges the columns of the table network area according to the first direction, and merges the rows of the table network area according to the second direction to generate the table.

2. The text recognition method combining RPA and AI as claimed in claim 1, characterized in that: The AI ​​platform merges the columns of the table network area according to the first direction, and merges the rows of the table network area according to the second direction to generate the table, including: The AI ​​platform extracts two adjacent columns from the table network area from left to right, and if the cell values ​​corresponding to the two columns are equal, deletes either column; The AI ​​platform extracts two adjacent rows from the table network area from top to bottom, and if the cell values ​​corresponding to the two rows are equal, deletes either row.

3. The text recognition method combining RPA and AI as claimed in claim 1, characterized in that: The AI ​​platform extracts text information in each cell from the image, generates a text recognition result according to the text information and the table, and sends it to the RPA robot, including: The AI ​​platform determines the positions of the cells in the image according to the position coordinate information of the table; The AI ​​platform extracts text information from various positions in the image, fills the text information into the corresponding cells, and sends it to the RPA robot.

4. The text recognition method combining RPA and AI as claimed in claim 1, characterized in that: Before the AI ​​platform constructs a table according to the position information of each cell, it also includes: The AI ​​platform detects whether there is any overlapping coverage of cell areas among the cells, and / or detects whether the area of ​​a blank area between two cells is smaller than the area of ​​a cell, and / or detects whether there is any unrecognized cell among the cells; If there is overlapping cell areas, and / or the area of ​​the blank area between two cells is smaller than the area of ​​one cell, the AI ​​platform adjusts the target cell according to the principle of minimum adjustment times, and / or if there are unidentified cells, the AI ​​platform completes the unidentified cells with maximized cells that meet the alignment principle until there are no unidentified cells.

5. An AI platform, characterized in that: include: A first acquisition module, used for acquiring an image; A second acquisition module is used to intercept an image including only a table portion from the image to obtain a table area image; An extraction module, used for extracting each cell from the table area image; A construction module, used to construct a table according to the position information of each cell; A generating module, used for extracting text information in each cell from the image, and generating a text recognition result according to the text information and the table; The second acquisition module is specifically used for: The AI ​​platform processes the image by an object detection algorithm to obtain a plurality of first arrays; The AI ​​platform determines the table category according to the category identifiers in the multiple first arrays; The AI ​​platform obtains a target array corresponding to the table category, and determines a table area image according to each value in the target array; The extraction module is specifically used for: The AI ​​platform processes the table area image by an instance segmentation detection algorithm to obtain a plurality of second arrays; The AI ​​platform generates each cell according to each value in the plurality of second arrays; The building blocks are specifically used for: The AI ​​platform determines the minimum cell width and the minimum cell height of each cell according to the position information of each cell; The AI ​​platform determines a first step length according to the minimum cell width, and determines a second step length according to the minimum cell height; The AI ​​platform divides the total table area corresponding to each cell in a first direction according to the first step length, and divides the total table area in a second direction according to the second step length to generate a table network area; The AI ​​platform numbers each of the cells and maps each of the cells to the table network area according to the coordinate position; The AI ​​platform merges the columns of the table network area according to the first direction, and merges the rows of the table network area according to the second direction to generate the table.

6. A text recognition system combining RPA and AI, characterized in that: include: The AI ​​platform, business system and RPA robot described in claim 5; The business system obtains the image by photographing it with a camera, or the business system obtains the image by scanning it with a scanner and sends it to the RPA robot; The RPA robot sends the image to the AI ​​platform for processing, generates a text recognition result, and returns it to the RPA robot; The RPA robot sends the text recognition result to the database of the business system for storage.

7. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the text recognition method combining RPA and AI as described in any one of claims 1 to 4 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the text recognition method combining RPA and AI as described in any one of claims 1 to 4 is implemented.

Images