Image text extraction method, device and electronic device combining RPA and AI
By combining RPA and AI, the object detection and character detection frames of images are merged, and the cumbersome problem of text extraction caused by the sequential extraction of multiple models in the prior art is solved, and efficient image text extraction is achieved.
Patent Information
- Application Number
- CN202010886737.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-08-28
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2040-08-28
AI Technical Summary
In the prior art, documents contain multiple types of content (such as text, tables, and red chapters) that require multiple models to extract text separately, resulting in cumbersome and inefficient.
Combining RPA and AI, by detecting the image to be processed, the position information and type of the detection box is determined, and the character detection box is combined to determine the text box, and then text recognition is performed to achieve the extraction of different types of data.
The image text extraction process is simplified, the text extraction efficiency is improved, and the different types of data content in the image can be determined through one detection, which improves the accuracy and efficiency of extraction.
Smart Images

Figure CN112149663B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of automation technology, and in particular to a method, device, electronic device and storage medium for extracting text from images combining RPA and AI. Background Art
[0002] Robotic Process Automation (RPA) uses specific "robot software" to simulate human operations on computers and automatically execute process tasks according to rules.
[0003] Artificial Intelligence (AI) is a technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.
[0004] With the development of AI, Optical Character Recognition (OCR) technology has penetrated into various fields to help people reduce repetitive and inefficient work, especially work that requires transcribing text information to a computer. The combination of RPA technology and OCR technology has become a new trend in the RPA field, helping companies process text and image data more efficiently and improve work efficiency.
[0005] However, in the related art, for documents including multiple types of content such as text, tables, and red seals, multiple models are usually required to extract text from different types of content in turn, which makes the text extraction process cumbersome and inefficient. Summary of the invention
[0006] The image text extraction method, device, electronic device and storage medium combining RPA and AI proposed in this application are used to solve the problem in the related art that for documents including various types of content such as text, tables, red seals, etc., multiple models are usually required to extract text from different types of content in turn, which leads to a cumbersome and inefficient text extraction process.
[0007] An embodiment of one aspect of the present application proposes a method for extracting text from an image by combining RPA and AI, including: performing target detection on the image to be processed to determine the position information of each detection frame contained in the image to be processed and the type of each detection frame, wherein the type of each detection frame includes: character, non-character, beginning of a text line and end of a text line; based on the position information of each detection frame and the type of each detection frame, merging the detection frames of the character type to determine the text frames contained in the image to be processed; and performing text recognition on each text frame to determine the text contained in the image to be processed.
[0008] Optionally, in a possible implementation manner of the embodiment of the first aspect of the present application, performing target detection on the image to be processed to determine the position information of each detection frame contained in the image to be processed and the type of each detection frame specifically includes:
[0009] Extracting multiple dimensional features of each detection frame from the image to be processed;
[0010] Performing attention mechanism learning on the multiple dimensional features to obtain adjacent frame information of each detection frame and the beginning and end information of the text line corresponding to each detection frame;
[0011] The type of each detection frame is determined according to the adjacent frame information of each detection frame and the corresponding text line beginning and end information.
[0012] Optionally, in another possible implementation manner of the embodiment of the first aspect of the present application, before extracting the multiple dimensional features of each of the detection boxes from the image to be processed, the method further includes:
[0013] Preprocessing the image to be processed to obtain a plurality of feature maps corresponding to the image to be processed;
[0014] The extracting multiple dimensional features of each detection frame from the image to be processed specifically includes:
[0015] Multiple dimensional features of each of the detection boxes are extracted from the multiple feature maps.
[0016] Optionally, in another possible implementation of the first aspect of the present application, extracting multiple dimensional features of each detection frame from the image to be processed specifically includes:
[0017] The image to be processed is convolved using at least two filters to obtain at least two dimensional features of each of the detection frames, wherein the receptive fields of the at least two filters are different.
[0018] Optionally, in another possible implementation manner of the embodiment of the first aspect of the present application, before performing attention mechanism learning on the multiple dimensional features to obtain adjacent frame information of each detection frame and the beginning and end information of the text line corresponding to each detection frame, it also includes:
[0019] The multiple dimensional features are concatenated to generate features of each detection frame.
[0020] Optionally, in another possible implementation manner of the embodiment of the first aspect of the present application, before performing attention mechanism learning on the multiple dimensional features to obtain adjacent frame information of each detection frame and the beginning and end information of the text line corresponding to each detection frame, it also includes:
[0021] The multiple dimensional features are normalized.
[0022] Optionally, in another possible implementation manner of the embodiment of the first aspect of the present application, the position information of each detection frame includes the coordinates of each detection frame in the first direction and the offset in the second direction, and the merging of the detection frames of the character type according to the position information of each detection frame and the type of each detection frame to determine each text frame included in the image to be processed specifically includes:
[0023] If the type of any detection frame is the beginning of a text line, obtaining, from the respective detection frames, a candidate detection frame whose second coordinate in the first direction matches the first coordinate, according to the first coordinate of the any detection frame in the first direction;
[0024] According to the first offset of any detection frame in the second direction, acquiring, from the candidate detection frames, an adjacent detection frame adjacent to any detection frame in the second direction;
[0025] If the type of the adjacent detection frame is a character, the adjacent detection frame is merged with any one of the detection frames.
[0026] Optionally, in another possible implementation manner of the embodiment of the first aspect of the present application, after merging the detection frames of type character according to the position information of each detection frame and the type of each detection frame to determine each text frame included in the image to be processed, the method further includes:
[0027] Performing a connected domain analysis on each of the text boxes to determine a connected domain shape corresponding to each of the text boxes;
[0028] If the shape of the connected domain corresponding to any text box is a circle, it is determined that any text box contains a red seal.
[0029] On the other hand, an embodiment of the present application proposes an image text extraction device combining RPA and AI, including: a first determination module, used to perform target detection on the image to be processed to determine the position information of each detection frame contained in the image to be processed and the type of each detection frame, wherein the type of each detection frame includes: character, non-character, beginning of text line and end of text line; a second determination module, used to merge the detection frames of type character according to the position information of each detection frame and the type of each detection frame, so as to determine the text frames contained in the image to be processed; a third determination module, used to perform text recognition on each text frame, so as to determine the text contained in the image to be processed.
[0030] Optionally, in a possible implementation manner of the embodiment of the first aspect of the present application, the first determining module specifically includes:
[0031] An extraction unit, used to extract multiple dimensional features of each detection frame from the image to be processed;
[0032] A first acquisition unit is used to perform attention mechanism learning on the multiple dimensional features to obtain adjacent frame information of each detection frame and the beginning and end information of the text line corresponding to each detection frame;
[0033] A determination unit is used to determine the type of each detection frame according to the adjacent frame information of each detection frame and the corresponding text line beginning and end information.
[0034] Optionally, in another possible implementation manner of the embodiment of the first aspect of the present application, the first determining module further includes:
[0035] A second acquisition unit, used for preprocessing the image to be processed to obtain a plurality of feature maps corresponding to the image to be processed;
[0036] The extraction unit specifically comprises:
[0037] The extraction subunit is used to extract multiple dimensional features of each detection box from the multiple feature maps.
[0038] Optionally, in another possible implementation of the first aspect of the present application, the extraction unit specifically includes:
[0039] The acquisition subunit is used to perform convolution processing on the image to be processed using at least two filters to obtain at least two dimensional features of each of the detection frames, wherein the receptive fields of the at least two filters are different.
[0040] Optionally, in another possible implementation manner of the embodiment of the first aspect of the present application, the first determining module further includes:
[0041] A splicing unit is used to splice the multiple dimensional features to generate features of each detection frame.
[0042] Optionally, in another possible implementation manner of the embodiment of the first aspect of the present application, the first determining module further includes:
[0043] A normalization unit is used to perform normalization processing on the multiple dimensional features.
[0044] Optionally, in another possible implementation manner of the embodiment of the first aspect of the present application, the position information of each detection frame includes the coordinates of each detection frame in the first direction and the offset in the second direction, and the second determination module specifically includes:
[0045] A third acquisition unit is used for acquiring, when the type of any detection frame is the beginning of a text line, from the detection frames according to the first coordinate of any detection frame in the first direction, a candidate detection frame whose second coordinate in the first direction matches the first coordinate;
[0046] A fourth acquisition unit, configured to acquire, from the candidate detection frames, an adjacent detection frame adjacent to the any detection frame in the second direction according to the first offset of the any detection frame in the second direction;
[0047] A merging unit is used to merge the adjacent detection box with any one of the detection boxes when the type of the adjacent detection box is a character.
[0048] Optionally, in another possible implementation of the embodiment of the first aspect of the present application, the device further includes:
[0049] A fourth determination module is used to perform a connected domain analysis on each of the text boxes to determine a connected domain shape corresponding to each of the text boxes;
[0050] The fifth determining module is used to determine that any text box contains a red seal when the shape of the connected domain corresponding to any text box is a circle.
[0051] In another aspect of the present application, an electronic device is proposed in an embodiment, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for extracting text from an image combining RPA and AI as described above is implemented.
[0052] In another aspect of the present application, a computer-readable storage medium is proposed in an embodiment, on which a computer program is stored, and it is characterized in that when the program is executed by a processor, the method for extracting image text combining RPA and AI as described above is implemented.
[0053] Another aspect of the present application provides a computer program, which, when executed by a processor, implements the method for extracting image text combining RPA and AI as described in the embodiment of the present application.
[0054] The method, device, electronic device, computer-readable storage medium and computer program for extracting text from an image combined with RPA and AI provided in the embodiments of the present application perform target detection on the image to be processed to determine the position information of each detection frame contained in the image to be processed and the type of each detection frame, and merge the detection frames of the character type according to the position information of each detection frame and the type of each detection frame to determine the text frames contained in the image to be processed, and then perform text recognition on each text frame to determine the text contained in the image to be processed. Thus, by determining the type of each detection frame while performing target detection on the image to be processed, different types of data content in the image can be determined through a single detection, thereby simplifying the process of extracting text from an image and improving the efficiency of text extraction.
[0055] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0057] Figure 1 A flowchart of a method for extracting text from an image by combining RPA and AI provided in an embodiment of the present application;
[0058] Figure 2 It is a schematic diagram of the position of the image to be processed and a detection frame;
[0059] Figure 3 A flowchart of another method for extracting text from an image by combining RPA and AI provided in an embodiment of the present application;
[0060] Figure 4 A flowchart of another method for extracting text from an image by combining RPA and AI provided in an embodiment of the present application;
[0061] Figure 5 A schematic diagram of the structure of an image text extraction device combining RPA and AI provided in an embodiment of the present application;
[0062] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0063] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.
[0064] In view of the problem in the related art that for documents including various types of content such as text, tables, and red seals, multiple models are usually required to extract text from different types of content in turn, resulting in a cumbersome and inefficient text extraction process, the embodiments of the present application propose a method for extracting image text that combines RPA and AI.
[0065] The method for extracting text from an image that combines RPA and AI provided in an embodiment of the present application performs target detection on the image to be processed to determine the position information of each detection frame contained in the image to be processed and the type of each detection frame, and merges the detection frames of the character type according to the position information of each detection frame and the type of each detection frame to determine the text frames contained in the image to be processed, and then performs text recognition on each text frame to determine the text contained in the image to be processed. Thus, by determining the type of each detection frame while performing target detection on the image to be processed, different types of data content in the image can be determined through a single detection, thereby simplifying the process of extracting text from an image and improving the efficiency of text extraction.
[0066] The following is a detailed description of the image text extraction method, device, electronic device, storage medium and computer program combined with RPA and AI provided in the present application with reference to the accompanying drawings.
[0067] Figure 1 A flowchart of a method for extracting text from an image by combining RPA and AI provided in an embodiment of the present application.
[0068] like Figure 1 As shown, the method for extracting text from an image by combining RPA and AI includes the following steps:
[0069] Step 101, performing target detection on the image to be processed to determine the position information of each detection frame contained in the image to be processed and the type of each detection frame, wherein the type of each detection frame includes: character, non-character, beginning of a text line and end of a text line.
[0070] It should be noted that RPA technology can intelligently understand the existing applications of the electronic device through the user interface, and automate repetitive, rule-based, and large-scale routine operations, such as automatically and repeatedly reading emails, reading Office components, operating databases and web pages, client software, etc., collecting data and performing tedious calculations to generate files and reports in large quantities, thereby greatly reducing the investment in labor costs and effectively improving office efficiency through RPA technology. Therefore, in the scene of image text extraction, the RPA program can be configured in the electronic device used for image text extraction, so that the electronic device can automatically extract text from the acquired image according to the rules set in the RPA program.
[0071] In actual use, the image text extraction method combining RPA and AI in the embodiment of the present application can be applied to any scenario of extracting text from an image, and the embodiment of the present application does not limit this. For example, it can be applied to the input scenario of paper documents such as certificates and bills.
[0072] The image to be processed may refer to an image acquired by the RPA robot. For example, when the image text extraction method combining RPA and AI in the embodiment of the present application is applied to the document information upload scenario of the accounting department, the image to be processed may be an image of various documents such as travel expenses, transportation expenses, banquet expenses, etc. acquired by the RPA robot and uploaded by the user through an electronic device.
[0073] Among them, the position information of the detection frame may include the coordinates of each vertex of the detection frame in the image to be processed; or, when the coordinate system corresponding to the image to be processed includes a first direction and a second direction, the position information of the detection frame may also include the coordinates of the detection frame in the first direction and the offset in the second direction, so as to determine the specific position of the detection frame in the image to be processed through the position information of the detection frame.
[0074] For example, if Figure 2 As shown in FIG. 1 , it is a schematic diagram of the position of the image to be processed and a detection frame. Wherein, O is the origin of the coordinate system corresponding to the image to be processed 20, the Y axis is the first direction of the coordinate system corresponding to the image to be processed 20, and the X axis is the second direction of the coordinate system corresponding to the image to be processed 20. Then, y1 is the coordinate of the detection frame 21 in the first direction, and x1 is the offset of the detection frame 21 in the second direction, that is, the position information of the detection frame 21 is (y1, x1).
[0075] As a possible implementation method, an OCR algorithm based on CTPN (Detecting Text in Natural Image with Connectionist Text Proposal Network) can be used to perform target detection on the image to be processed to determine the position information of each to-be-detected frame contained in the image to be processed and the type of each detection frame. Specifically, Transformer can be used to replace the Long Short-Term Memory (LSTM) network under the CTPN framework to perform target detection on the image to be processed and obtain the text line association information in the image, so as to determine not only the position information of the detection frame corresponding to each target in the image, and whether the type of each detection frame is a character, but also whether the type of each detection frame is the beginning of a text line and whether it is the end of a text line.
[0076] Step 102: merging the detection frames of character type according to the position information of each detection frame and the type of each detection frame to determine the text frames included in the image to be processed.
[0077] The text box contained in the image to be processed may include a complete and independent text in the image to be processed.
[0078] In an embodiment of the present application, after determining the position information of each detection frame and the type of each detection frame in the image to be processed, the detection frames in the same row can be determined first according to the position information of each detection frame, and then according to the type of the detection frame in the same row, it can be determined whether the detection frame is the beginning of a text line. If a detection frame A is the beginning of a text line, the next detection frame B in the same row and adjacent to it can be determined according to the position information of the detection frame A; if the type of the detection frame B is a character and it is the end of the text line, the detection frame A and the detection frame B can be merged as a text frame; if the type of the detection frame B is a character and it is not the end of the text line, the detection frame A can be merged with the detection frame B, and the next detection frame C adjacent to the detection frame B can be determined, and then the above steps are repeated to determine whether the detection frame C can be merged with the detection frame A and the detection frame B, until the detection frame D of the type of the end of the text line is traversed, and all the detection frames between the detection frame A and the detection frame D can be merged to generate a text frame. Thus, repeating the above steps can determine all the text frames included in the image to be processed.
[0079] Step 103: Perform text recognition on each text box to determine the text contained in the image to be processed.
[0080] In an embodiment of the present application, after determining the text boxes included in the image to be processed, any text recognition algorithm can be used to recognize the text in each text box to determine the text corresponding to each text box, and then determine the text contained in the image to be processed.
[0081] The method for extracting text from an image that combines RPA and AI provided in an embodiment of the present application performs target detection on the image to be processed to determine the position information of each detection frame contained in the image to be processed and the type of each detection frame, and merges the detection frames of the character type according to the position information of each detection frame and the type of each detection frame to determine the text frames contained in the image to be processed, and then performs text recognition on each text frame to determine the text contained in the image to be processed. Thus, by determining the type of each detection frame while performing target detection on the image to be processed, different types of data content in the image can be determined through a single detection, thereby simplifying the process of extracting text from an image and improving the efficiency of text extraction.
[0082] In a possible implementation form of the present application, multiple dimensional features of the detection frame can be extracted to determine the type of the detection frame, thereby further improving the accuracy of image text extraction.
[0083] Combine the following Figure 3 , the image text extraction method combining RPA and AI provided in the embodiment of the present application is further explained.
[0084] Figure 3 A flowchart of another method for extracting text from images that combines RPA and AI provided in an embodiment of the present application.
[0085] like Figure 3 As shown, the method for extracting text from an image by combining RPA and AI includes the following steps:
[0086] Step 201 : performing object detection on the image to be processed to determine the position information of each detection frame contained in the image to be processed.
[0087] The specific implementation process and principle of the above step 201 can refer to the detailed description of the above embodiment, which will not be repeated here.
[0088] Step 202: extract multiple dimensional features of each detection box from the image to be processed.
[0089] Among them, multiple dimensional features may refer to features that can represent the features of the detection box in the image to be processed from different granularities.
[0090] In an embodiment of the present application, the image to be processed can be convolved with convolution kernels of different sizes to generate multiple dimensional features of each detection frame in the image to be processed, wherein the result of using one convolution and convolution processing on the image to be processed is a dimensional feature of each detection frame in the image to be processed; or, the image to be processed can be convolved with convolution kernels of the same size and different convolution methods to generate multiple dimensional features of each detection frame in the image to be processed, wherein the convolution result corresponding to one convolution method is a dimensional feature of each detection frame in the image to be processed. Thus, the features of different granularities of the image to be processed are extracted, thereby improving the accuracy of detection frame recognition.
[0091] As a possible implementation, different filters may be used to perform convolution processing on the image to be processed to generate different dimensional features of each detection frame. That is, in a possible implementation form of the embodiment of the present application, the above step 202 may include:
[0092] The image to be processed is convolved using at least two filters to obtain at least two dimensional features of each detection frame, wherein the receptive fields of at least two filters are different.
[0093] The filter may be a convolution kernel with a dilation coefficient. For example, the size of the filter may be 3×3, and the dilation coefficient may be 1, 2, 5, and so on.
[0094] As a possible implementation method, the image to be processed can be convolved with n1 filters of size 3×3, that is, the size of the receptive field of the filter is 3×3, to generate n1-dimensional features of each detection frame in the image to be processed; and the image to be processed can be convolved with n2 filters of size 3×3 and expansion coefficient 2, that is, the size of the receptive field of the filter is 7×7, to generate n2-dimensional features of each detection frame in the image to be processed; finally, the image to be processed can be convolved with n3 filters of size 3×3 and expansion coefficient 5, that is, the size of the receptive field of the filter is 19×19, to generate n3-dimensional features of each detection frame in the image to be processed, thereby generating n1+n2+n3-dimensional features of each detection frame in the image to be processed.
[0095] It should be noted that in actual use, the specific values of n1, n2, and n3 can be determined according to actual needs, and the embodiments of the present application do not limit this. For example, n1 can be 256, n2 can be 128, and n3 can be 128.
[0096] Furthermore, before extracting the multi-dimensional features of the image to be processed, the image to be processed may be preprocessed to generate a feature map corresponding to the image to be processed, so as to further improve the accuracy of identifying the image to be processed. That is, in a possible implementation form of the embodiment of the present application, before the above step 202, the following may also be included:
[0097] Preprocessing the image to be processed to obtain multiple feature maps corresponding to the image to be processed;
[0098] Accordingly, the above step 202 may include:
[0099] Multiple dimensional features of each detection box are extracted from multiple feature maps.
[0100] As a possible implementation method, densenet121 can be used to extract features from the image to be processed to generate multiple feature maps corresponding to the image to be processed. For example, a feature map of size 512×(pic_height / 8)×(pic_width / 8) corresponding to the image to be processed can be generated, that is, 512 feature maps of size (pic_height / 8)×(pic_width / 8) corresponding to the image to be processed are generated, where pic_height is the height of the image to be processed and pic_width is the width of the image to be processed. After determining the multiple feature maps corresponding to the image to be processed, each feature map can be convolved to generate multiple dimensional features of each detection box extracted from multiple feature maps.
[0101] Specifically, the above-mentioned method of convolving the image to be processed can be used to perform the same convolution processing on the feature map to extract multiple dimensional features of each detection frame from the feature map. For example, if there are 512 feature maps corresponding to the image to be processed, the 512 feature maps can be convolved by n1 filters with a size of 3×3×512 to generate n1-dimensional features of each detection frame; and the 512 feature maps can be convolved by n2 filters with a size of 3×3×512 and a dilation coefficient of 2 to generate n2-dimensional features of each detection frame; finally, the 512 feature maps can be convolved by n3 filters with a size of 3×3×512 and a dilation coefficient of 5 to generate n3-dimensional features of each detection frame, thereby generating n1+n2+n3-dimensional features of each detection frame of the image to be processed.
[0102] Step 203, perform attention mechanism learning on multiple dimensional features to obtain adjacent frame information of each detection frame and the beginning and end information of the text line corresponding to each detection frame.
[0103] In an embodiment of the present application, multiple dimensional features can be learned through a target detection model including a multi-layer decoder and an attention mechanism to obtain adjacent frame information of each detection frame and the corresponding text line beginning and end information, that is, to obtain the type information of the adjacent detection frames of each detection frame, as well as whether each detection frame is the beginning of a text line, and whether each detection frame is the end of a text line.
[0104] As a possible implementation method, when the image to be processed is recognized by the above-mentioned target detection model, if it is detected that the image to be processed includes k detection frames, the target detection model can input 2k coordinate information to represent the position information of each detection frame (such as the coordinates of each detection frame in the first direction, and the offset in the second direction), and can output 4k score information to represent the category of each detection frame, that is, each detection frame can correspond to 4 score information, which are respectively used to represent the probability of the detection frame being a character, the probability of being a non-character, the probability of being the beginning of a text line, and the probability of being the end of a text line.
[0105] Furthermore, before learning the attention mechanism for multiple dimensional features, the multiple dimensional features can also be fused to identify the multiple dimensional features as a whole, further improving the accuracy of identifying the detection frame in the image to be processed. That is, in a possible implementation form of the embodiment of the present application, before the above step 203, it can also include:
[0106] Multiple dimensional features are concatenated to generate the features of each detection box.
[0107] As a possible implementation method, in order to enable the target detection mode to combine the features of different granularities of the image to be processed when identifying the image to be processed, the multiple dimensional features of each detection frame in the image to be processed can be spliced to generate the features of each detection frame, that is, the features of each detection frame are represented by an overall feature vector, so that the features of each detection frame can contain the feature information of the image to be processed at different granularities, thereby improving the accuracy of identifying the detection frames in the image to be processed.
[0108] For example, by convolving the image to be processed with 256 filters of size 3×3, 256-dimensional features of each detection frame in the image to be processed are generated; by performing dilated convolution on the image to be processed with 128 filters of size 3×3 and dilation coefficient 2, 128-dimensional features of each detection frame in the image to be processed are generated; by performing dilated convolution on the image to be processed with 128 filters of size 3×3 and dilation coefficient 5, 128-dimensional features of each detection frame in the image to be processed are generated, so that the generated 256-dimensional features can be concatenated with the two 128-dimensional features to generate 512-dimensional features of each detection frame in the image to be processed.
[0109] As a possible implementation method, after concatenating the multi-dimensional features, the concatenated features can be compressed through a fully connected layer to reduce the size of the concatenated features, thereby reducing the amount of computation required to identify the concatenated features. For example, if the concatenated features are 512-dimensional, the concatenated features can be compressed to 256-dimensional through a fully connected layer.
[0110] Furthermore, since the feature measurements determined by different methods may be different, it is easy to ignore features with too low brightness, thereby affecting the reliability of identifying the image to be processed. That is, in a possible implementation form of the embodiment of the present application, before the above step 203, it may also include:
[0111] Normalize multiple dimensional features.
[0112] As a possible implementation method, after determining the multiple dimensional features of each detection box in the image to be processed by different methods, the multiple dimensional features can be normalized so that the measurements of each dimensional feature are in the same numerical range, thereby reducing the impact of different measurements of multiple dimensional features on the recognition accuracy of the image to be processed.
[0113] As another possible implementation method, after concatenating features of multiple dimensions, normalization may be performed on the concatenated features.
[0114] Step 204, determining the type of each detection box according to the adjacent box information of each detection box and the corresponding text line beginning and end information, wherein the type of each detection box includes: character, non-character, text line beginning and text line end.
[0115] In an embodiment of the present application, after the adjacent detection frame information of each detection frame in the image to be processed and the corresponding text line head and tail information are determined through the attention mechanism, the type of each detection frame can be determined based on the adjacent detection frame information of each detection frame and the text line head and tail information.
[0116] As a possible implementation, the type of each detection frame can also be determined based only on the beginning and end information of the text line corresponding to each detection frame. For example, the four score information of the detection frame A output by the target detection mode are [0.99, 0, 1, 0], where the four score information sequentially represent the probability of the detection frame A being a character, the probability of being a non-character, the probability of being the beginning of a text line, and the probability of being the end of a text line, so that the type of the detection frame can be determined to be a character or the beginning of a text line.
[0117] As another possible implementation, the type of each detection frame can also be determined by the adjacent frame information of each detection frame and the corresponding text line head and tail information. Specifically, the type of each detection frame can be first determined according to the text line head and tail information corresponding to each detection frame, and then the type of each detection frame can be checked according to the adjacent frame information of each detection frame to assist in determining whether the determined detection frame type is accurate, so as to further improve the accuracy of the detection frame type determination.
[0118] For example, the four score information of the detection frame A output by the target detection mode are [0.99, 0, 1, 0], respectively, where the four score information represent the probability of the detection frame A being a character, the probability of being a non-character, the probability of being the beginning of a text line, and the probability of being the end of a text line in order, so that the type of the detection frame can be determined to be a character or the beginning of a text line. In addition, the score information of the adjacent frame B before the detection frame A is [0.9, 0.05, 0.1, 0.95], and the score information of the adjacent frame C after the detection frame A is [0.92, 0.1, 0.1, 0.1], so that it can be determined that the probability of the detection frame A being the beginning of a text line is very high, so that the type of the detection frame A can be determined to be a character or the beginning of a text line.
[0119] Step 205 , based on the position information of each detection frame and the type of each detection frame, the detection frames of the character type are merged to determine the text frames included in the image to be processed.
[0120] Step 206: Perform text recognition on each text box to determine the text contained in the image to be processed.
[0121] The specific implementation process and principle of the above steps 205-206 can refer to the detailed description of the above embodiment, which will not be repeated here.
[0122] The method for extracting text from an image that combines RPA and AI provided in an embodiment of the present application extracts multiple dimensional features of each detection frame from the image to be processed, and performs attention mechanism learning on the multiple dimensional features to obtain the adjacent frame information of each detection frame and the beginning and end information of the text line corresponding to each detection frame. Then, the type of each detection frame is determined based on the adjacent frame information of each detection frame and the beginning and end information of the corresponding text line. Then, based on the position information of each detection frame and the type of each detection frame, the detection frames of type character are merged to determine the text frames contained in the image to be processed, and text recognition is performed on each text frame to determine the text contained in the image to be processed. Thus, by extracting multi-dimensional features of different granularities of the image to be processed and performing feature representation on each detection frame in the image to be processed, not only can different types of data content in the image be determined through one detection, but the process of image text extraction is simplified, the efficiency of text extraction is improved, and the accuracy and reliability of text extraction are further improved.
[0123] In a possible implementation form of the present application, the detection frames of the character type can be merged according to the position information of the detection frames to determine the various text frames included in the image, and the text frames can be subjected to connected domain analysis to realize the recognition of the red seal in the image, thereby further improving the practicality and versatility of image text extraction.
[0124] Combine the following Figure 4 , the image text extraction method combining RPA and AI provided in the embodiment of the present application is further explained.
[0125] Figure 4 A flowchart of another method for extracting text from images that combines RPA and AI provided in an embodiment of the present application.
[0126] like Figure 4 As shown, the method for extracting text from an image by combining RPA and AI includes the following steps:
[0127] Step 301, performing target detection on the image to be processed to determine the position information of each detection frame contained in the image to be processed and the type of each detection frame, wherein the type of each detection frame includes: character, non-character, text line beginning and text line end.
[0128] The specific implementation process and principle of the above step 301 can be referred to the detailed description of the above embodiment, which will not be repeated here.
[0129] Step 302: If the type of any detection frame is the beginning of a text line, then according to the first coordinate of any detection frame in the first direction, a candidate detection frame whose second coordinate in the first direction matches the first coordinate is obtained from each detection frame.
[0130] The position information of the detection frame includes the coordinates of each detection frame in the first direction and the offset in the second direction. It should be noted that the first direction may refer to the Y-axis direction of the coordinate system corresponding to the image to be processed, and the second direction may refer to the X-axis direction of the coordinate system corresponding to the image to be processed, which is not limited in the present embodiment. For a specific schematic diagram, please refer to Figure 2 And in the above embodiments, Figure 2 The explanation will not be repeated here.
[0131] In an embodiment of the present application, if the type of a detection frame is the beginning of a text line, the detection frames in the same independent text as the detection frame can be determined from the detection frames in the same line and merged. Specifically, assuming that the type of detection frame A is the beginning of a text line, the detection frame whose difference between the second coordinate and the first coordinate is less than the first threshold can be first determined based on the first coordinate of the detection frame A in the first direction and the second coordinates of the other detection frames in the image to be processed in the first direction, and determined as the candidate detection frame whose second coordinate matches the first coordinate, that is, the candidate detection frame is in the same line as the detection frame A.
[0132] It should be noted that, in actual use, the specific value of the first threshold can be determined according to actual needs and the height of the detection frame, and the embodiment of the present application does not limit this. For example, the first threshold can be 1 / 3 of the height of the detection frame.
[0133] Step 303: According to the first offset of any detection frame in the second direction, an adjacent detection frame adjacent to any detection frame in the second direction is obtained from the candidate detection frames.
[0134] In an embodiment of the present application, after determining the candidate detection frame that is in the same row as the detection frame of the type of the beginning of a text line, the adjacent detection frame adjacent to the detection frame can be determined based on the first offset of the detection frame in the second direction and the second offset of each candidate detection frame in the second direction. Specifically, assuming that detection frame A is a detection frame of the type of the beginning of a text line, if the difference between the second offset of a candidate detection frame B corresponding to detection frame A and the first offset of detection frame A is less than or equal to the width of the detection frame, then candidate detection frame B can be determined to be an adjacent detection frame of detection frame A; otherwise, it can be determined that candidate detection frame B is not an adjacent detection frame of detection frame A.
[0135] Step 304: If the type of the adjacent detection box is a character, merge the adjacent detection box with any detection box.
[0136] In the embodiment of the present application, after the adjacent detection frame of the detection frame is determined, if the type of the adjacent detection frame is a character, the adjacent detection frame can be merged with the detection frame. Afterwards, the adjacent detection frame adjacent to the adjacent detection frame is determined in the same manner, and it is determined whether the merge process can be performed. It can be understood that after traversing all the detection frames in the image to be processed by the above method, all the text frames included in the image to be processed can be determined.
[0137] Step 305: Perform a connected domain analysis on each text box to determine the connected domain shape corresponding to each text box.
[0138] In the disclosed embodiment, the red seal included in the image to be processed can also be identified by performing a connected domain analysis on the text box. Specifically, since the text box containing characters is usually square, even if the connected domain analysis is performed on the text box containing characters, the shape of the generated connected domain is usually square; and since the red seal is usually a high circle, and the red seal usually contains different types of content such as characters and images, and the characters are not distributed in rows, the red seal part can usually be divided into multiple text boxes, and the text boxes corresponding to the red seal have the same image features. Therefore, by performing a connected domain analysis on each text box included in the image to be processed, the text boxes corresponding to the red seal can be connected together to form a complete connected domain, and the connected domain corresponding to the red seal is usually a circle.
[0139] Step 306: If the shape of the connected domain corresponding to any text box is a circle, it is determined that any text box contains a red seal.
[0140] In the embodiment of the present application, since the shape of the red seal in the text is usually circular, after performing a connected domain analysis on each text box in the image to be processed, since the shape of the connected domain of the text box corresponding to the character is usually square, and the shape of the connected domain corresponding to the red seal is usually circular, after performing a connected domain analysis on each text box, the text box whose corresponding connected domain shape is circular can be determined as the text box containing the red seal.
[0141] Step 307: Perform text recognition on each text box to determine the text contained in the image to be processed.
[0142] The specific implementation process and principle of the above step 307 can be referred to the detailed description of the above embodiment, which will not be repeated here.
[0143] The method for extracting text from an image that combines RPA and AI provided in an embodiment of the present application determines the position information of each detection frame contained in the image to be processed and the type of each detection frame, and when the detection frame is the beginning of a text line, determines the candidate detection frame based on the first coordinate of the detection frame in the first direction and the second coordinate of other detection frames in the first direction, and merges the adjacent detection frames that are adjacent to the detection frame and are of character type with the detection frame based on the first offset of the detection frame in the second direction and the type of the candidate detection frame, and then determines the red seal in the image to be processed through connected domain analysis, and performs text recognition on each text frame to determine the text contained in the image to be processed. Thus, by performing connected domain analysis on the text frame to identify the red seal contained in the image, not only can different types of data content in the image be determined through one detection, the process of image text extraction is simplified, the efficiency of text extraction is improved, and the practicality and versatility of image text extraction are further improved.
[0144] In order to implement the above embodiments, the present application also proposes an image text extraction device that combines RPA and AI.
[0145] Figure 5 A schematic diagram of the structure of an image text extraction device combining RPA and AI provided in an embodiment of the present application.
[0146] like Figure 5 As shown, the image text extraction device 40 combining RPA and AI includes:
[0147] The first determination module 41 is used to perform target detection on the image to be processed to determine the position information of each detection frame contained in the image to be processed and the type of each detection frame, wherein the type of each detection frame includes: character, non-character, text line beginning and text line end;
[0148] A second determination module 42 is used to merge the detection frames of the character type according to the position information of each detection frame and the type of each detection frame to determine each text frame included in the image to be processed;
[0149] The third determination module 43 is used to perform text recognition on each text box to determine the text contained in the image to be processed.
[0150] In actual use, the image text extraction device combining RPA and AI provided in the embodiment of the present application can be configured in any electronic device to execute the aforementioned image text extraction method combining RPA and AI.
[0151] The device for extracting text from an image that combines RPA and AI provided in an embodiment of the present application performs target detection on the image to be processed to determine the position information of each detection frame contained in the image to be processed and the type of each detection frame, and merges the detection frames of the character type according to the position information of each detection frame and the type of each detection frame to determine the text frames contained in the image to be processed, and then performs text recognition on each text frame to determine the text contained in the image to be processed. Thus, by determining the type of each detection frame while performing target detection on the image to be processed, different types of data content in the image can be determined through a single detection, thereby simplifying the process of extracting text from an image and improving the efficiency of text extraction.
[0152] In a possible implementation form of the present application, the first determining module 41 specifically includes:
[0153] An extraction unit, used to extract multiple dimensional features of each detection frame from the image to be processed;
[0154] A first acquisition unit is used to perform attention mechanism learning on the multiple dimensional features to obtain adjacent frame information of each detection frame and the beginning and end information of the text line corresponding to each detection frame;
[0155] A determination unit is used to determine the type of each detection frame according to the adjacent frame information of each detection frame and the corresponding text line beginning and end information.
[0156] Furthermore, in another possible implementation form of the present application, the first determining module 41 further includes:
[0157] A second acquisition unit is used to pre-process the image to be processed to obtain a plurality of feature maps corresponding to the image to be processed;
[0158] Accordingly, the above extraction unit specifically includes:
[0159] The extraction subunit is used to extract multiple dimensional features of each detection box from multiple feature maps.
[0160] Furthermore, in another possible implementation form of the present application, the extraction unit specifically includes:
[0161] The acquisition subunit is used to perform convolution processing on the image to be processed using at least two filters to obtain at least two dimensional features of each detection frame, wherein the receptive fields of at least two filters are different.
[0162] Furthermore, in another possible implementation form of the present application, the first determining module 41 further includes:
[0163] The splicing unit is used to splice multiple dimensional features to generate the features of each detection box.
[0164] Furthermore, in another possible implementation form of the present application, the first determining module 41 further includes:
[0165] The normalization unit is used to normalize features of multiple dimensions.
[0166] Further, in another possible implementation form of the present application, the position information of each detection frame includes the coordinates of each detection frame in the first direction and the offset in the second direction; accordingly, the second determination module 42 specifically includes:
[0167] A third acquisition unit is used to acquire, from each detection frame, a candidate detection frame whose second coordinate in the first direction matches the first coordinate, according to the first coordinate of any detection frame in the first direction when the type of any detection frame is the beginning of a text line;
[0168] A fourth acquisition unit, configured to acquire, from the candidate detection frames, an adjacent detection frame adjacent to any detection frame in the second direction according to a first offset of any detection frame in the second direction;
[0169] The merging unit is used to merge the adjacent detection box with any detection box when the type of the adjacent detection box is a character.
[0170] Furthermore, in another possible implementation form of the present application, the above-mentioned image text extraction device 40 combining RPA and AI further includes:
[0171] A fourth determination module is used to perform a connected domain analysis on each text box to determine a connected domain shape corresponding to each text box;
[0172] The fifth determination module is used to determine that any text box contains a red seal when the shape of the connected domain corresponding to any text box is a circle.
[0173] It should be noted that the above Figure 1 , Figure 3 , Figure 4 The explanation of the embodiment of the method for extracting image text combining RPA and AI shown is also applicable to the image text extraction device 40 combining RPA and AI of this embodiment, and will not be repeated here.
[0174] The device for extracting text from an image that combines RPA and AI provided in the embodiment of the present application extracts multiple dimensional features of each detection frame from the image to be processed, and performs attention mechanism learning on the multiple dimensional features to obtain the adjacent frame information of each detection frame and the beginning and end information of the text line corresponding to each detection frame, and then determines the type of each detection frame based on the adjacent frame information of each detection frame and the beginning and end information of the corresponding text line, and then merges the detection frames of type characters based on the position information of each detection frame and the type of each detection frame to determine the text frames contained in the image to be processed, and performs text recognition on each text frame to determine the text contained in the image to be processed. Thus, by extracting multi-dimensional features of different granularities of the image to be processed and performing feature representation on each detection frame in the image to be processed, not only can different types of data content in the image be determined through one detection, but also the process of image text extraction is simplified, the efficiency of text extraction is improved, and the accuracy and reliability of text extraction are further improved.
[0175] In order to implement the above embodiments, the present application also proposes an electronic device.
[0176] Figure 6 The figure is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention.
[0177] like Figure 6 As shown, the electronic device 200 includes:
[0178] The memory 210 and the processor 220, a bus 230 connecting different components (including the memory 210 and the processor 220), the memory 210 stores a computer program, and when the processor 220 executes the program, the method for extracting image text combining RPA and AI described in the embodiment of the present application is implemented.
[0179] Bus 230 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.
[0180] The electronic device 200 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by the electronic device 200, including volatile and non-volatile media, removable and non-removable media.
[0181] The memory 210 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 240 and / or cache memory 250. The electronic device 200 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 260 may be used to read and write non-removable, non-volatile magnetic media ( Figure 6 not shown, usually called a "hard drive"). Although Figure 6 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical medium) may be provided. In these cases, each drive may be connected to bus 230 via one or more data medium interfaces. Memory 210 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present application.
[0182] A program / utility 280 having a set (at least one) of program modules 270 may be stored, for example, in the memory 210, such program modules 270 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 270 generally perform the functions and / or methods of the embodiments described herein.
[0183] The electronic device 200 may also communicate with one or more external devices 290 (e.g., keyboard, pointing device, display 291, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 200, and / or communicate with any device that enables the electronic device 200 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed via an input / output (I / O) interface 292. Furthermore, the electronic device 200 may also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and / or public network, such as the Internet) via a network adapter 293. As shown, the network adapter 293 communicates with other modules of the electronic device 200 via a bus 230. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0184] The processor 220 executes various functional applications and data processing by running the programs stored in the memory 210 .
[0185] It should be noted that the implementation process and technical principles of the electronic device of this embodiment refer to the aforementioned explanation of the image text extraction method combining RPA and AI in the embodiment of the present application, and will not be repeated here.
[0186] The electronic device provided in the embodiment of the present application can execute the image text extraction method combining RPA and AI as described above, by performing target detection on the image to be processed to determine the position information of each detection frame contained in the image to be processed and the type of each detection frame, and according to the position information of each detection frame and the type of each detection frame, the detection frames of the character type are merged to determine the text frames contained in the image to be processed, and then the text recognition is performed on each text frame to determine the text contained in the image to be processed. Thus, by determining the type of each detection frame while performing target detection on the image to be processed, different types of data content in the image can be determined through a single detection, thereby simplifying the process of image text extraction and improving the efficiency of text extraction.
[0187] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium.
[0188] Among them, the computer-readable storage medium stores a computer program, and when the program is executed by the processor, it implements the method for extracting image text combining RPA and AI as described in the embodiment of the present application.
[0189] In order to implement the above-mentioned embodiments, another embodiment of the present application provides a computer program, which, when executed by a processor, implements the method for extracting image text combining RPA and AI as described in the embodiment of the present application.
[0190] In an optional implementation form, this embodiment can use any combination of one or more computer-readable media. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.
[0191] Computer-readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0192] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0193] Computer program code for performing the operation of the present application can be written in one or more programming languages or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user electronic device, partially on the user electronic device, as an independent software package, partially on the user electronic device and partially on the remote electronic device, or entirely on the remote electronic device or server. In the case of a remote electronic device, the remote electronic device can be connected to the user electronic device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external electronic device (e.g., using an Internet service provider to connect through the Internet).
[0194] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the inventions claimed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary techniques in the art that are not invented by the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.
[0195] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. A method for extracting text from images by combining RPA and AI. It is characterized in that include: Performing target detection on the image to be processed to determine the position information of each detection frame contained in the image to be processed and the type of each detection frame, wherein the type of each detection frame includes: character, non-character, beginning of a text line and end of a text line; According to the position information of each detection frame and the type of each detection frame, the detection frames of the character type are merged to determine each text frame included in the image to be processed; Performing text recognition on each of the text boxes to determine the text contained in the image to be processed; The performing target detection on the image to be processed to determine the position information of each detection frame contained in the image to be processed and the type of each detection frame specifically includes: Extracting multiple dimensional features of each detection frame from the image to be processed; Performing attention mechanism learning on the multiple dimensional features to obtain adjacent frame information of each detection frame and the beginning and end information of the text line corresponding to each detection frame; The type of each detection frame is determined according to the adjacent frame information of each detection frame and the corresponding text line beginning and end information.
2. The method according to claim 1, It is characterized in that Before extracting the multiple dimensional features of each detection frame from the image to be processed, the method further includes: Preprocessing the image to be processed to obtain a plurality of feature maps corresponding to the image to be processed; The extracting multiple dimensional features of each detection frame from the image to be processed specifically includes: Multiple dimensional features of each of the detection boxes are extracted from the multiple feature maps.
3. The method according to claim 1, It is characterized in that The extracting multiple dimensional features of each detection frame from the image to be processed specifically includes: The image to be processed is convolved using at least two filters to obtain at least two dimensional features of each of the detection frames, wherein the receptive fields of the at least two filters are different.
4. The method according to claim 1, It is characterized in that Before performing attention mechanism learning on the multiple dimensional features to obtain adjacent frame information of each detection frame and the beginning and end information of the text line corresponding to each detection frame, the method further includes: The multiple dimensional features are concatenated to generate features of each detection frame.
5. The method according to any one of claims 1 to 4, It is characterized in that Before performing attention mechanism learning on the multiple dimensional features to obtain adjacent frame information of each detection frame and the beginning and end information of the text line corresponding to each detection frame, the method further includes: The multiple dimensional features are normalized.
6. The method according to any one of claims 1 to 4, It is characterized in that The position information of each detection frame includes the coordinates of each detection frame in a first direction and the offset in a second direction, and the merging of the detection frames of the character type according to the position information of each detection frame and the type of each detection frame to determine the text frames contained in the image to be processed specifically includes: If the type of any detection frame is the beginning of a text line, obtaining, from the respective detection frames, a candidate detection frame whose second coordinate in the first direction matches the first coordinate, according to the first coordinate of the any detection frame in the first direction; According to the first offset of any detection frame in the second direction, acquiring, from the candidate detection frames, an adjacent detection frame adjacent to any detection frame in the second direction; If the type of the adjacent detection frame is a character, the adjacent detection frame is merged with any one of the detection frames.
7. The method according to any one of claims 1 to 4, It is characterized in that After merging the detection frames of character type according to the position information of each detection frame and the type of each detection frame to determine each text frame included in the image to be processed, the method further includes: Performing a connected domain analysis on each of the text boxes to determine a connected domain shape corresponding to each of the text boxes; If the shape of the connected domain corresponding to any text box is a circle, it is determined that any text box contains a red seal.
8. An image text extraction device combining RPA and AI, It is characterized in that include: A first determination module is used to perform target detection on the image to be processed to determine the position information of each detection frame contained in the image to be processed and the type of each detection frame, wherein the type of each detection frame includes: character, non-character, beginning of a text line and end of a text line; A second determination module is used to merge the detection frames of the character type according to the position information of each detection frame and the type of each detection frame to determine each text frame included in the image to be processed; A third determination module, configured to perform text recognition on each of the text boxes to determine the text contained in the image to be processed; The first determining module specifically includes: An extraction unit, used to extract multiple dimensional features of each detection frame from the image to be processed; A first acquisition unit is used to perform attention mechanism learning on the multiple dimensional features to obtain adjacent frame information of each detection frame and the beginning and end information of the text line corresponding to each detection frame; A determination unit is used to determine the type of each detection frame according to the adjacent frame information of each detection frame and the corresponding text line beginning and end information.
9. The device as claimed in claim 8, It is characterized in that The first determining module further includes: A second acquisition unit, used for preprocessing the image to be processed to obtain a plurality of feature maps corresponding to the image to be processed; The extraction unit specifically comprises: The extraction subunit is used to extract multiple dimensional features of each detection box from the multiple feature maps.
10. The device according to claim 8, It is characterized in that The extraction unit specifically comprises: The acquisition subunit is used to perform convolution processing on the image to be processed using at least two filters to obtain at least two dimensional features of each of the detection frames, wherein the receptive fields of the at least two filters are different.
11. The device according to claim 8, It is characterized in that The first determining module further includes: A splicing unit is used to splice the multiple dimensional features to generate features of each detection frame.
12. The device according to any one of claims 8 to 11, It is characterized in that The first determining module further includes: A normalization unit is used to perform normalization processing on the multiple dimensional features.
13. The device according to any one of claims 8 to 11, It is characterized in that The position information of each detection frame includes the coordinates of each detection frame in the first direction and the offset in the second direction. The second determination module specifically includes: A third acquisition unit is used for acquiring, when the type of any detection frame is the beginning of a text line, from the detection frames according to the first coordinate of any detection frame in the first direction, a candidate detection frame whose second coordinate in the first direction matches the first coordinate; A fourth acquisition unit, configured to acquire, from the candidate detection frames, an adjacent detection frame adjacent to the any detection frame in the second direction according to the first offset of the any detection frame in the second direction; A merging unit is used to merge the adjacent detection box with any one of the detection boxes when the type of the adjacent detection box is a character.
14. The device according to any one of claims 8 to 11, It is characterized in that Also includes: A fourth determination module is used to perform a connected domain analysis on each of the text boxes to determine a connected domain shape corresponding to each of the text boxes; The fifth determining module is used to determine that any text box contains a red seal when the shape of the connected domain corresponding to any text box is a circle.
15. An electronic device, It is characterized in that include: A memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for extracting image text combining RPA and AI as described in any one of claims 1 to 7 is implemented.
16. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the method for extracting image text combining RPA and AI as described in any one of claims 1 to 7 is implemented.
