Two-stage electric power operation site document identification method and system

Abstract

The embodiment of the invention provides a two-stage electric power operation site document identification method and system, and belongs to the field of image processing. The recognition method comprises the following steps: acquiring various job document images; constructing a text information acquisition model; inputting the various job document images into the text information acquisition model to acquire text and layout information; the method comprises the following steps of: constructing a multi-mode Transform encoder; inputting the text and the layout information into the multi-mode Transform encoder so as to obtain structured data, and inputting the text and the layout information into the multi-mode Transform encoder so as to obtain the structured data; constructing an electric power operation field document identification and analysis model; and inputting the structured data into the electric power operation site document identification and analysis model to train the document identification and analysis model. According to the method, different types of document images can be efficiently processed, text information can be extracted, the layout structure of the document can be recognized, and therefore the accuracy and the automation degree of electric power operation site document analysis are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a two-stage power operation site document recognition method and system. BACKGROUND

[0002] With the rapid development of the power industry, the demand for document processing in power operation sites has gradually increased. Traditional manual document recognition methods are not only inefficient, but also susceptible to human factors, resulting in high error rates. Especially in power operation sites, there are various types of documents with complex formats, involving a large amount of images, texts, and layout information. Therefore, how to efficiently and accurately automate the processing of these documents has become a problem that needs to be solved in the power industry.

[0003] In recent years, with the continuous progress of artificial intelligence and deep learning technology, document automated analysis methods based on image recognition and natural language processing have gradually become a research hotspot. In particular, the application of deep learning models such as Transformer can extract complex features in image and text data, improving recognition accuracy. However, existing technologies still have problems such as incomplete text information extraction, difficulty in recognizing layout structure, and lack of context information when processing multi-modal documents. Therefore, how to effectively combine text, image, and layout information and propose a document recognition method with high robustness and accuracy has become a key to industry development. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a two-stage power operation site document recognition method and system, which solves the problem of structured information extraction for complex format documents.

[0005] To achieve the above-mentioned purpose, the embodiments of the present application provide a two-stage power operation site document recognition method, which comprises: acquiring various types of operation document images; constructing a text information acquisition model; inputting the various types of operation document images into the text information acquisition model to obtain text and layout information; constructing a multi-modal Transformer encoder; inputting the text and layout information into the multi-modal Transformer encoder to obtain structured data; constructing a power operation site document recognition analysis model; inputting the structured data into the power operation site document recognition analysis model to train the document recognition analysis model.

[0006] Optionally, the text information acquisition model is constructed, comprising: The Vision Transform acquirer is used as the visual encoder; The PaddleOCR model was used for text detection training. CRNN is used as a text recognizer for text recognition training.

[0007] Optionally, a Vision Transform acquirer is used as the visual encoder, including: The image is divided into blocks to obtain image patches; The image block is added element by element to the location encoding vector to obtain an input sequence with location information.

[0008] Optionally, text detection training based on the DB algorithm includes: The image is input into a CNN layer for multi-layer feature extraction to obtain feature maps at different resolutions; The DB algorithm is used to combine the feature maps using a strategy of layer-by-layer up-and-down addition; Two prediction branches are introduced, which include a probability graph and a threshold graph; A binary text region map is generated based on the probability map and the threshold map; A post-processing algorithm is applied to the binarized text region map to generate the final text detection box.

[0009] Optionally, generating a binarized text region map based on the probability map and the threshold map includes: Determine whether the pixel probability of the image is higher than a first preset threshold. If the probability of the pixel is higher than a first preset threshold, the image is determined to be a text region; If the probability of the pixel is not higher than a first preset threshold, the image is determined to be the background.

[0010] Optionally, a CRNN is used as a text recognizer for text recognition training, including: The image is transformed into a two-dimensional feature map by entering the convolutional layer; The feature map is fed into a recursive layer for feature extraction to output a predicted probability distribution; The predicted probability distribution is fed into the transcription layer to be transformed into a text sequence.

[0011] Optionally, a multimodal Transformer encoder is constructed, including: Obtain the correct order of document pages based on the document's page number information; Correct the text position on the current page according to the correct order of the document pages; Sort all text according to the corrected current page text position; The images, text, and modified layout information are converted into feature sequences. The Transformer encoder is used to encode the feature sequence to combine the encoded image, text, and layout information.

[0012] Optionally, correcting the text position on the current page according to the correct order of the document pages includes: Calculate the average line spacing per page of the document according to formula (1). (1) Adjust the vertical coordinate of the text position according to formula (2). (2) in, For average line spacing, For the first Page line spacing, For page number, Total number of pages This represents the vertical coordinate of the current text position. The vertical coordinate of the adjusted text position. for The maximum vertical coordinate of the text was detected in the page document.

[0013] On the other hand, the present invention also provides a two-stage power operation site document recognition system, the system including a processor for performing the text recognition method as described above.

[0014] In another aspect, the present invention also provides a computer-readable storage medium storing instructions for being read by a computer to cause the computer to perform any of the identification methods described above. Through the above technical solution, this invention provides a two-stage method and system for identifying power operation site documents. First, scene text recognition technology is used to obtain all text information, including text content and location information, from various document images. Then, key information extraction technology is combined to obtain structured data, based on which a power operation site document recognition and analysis model is constructed. This multimodal approach can efficiently process different types of document images, not only extracting text information but also recognizing the document's layout and structure, thereby improving the accuracy and automation of power operation site document analysis.

[0015] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of an identification method according to an embodiment of the present invention; Figure 2 This is a flowchart of constructing a text information acquisition model according to an embodiment of the present invention; Figure 3 This is a flowchart of data processing according to one embodiment of the present invention; Figure 4 This is a flowchart of text detection training based on the DB algorithm according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a text detector according to an embodiment of the present invention; Figure 6 This is a flowchart of obtaining a text region map according to an embodiment of the present invention; Figure 7 This is a flowchart of text recognition training according to one embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of a text recognizer according to an embodiment of the present invention; Figure 9 This is a flowchart of information extraction according to one embodiment of the present invention; Figure 10 This is a flowchart illustrating the correction of the current page text position according to an embodiment of the present invention. Detailed Implementation

[0017] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0018] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application all comply with relevant laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0019] Figure 1 This is a flowchart of an identification method according to an embodiment of the present invention, in which the identification method includes: In step S1, images of various job documents are acquired.

[0020] In step S2, a text information acquisition model is constructed.

[0021] In step S3, various job document images are input into the text information acquisition model to obtain text and layout information.

[0022] In step S4, a multimodal Transformer encoder is constructed.

[0023] In step S5, the text and layout information are input into the multimodal Transformer encoder to obtain structured data.

[0024] In step S6, a document recognition and analysis model for power operation sites is constructed.

[0025] In step S7, structured data is input into the power operation site document recognition and analysis model to train the document recognition and analysis model.

[0026] In steps S1 to S7, scene text recognition technology is first used to obtain all text information in the work document images, including text content and location information. Then, key information extraction technology is combined to obtain structured data, based on which a power work site document recognition and analysis model is constructed. This multimodal method can efficiently process different types of document images, not only extracting text information but also recognizing the document layout structure, thereby improving the accuracy and automation of power work site document analysis.

[0027] In this embodiment, the methods for constructing the text information acquisition model can be various and known to those skilled in the art. In one example of the present invention, the method for constructing the text information acquisition model can be as follows: Figure 2 The method shown herein specifically includes the following steps: In step S21, the Vision Transform acquirer is used as the visual encoder.

[0028] In step S22, the PaddleOCR model is used for text detection training.

[0029] In step S23, CRNN is used as a text recognizer for text recognition training.

[0030] In steps S21 to S23, the entire process efficiently and accurately extracts text information from documents through ViT's visual encoder, PaddleOCR's text detection, and CRNN's text recognition. First, the text detection algorithm detects text regions in the document. Then, the character or sequence recognition module identifies each character within the text region, ultimately forming a complete text sequence. This invention demonstrates strong robustness in handling complex backgrounds, different fonts, and multilingual documents, and is widely applicable to automated document processing.

[0031] To transform the image into a sequence-like form, making it compatible with the Transformer's ability to process sequential data, this embodiment employs a Vision Transformer as the visual encoder. The data processing methods can be various those known to those skilled in the art. In one example of the invention, the method of using a Vision Transformer as the visual encoder for data processing can be... Figure 3 The method shown may specifically include the following steps: In step S211, the image is divided into blocks to obtain image blocks.

[0032] In step S212, the image block is added element by element to the position encoding vector to obtain an input sequence with position information.

[0033] In steps S211 to S212, ViT does not directly perform convolution operations on each pixel of the image. Instead, it divides the input image into several fixed-size image blocks. The purpose of this operation is to transform the image into a sequence-like form, making it compatible with the Transformer's ability to process sequential data.

[0034] Considering the need to locate text regions in the image, this implementation uses the DB algorithm for text detection training. The steps for text training and detection can be varied and known to those skilled in the art. In one example of this invention, the steps for text training and detection can be... Figure 4 The method shown is illustrated. Figure 4 The identification method also includes: In step S221, the image is input into a CNN layer for multi-layer feature extraction to obtain feature maps at different resolutions. Specifically, the input image first undergoes multi-layer feature extraction via a CNN, producing a series of feature maps at different resolutions, progressively reduced to 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image. These feature maps capture semantic information at different levels.

[0035] In step S222, the DB algorithm is used to combine the feature maps using a strategy of layer-by-layer application and layer-by-layer addition. Specifically, in order to combine feature maps of different resolutions, the DB algorithm uses a strategy of layer-by-layer application and layer-by-layer addition, ultimately generating a 1 / 4 resolution feature map.

[0036] In step S223, two prediction branches are introduced, including a probability map and a threshold map. Specifically, on the fused high-resolution feature map, two prediction branches are introduced: a probability map, which represents the probability value of each pixel in the image belonging to the text region; and a threshold map, which learns a dynamic binarization threshold to determine the boundaries of the text region.

[0037] In step S224, a binarized text region map is generated based on the probability map and the threshold map.

[0038] In step S225, a post-processing algorithm is applied to the binarized text region map to generate the final text detection box.

[0039] In steps S221 to S225, in PaddleOCR, differentiable binarization (DB) is used as a text detector based on a simple segmentation network. A detailed structure diagram is shown below. Figure 5 As shown. To further improve its effectiveness and efficiency, the following six strategies were used: optical backbone, optical head, removal of SE modules, cosine learning rate decay, learning rate preheating, and FPGM pruning. Finally, the model size of the text detector was reduced by 1.4M.

[0040] In this embodiment, the methods for obtaining the text region map can be various and known to those skilled in the art. In one example of the present invention, the method for obtaining the text region map can be as follows: Figure 6 The method shown is illustrated. Figure 6 The identification method also includes: In step S2241, it is determined whether the pixel probability of the image is higher than a first preset threshold. If the pixel probability is higher than the first preset threshold, step S2242 is executed; otherwise, step S2243 is executed.

[0041] In step S2242, the image is determined to be a text region.

[0042] In step S2243, the image is determined to be the background.

[0043] In steps S2241 to S2243, a binarized text region map is generated by combining the probability map and the threshold map. If the probability value of a pixel is higher than the corresponding threshold, it is determined to be a text region; otherwise, it is determined to be background.

[0044] In this embodiment, the steps for performing text recognition training can be of various kinds known to those skilled in the art. In one example of the present invention, the steps for performing text recognition training can be... Figure 7 The method shown is illustrated. Figure 7 The identification method also includes: In step S231, the image is converted into a two-dimensional feature map by entering the convolutional layer.

[0045] In step S232, the feature map enters the recursive layer for feature extraction to output the predicted probability distribution.

[0046] In step S233, the predicted probability distribution is fed into the transcription layer to be transformed into a text sequence.

[0047] In steps S231 to S233, the PaddleOCR algorithm uses CRNN as the text recognizer, and the detailed structure is as follows: Figure 8 As shown, CRNN integrates feature extraction and sequence modeling, avoiding inconsistencies between predictions and labels. To enhance model capabilities and reduce the model size of the text recognizer, the following nine strategies were used: bare backbone, data augmentation, cosine learning rate decay, feature map resolution, regularization parameters, learning rate warm-up, bare head, pre-trained model, and PACT quantization. Finally, the model size of the text recognizer for Chinese and English text recognition is only 1.6M, and for alphanumeric symbol recognition it is only 900kB.

[0048] After extracting image text and layout information, the next stage is the extraction of key information. Various methods for information extraction are known to those skilled in the art. In one example of this invention, the information extraction method could be... Figure 9 The method shown is illustrated. Figure 9 The identification method also includes: In step S41, the correct order of the document pages is obtained based on the document's page number information.

[0049] In step S42, the text position on the current page is corrected according to the correct order of the document pages.

[0050] In step S43, all text is sorted according to the corrected text position on the current page.

[0051] In step S44, the image, text, and modified layout information are converted into feature sequences.

[0052] In step S45, the feature sequence is encoded using a Transformer encoder to combine the encoded image, text, and layout information.

[0053] In steps S41 to S45, since documents such as safety briefing cards may have multiple pages, tables and paragraphs may be split across different pages. Treating each page as an independent entity to extract key information will result in the loss of a lot of contextual information. To avoid this situation, it is necessary to correct the text position on the current page according to the correct order of the document pages, convert the image, text, and modified layout information into feature sequences, encode the feature sequences using a Transformer encoder, and use a multi-task head to combine the encoded image, text, and layout information for multi-task learning, thereby achieving key information extraction.

[0054] To obtain the correct order of document pages, this implementation requires correcting the text position on the current page according to the correct document page order. Specifically, this can be done as follows: Figure 9 The method shown is illustrated. Figure 9 The identification method also includes: In step S421, the average line spacing of each page of the document is calculated according to formula (1). (1) In step S422, the vertical coordinate of the text position is adjusted according to formula (2). (2) in, For average line spacing, For the first Page line spacing, For page number, Total number of pages This represents the vertical coordinate of the current text position. The vertical coordinate of the adjusted text position. for The maximum vertical coordinate of the text was detected in the page document.

[0055] On the other hand, the present invention also provides a two-stage power operation site document recognition system, the system including a processor for executing any of the text recognition methods described above.

[0056] In another aspect, the present invention also provides a computer-readable storage medium storing instructions for being read by a computer to cause the computer to perform any of the identification methods described above.

[0057] Through the above technical solution, this invention provides a two-stage method and system for identifying power operation site documents. First, scene text recognition technology is used to obtain all text information, including text content and location information, from various document images. Then, key information extraction technology is combined to obtain structured data, based on which a power operation site document recognition and analysis model is constructed. This multimodal approach can efficiently process different types of document images, not only extracting text information but also recognizing the document's layout and structure, thereby improving the accuracy and automation of power operation site document analysis.

[0058] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0059] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0060] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0061] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0062] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0063] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0064] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0065] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0066] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A two-stage method for identifying on-site documents in power operations, characterized in that, The identification method includes: Acquire images of various assignment documents; Construct a text information retrieval model; The images of the various types of assignment documents are input into the text information acquisition model to obtain text and layout information; Construct a multimodal Transformer encoder; The text and layout information are input into the multimodal Transformer encoder to obtain structured data; Construct a document recognition and analysis model for power operation sites; The structured data is input into the power operation site document recognition and analysis model to train the document recognition and analysis model.

2. The identification method according to claim 1, characterized in that, Constructing a text information retrieval model includes: The Vision Transform acquirer is used as the visual encoder; The PaddleOCR model was used for text detection training. CRNN is used as a text recognizer for text recognition training.

3. The identification method according to claim 2, characterized in that, The Vision Transform acquirer is used as the visual encoder, including: The image is divided into blocks to obtain image patches; The image block is added element by element to the location encoding vector to obtain an input sequence with location information.

4. The identification method according to claim 2, characterized in that, Text detection training based on the DB algorithm includes: The image is input into a CNN layer for multi-layer feature extraction to obtain feature maps at different resolutions; The DB algorithm is used to combine the feature maps using a strategy of layer-by-layer up-and-down addition; Two prediction branches are introduced, which include a probability graph and a threshold graph; A binary text region map is generated based on the probability map and the threshold map; A post-processing algorithm is applied to the binarized text region map to generate the final text detection box.

5. The identification method according to claim 3, characterized in that, Generate a binarized text region map based on the probability map and the threshold map, including: Determine whether the pixel probability of the image is higher than a first preset threshold. If the probability of the pixel is higher than a first preset threshold, the image is determined to be a text region; If the probability of the pixel is not higher than a first preset threshold, the image is determined to be the background.

6. The identification method according to claim 2, characterized in that, Using CRNN as a text recognizer for text recognition training includes: The image is transformed into a two-dimensional feature map by entering the convolutional layer; The feature map is fed into a recursive layer for feature extraction to output a predicted probability distribution; The predicted probability distribution is fed into the transcription layer to be transformed into a text sequence.

7. The identification method according to claim 1, characterized in that, Constructing a multimodal Transformer encoder includes: Obtain the correct order of document pages based on the document's page number information; Correct the text position on the current page according to the correct order of the document pages; Sort all text according to the corrected text position on the current page; The images, text, and modified layout information are converted into feature sequences. The Transformer encoder is used to encode the feature sequence to combine the encoded image, text, and layout information.

8. The identification method according to claim 7, characterized in that, Correcting the text position on the current page according to the correct order of the document pages includes: Calculate the average line spacing per page of the document according to formula (1). ，（1） Adjust the vertical coordinate of the text position according to formula (2). ，（2） in, For average line spacing, For the first Page line spacing, For page number, Total number of pages This represents the vertical coordinate of the current text position. The vertical coordinate of the adjusted text position. for The maximum vertical coordinate of the text was detected in the page document.

9. A two-stage power operation site document recognition system, characterized in that, The system includes a processor for performing the text recognition method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The storage medium stores instructions that are read by a computer to cause the computer to perform the identification method as described in any one of claims 1 to 8.

Images