Drawing text information extraction method, device and equipment

Abstract

The application relates to a drawing text information extraction method, device and equipment. For a to-be-processed drawing, the to-be-processed drawing is regionally divided based on the difference of drawing information expression modes, and a division result of a text description region and a table region is obtained; based on the possible difference of text semantic association features of each region, different text semantic relationship analysis technologies are used for processing respectively; then, comprehensive processing is performed based on the respective output results of the two technologies, and a final text information extraction result of the to-be-processed drawing is obtained, efficient and accurate extraction of key information needing to be audited in the building drawing is realized, and the drawing auditing efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of text recognition and processing, and in particular to methods, apparatus, and equipment for extracting text information from drawings. Background Technology

[0002] In the field of engineering construction, architectural drawings are not used directly for construction after they are completed. They need to be submitted to relevant review departments for review to check whether the design drawings meet the key points of relevant building industry specifications or other relevant requirements. Only architectural drawings that pass the review will be used for production and construction.

[0003] Purely manual drawing review relies heavily on the reviewers' experience and skills, and is also inefficient, especially for large-scale engineering projects with numerous drawings, making it time-consuming and labor-intensive. Therefore, a semi-automated drawing review approach is currently preferred. This semi-automated review involves extracting and identifying key information from the drawings, then outputting this information as a QA (Question and Answer) pair, automatically determining whether it meets relevant specifications, or presenting the information to reviewers for manual review or verification, thereby reducing the workload of manual drawing review.

[0004] Currently, important information in drawings is mainly extracted through text recognition OCR (optical character recognition) and regular expressions. However, the layout styles of text areas and table areas in drawings are numerous, and regular expression standards need to be prepared in advance. Moreover, it is theoretically impossible to exhaust all layout styles, which means that regular expressions cannot effectively extract text information from certain styles of drawings. At the same time, preparing regular expression standards in the early stage is also a time-consuming and laborious process. Summary of the Invention

[0005] To address the current issues of ineffective extraction of text information from drawings of certain styles during the review process, as well as the time-consuming and labor-intensive nature of such extractions, this application provides a method, apparatus, and equipment for extracting text information from drawings.

[0006] Firstly, the method for extracting text information from drawings provided in this application adopts the following technical solution:

[0007] A method for extracting text information from drawings, comprising:

[0008] Obtain the region division results of the drawing to be processed; the regions include text description regions and table regions;

[0009] The first data to be processed corresponding to the text description area is input into the NLP (Natural Language Processing) algorithm model, and the first extracted information containing several question-answer pairs is output; the first data to be processed includes the data in the text description area that corresponds to the set first data type;

[0010] The second data to be processed corresponding to the table area is input into the multimodal algorithm model, and the second extracted information containing several question-answer pairs is output; the second data to be processed includes the data in the table area corresponding to the set second data type;

[0011] Based on the first and second extracted information, the final extracted text information of the drawing to be processed is obtained and displayed.

[0012] By adopting the above technical solution, the drawings to be processed are divided into regions based on the differences in the way the information is expressed, resulting in the division of text description areas and table areas. Based on the possible differences in the semantic association features of the text in each region, different text semantic relationship parsing techniques are used for processing. Then, the output results of the two techniques are combined to obtain the final extracted text information of the drawings. This enables the efficient and accurate extraction of key information (i.e., question-answer pairs) that needs to be reviewed in architectural drawings, which is beneficial to improving the efficiency of drawing review.

[0013] Optionally, the model architecture of the NLP algorithm model includes m LSTM (long short-term memory) networks, m Multi-Head Attention networks, m Layernorm normalization networks, m Conv1*1 convolutional kernels, and m ADD feature fusion networks; where m is a natural number greater than or equal to 2.

[0014] Optionally, the model architecture of the NLP algorithm model consists of m parsing modules connected in series. Each parsing module includes an LSTM long short-term memory network, a Multi-Head Attention network, a Layernorm normalization network, and a Conv1*1 convolutional kernel connected in series. The output of the LSTM long short-term memory network is also transmitted to the Conv1*1 convolutional kernel without difference through an ADD feature fusion network. The m∈[5,15].

[0015] Optionally, the multimodal algorithm model includes an encoder consisting of a Transformer block module, a Mobile-ViT (Mobile Vision Transformer) network, a PAN (Pyramid Attention Network for Semantic Segmentation) semantic segmentation network, and a Concat feature fusion network, and a decoder consisting of a Concat feature fusion network, a Bi-LSTM bidirectional long short-term memory network, and a CRF layer.

[0016] Optionally, the encoder includes two branches, one of which is composed of n Tranformer block modules connected in series; the other branch is composed of a Mobile-ViT network and a PAN semantic segmentation network; the outputs of the two branches are fused and encoded by the Concat feature fusion network.

[0017] Optionally, the decoder sequentially processes the output of the Concat feature fusion network through K Bi-LSTM (Bi-directional Long Short-Term Memory, which is composed of forward LSTM and backward LSTM and can capture bidirectional semantic dependencies) networks and P CRF layers to obtain the second extracted information; where K is a natural number greater than or equal to 2, and P is a natural number greater than or equal to 2; K is equal to P.

[0018] Optionally, the first data type setting includes a text data type, and the second data type setting includes a text data type and a table layout image data type.

[0019] Optionally, the final extraction result of the text information of the drawing to be processed based on the first extraction information and the second extraction information includes:

[0020] For each question-answer pair in the second extracted information, determine the first cell coordinates corresponding to the question and the second cell coordinates corresponding to the answer;

[0021] Calculate the difference in row coordinates and the difference in column coordinates between the coordinates of the first cell and the coordinates of the second cell;

[0022] Determine whether the sum of the row coordinate difference and the column coordinate difference is less than a set threshold;

[0023] If yes, the question-answer pair is considered normal; if no, the question-answer pair is considered abnormal and marked as abnormal.

[0024] The questions and answers in the first and second extracted information are divided into normal question-answer pairs and abnormal question-answer pairs. The final extracted text information of the drawing to be processed is then output and displayed.

[0025] By adopting the above technical solution, further testing of the question answer output by the multimodal algorithm model based on cell coordinates can improve the accuracy of the final extraction results.

[0026] Secondly, the drawing text information extraction device provided in this application adopts the following technical solution:

[0027] A drawing text information extraction device, comprising:

[0028] The acquisition module is used to acquire the area division results of the drawing to be processed; the area includes a text description area and a table area.

[0029] An NLP algorithm model is used to predict and process the first data to be processed corresponding to the text description area, and output the first extracted information containing several question-answer pairs; the first data to be processed includes the data in the text description area that corresponds to a set first data type.

[0030] A multimodal algorithm model is used to predict and process the second data to be processed corresponding to the table area, and output second extracted information containing several question-answer pairs; the second data to be processed includes data in the table area corresponding to a set second data type;

[0031] The integrated processing module is used to obtain the final extraction result of the text information of the drawing to be processed based on the first extracted information and the second extracted information, and output and display it.

[0032] Thirdly, the drawing text information extraction device provided in this application adopts the following technical solution:

[0033] A drawing text information extraction device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described drawing text information extraction method.

[0034] In summary, this application includes at least the following beneficial technical effects:

[0035] 1. It enables the efficient and accurate extraction of key information that needs to be reviewed from architectural drawings, which helps improve the efficiency of drawing review.

[0036] 2. Further testing the question answers output by the multimodal algorithm model based on cell coordinates helps improve the accuracy of the final extraction results. Attached Figure Description

[0037] Figure 1 This is a flowchart of the method for extracting text information from drawings in the embodiments of this application;

[0038] Figure 2 This is a block diagram of the NLP algorithm model architecture in the embodiments of this application;

[0039] Figure 3 This is a block diagram of the multimodal algorithm model architecture in the embodiments of this application;

[0040] Figure 4 This is a schematic diagram of the text description area in the drawings of this application embodiment;

[0041] Figure 5 This is a schematic diagram of the table area in the drawings of this application embodiment;

[0042] Figure 6 This is a structural block diagram of the drawing text information extraction device in the embodiments of this application;

[0043] Figure 7 This is a structural block diagram of the drawing text information extraction device in the embodiments of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0045] This application discloses a method for extracting text information from drawings, which can be implemented through a software system that can be installed and run on hardware devices. (See reference...) Figure 1 The method for extracting text information from this drawing mainly includes the following steps:

[0046] S101. Obtain the area division results of the drawing to be processed.

[0047] In an optional embodiment of this application, the drawing to be processed supports multiple input formats, such as CAD format, PDF format, or image format.

[0048] Drawings contain a fusion of different types of data. A single line drawing is insufficient to fully express a designer's ideas or design content; other forms of expression are necessary. For example, information such as fire resistance rating, room noise levels, and total building area is difficult to illustrate visually. Therefore, architects typically use explanatory frames with textual descriptions or tables to present the information. This makes it easier to convey crucial information about the architectural design.

[0049] In an optional embodiment of this application, the drawing to be processed is divided into regions based on the different ways in which the drawing information is expressed. This allows for better extraction of text information using more suitable parsing techniques based on the differences in the information expression attributes of each region, thereby improving the accuracy and efficiency of text information extraction.

[0050] In this embodiment, the area of ​​the drawing to be processed may include a text description area, a table area, and a line image area. It should be understood that the area division results may differ depending on the drawing to be processed, that is, the information expression method may be different.

[0051] It should be noted that the division of the drawing area can be achieved using any existing technology, such as image recognition, table recognition, or text recognition, or a combination thereof, without any restrictions. This application mainly focuses on the targeted extraction and processing of text information based on the obtained area division results.

[0052] S102. Input the first data to be processed corresponding to the text description area into the NLP algorithm model, and output the first extracted information containing several question answer pairs; wherein the first data to be processed includes the data in the text description area that corresponds to the set first data type.

[0053] In this embodiment of the application, an NLP algorithm model is used to extract text semantics from the data in the text description area to achieve the classification and matching output of the question answer.

[0054] The first data type is set to a text data type (e.g., text format), meaning the first data to be processed is the text data contained in the text description area. This first data to be processed can be obtained by performing text recognition on the text description area of ​​the drawing to be processed.

[0055] In order to more efficiently identify and obtain text data, in an optional embodiment of this application, the text data can be obtained by calling the CAD element export interface to achieve rapid identification and export of elements.

[0056] It should be understood that points, lines, arcs, splines, text, etc. can all be called primitives. In CAD, all individual objects are considered primitives. A primitive can be a line segment (line), a circle (c), an arc segment (arc), a polyline (pl), a single-line text (dtext), a multi-line text segment (mtext), etc.

[0057] The exported graphic elements can be divided into line segment graphic elements and text graphic elements. Text graphic elements correspond to text data, thus enabling rapid identification and export of text data. Of course, in other optional embodiments of this application, other existing methods can be used to acquire text data, which will not be elaborated further.

[0058] For the text data in the text description area, the NLP algorithm model obtained by the prior improvement and training of this application is used to analyze and process the text data, realize the semantic association analysis of each text in the text data, classify each text to determine whether the text belongs to a question or an answer; after classifying each text, semantic matching is performed on each text to mine the answers corresponding to each question, realize the association mapping of question and answer, and output the first extracted information containing several question and answer pairs.

[0059] It should be noted that the question-answer pair, that is, one question corresponds to one answer, is essentially the process of extracting question-answer pairs in the drawing review process to determine whether the design specifications or requirements are met, thereby achieving the purpose of the drawing review.

[0060] For example, "fire truck access road" corresponds to a "question," and "clear width and clear height greater than 4 meters" corresponds to an "answer," forming a question-and-answer pair. Similarly, "project location" corresponds to a "question," and "south of Wulian Road and west of Jining Road in Rizhao City" corresponds to an "answer," also forming a question-and-answer pair. Much information in drawings is presented in this way. Therefore, by automating the extraction of question-and-answer pairs from drawings, reviewers no longer need to search and filter through large amounts of drawing information and confirm each item individually, thus improving the efficiency of drawing review.

[0061] In the field of Natural Language Processing (NLP), semantic relation extraction is a crucial semantic processing task. However, current NLP techniques still rely on keywords, parsers, and Named Entity Recognizers (NERs) to obtain advanced features. In contrast, in architectural drawings, important information can appear anywhere in a sentence, in any layout, and in varying sentence lengths. Current NLP-based extraction algorithms perform poorly in terms of efficiency and accuracy when extracting relations from long texts. Therefore, this application employs a specific model architecture for NLP algorithms to parse text data highlighting descriptive text regions, achieving efficient and accurate extraction and output of question-answer pairs.

[0062] refer to Figure 2 The NLP algorithm model used in this application embodiment contains m (m is greater than or equal to 2) parsing modules, each of which is connected in series (that is, the output of the previous parsing module is used as the input of the next parsing module). The input of the first parsing module is the text data of the text description area of ​​the drawing to be processed. The text data is input into the NLP algorithm model in rows to achieve parsing processing.

[0063] A parsing module consists of an LSTM (Long Short-Term Memory) network, a Multi-Head Attention network, a Layernorm normalization network, and a Conv1*1 convolutional kernel connected in series.

[0064] The LayerNorm normalization network normalizes all features of each sample. By normalizing the activation values ​​of the layers, the training process of the model can be accelerated, enabling it to converge faster.

[0065] For Multi-Head Attention networks and Layernorm normalization networks, a positive parsing result cannot be guaranteed. To address this, this application's embodiment uses an ADD feature fusion network to seamlessly transmit the output of the LSTM (Long Short-Term Memory) network to the Conv1*1 convolutional kernel. This allows one part of the output to be transformed by the Multi-Head Attention network and Layernorm normalization network, while the other part is directly fed into the next Conv1*1 convolutional kernel. The results from these two parts are then summed as the input to the next layer, ensuring that at least the information from the LSTM layer is preserved, effectively improving model performance.

[0066] Experiments show that when m is in the range of 5 to 15, the NLP algorithm model can better extract the abstract features of text and high semantic information.

[0067] S103. Input the second data to be processed corresponding to the table area into the multimodal algorithm model, and output the second extracted information containing several question answer pairs; the second data to be processed includes the data in the table area corresponding to the set second data type.

[0068] The second data type is defined as including text data type and table layout image data type. Text data can be obtained through the above-mentioned primitive export method, while table layout image data can be obtained by cropping table areas and converting them into image, PDF, or other formats.

[0069] The tables in drawings are diverse in form and contain a wealth of valuable information for review. In order to improve the efficiency of extracting text information from tables of various types and layouts, this application uses a multimodal algorithm model to output review information for different types of tables.

[0070] refer to Figure 3The multimodal algorithm model includes an encoder and a decoder. The encoder consists of a Tranformer block module, a Mobile-ViT (Mobile Vision Transformer) network, a PAN semantic segmentation network, and a Concat feature fusion network. The encoder includes two branches. One branch consists of n (n ≥ 2) Tranformer block modules connected in series. The other branch consists of the Mobile-ViT network and the PAN semantic segmentation network. The outputs of the two branches are fused and encoded by the Concat feature fusion network.

[0071] To improve model speed while ensuring global text preview, the number of Transformer blocks, n, ranges from 4 to 8. Experimental data shows that when n is less than 4, long text feature vectors cannot preview global text; when n is greater than 8, the model size is too large, resulting in slow execution.

[0072] Transformer blocks are not sensitive to table position information, but cell position information is important when matching questions and answers in a table. To solve this problem, Mobile-ViT was introduced for position encoding.

[0073] The decoder sequentially processes the output of the Concat feature fusion network through K (K ≥ 2) Bi-LSTM bidirectional long short-term memory networks and P (P ≥ 2) CRF layers (the CRF layers introduce constraints to ensure the validity of the final prediction result, which can be automatically learned by the CRF layers during training) to obtain the second extracted information. The second extracted information contains several question-answer pairs.

[0074] The number of Bi-LSTMs, K, ranges from 2 to 6. Experimental data shows that when K equals 1, the decoded information is incomplete; when K is greater than 6, there is redundancy in the decoding. The number of Bi-LSTMs, K, corresponds to the number of CRF layers, P, and both remain the same.

[0075] This application extracts semantic features of text through a multi-layer Tranformer block, and uses a backbone network composed of Mobile-ViT and PAN to detect general information of table text, extracting rich visual features of the table at multiple scales; it performs feature fusion encoding through a Concat layer; and then decodes the information through a Bi-LSTM and CRF layer, finally obtaining a table parsing result containing several question-answer pairs, realizing an end-to-end structured task (i.e., table text recognition, text classification, and semantic matching of question-answer pairs).

[0076] In an optional embodiment of this application, due to the special case where architectural engineers directly represent text using polylines during the drafting process, the text information will be converted into line segment information during the export of graphic elements, resulting in the loss of this text information and making effective classification and QA matching impossible. Therefore, OCR can be directly used to extract the text information from the drawings.

[0077] In an optional embodiment of this application, if there are abnormal texts in the output result that are not classified or do not achieve QA matching, for all the text contained in the table area, OCR is used to re-detect and recognize the text in the table area (using OCR may require converting the drawing to be processed into an image format), and then the multimodal algorithm model is used again for parsing and processing.

[0078] It should be understood that steps S102 and S103 can be processed serially or simultaneously in parallel.

[0079] S104. Based on the first and second extracted information, the final extracted text information of the drawing to be processed is obtained and displayed.

[0080] In an optional embodiment of this application, the QA pairs output by the multimodal algorithm model can also be examined to determine whether there may be incorrect matches in the multimodal output results. This improves the accuracy of text information extraction.

[0081] Specifically, the cell coordinates corresponding to the question and answer in the QA pair are obtained (the cell coordinates are parsed using a multimodal algorithm). It is then determined whether the coordinate positions satisfy a preset relationship. For example, the difference between the row coordinates and column coordinates of the corresponding cells is calculated. If the sum of the differences in row and column coordinates equals 1, the coordinate positions satisfy the preset relationship, and the output QA pair is considered a correct match. If the sum of the differences in row and column coordinates is not equal to 1 (i.e., the sum of the differences in row and column coordinates is greater than or equal to 2, since the two cannot be in the same cell, the sum of the differences in row and column coordinates cannot be 0), it indicates that the cells containing the question and answer are not adjacent cells. Therefore, the QA pair is considered to have a risk of mismatch, and is marked as an abnormal QA pair for manual review and confirmation, thus improving the accuracy of extracting table text information.

[0082] The first and second extracted information are displayed in separate columns (a text description area and a table area). At the same time, QA pairs marked as abnormal in the second extracted information are highlighted (e.g., by differentiating the font or color) or prompted (e.g., by providing a separate reminder for abnormal QA pairs) to facilitate targeted review by the drawing reviewers and improve the efficiency of drawing review.

[0083] Extracting text, table, and image areas from drawings is crucial, as these areas convey vital information. Table and text areas, in particular, contain significant information. Current technology uses regular expressions to extract text relationships; however, due to the diverse layout styles of text and table areas in drawings, and the need for data organization for regular expression standards, the time and manpower costs are high. While text and table areas are essentially textual descriptions, when outputting structured table information, the positional information of the text is equally important. Current technology requires writing multiple matching rules for each type of table, resulting in low efficiency. (See the reference for drawing text areas.) Figure 4 As shown, some text descriptions are quite long, with keywords and corresponding text relationships wrapped on separate lines. Current NLP technology struggles to extract these effectively, resulting in a high error rate in relationship matching. (Reference) Figure 5 As shown, the table area has a complex and varied format. As can be seen from the figure, there will be a lot of line interference. It may not be a big problem for recognizing table rows, but it is easy to parse errors for recognizing columns.

[0084] This application addresses the issue of drawings to be processed by dividing them into regions based on differences in the way information is expressed. This results in the division of text description regions and table regions. Given the potential differences in semantic association features between the text in each region, a multimodal algorithm is used to extract key information from different types of tables in the table regions. Each detected text is classified into categories such as questions and answers, and then the corresponding answer is found for each question related to building codes. For the text descriptions in the text regions, NLP techniques are used to extract relationships, thus improving the efficiency and accuracy of text relationship extraction.

[0085] Based on the same design concept, this embodiment also discloses a drawing text information extraction device.

[0086] refer to Figure 6 A drawing text information extraction device, comprising:

[0087] The acquisition module 61 is used to acquire the area division results of the drawing to be processed; the area includes the text description area and the table area.

[0088] NLP algorithm model 62 is used to predict and process the first data to be processed corresponding to the text description area, and output the first extracted information containing several question answer pairs; the first data to be processed includes the data in the text description area that corresponds to the set first data type.

[0089] The multimodal algorithm model 63 is used to predict and process the second data to be processed corresponding to the table area, and outputs the second extracted information containing several question answer pairs; the second data to be processed includes the data in the table area corresponding to the set second data type.

[0090] The comprehensive processing module 64 is used to obtain the final extraction result of the text information of the drawing to be processed based on the first extraction information and the second extraction information, and output and display it.

[0091] The various variations and specific examples of the methods provided in the above embodiments are also applicable to the drawing text information extraction device of this embodiment. Through the foregoing detailed description of the drawing text information extraction method, those skilled in the art can clearly understand the implementation method of the drawing text information extraction device of this embodiment. For the sake of brevity, it will not be described in detail here.

[0092] To better execute the above method, embodiments of this application also provide a drawing text information extraction device, such as... Figure 7 As shown, the drawing text information extraction device includes a processor 71 and a memory 72.

[0093] Drawing text information extraction devices can be implemented in various forms, including mobile phones, tablets, PDAs, laptops, and desktop computers.

[0094] The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for at least one function (such as selecting marker layers for known wells, determining candidate regions for target wells, and delineating marker layers for target wells based on formation boundary values), and instructions for implementing the intelligent marker layer location and delineation method provided in the above embodiments. The data storage area may store data involved in the intelligent marker layer location and delineation method provided in the above embodiments.

[0095] A processor may include one or more processing cores. The processor executes instructions, programs, code sets, or instruction sets stored in memory, and calls data stored in memory to perform various functions and process data as described in this application. The processor may be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), controller, microcontroller, and microprocessor. It is understood that, for different devices, the electronic devices used to implement the above-described processor functions may also be other types, and the embodiments of this application do not specifically limit this.

[0096] This application provides a computer-readable storage medium, including, for example, various media capable of storing program code such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk. This computer-readable storage medium stores a computer program that can be loaded by a processor and execute the intelligent marker stratigraphic positioning and delineation method of the above embodiments.

[0097] The above description of the embodiments is only used to provide a detailed introduction to the technical solutions of this application. However, the description of the above embodiments is only for the purpose of helping to understand the methods and core ideas of this application, and should not be construed as a limitation of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. A method of extracting information from a drawing, characterized by, The drawing text information extraction method comprises: Obtaining a region division result of a to-be-processed drawing; the region comprises a text description region and a table region; Inputting first to-be-processed data corresponding to the text description region into an NLP algorithm model to output first extraction information comprising a plurality of question-answer pairs; the first to-be-processed data comprises data corresponding to a set first data type in the text description region; Inputting second to-be-processed data corresponding to the table region into a multi-modal algorithm model to output second extraction information comprising a plurality of question-answer pairs; the second to-be-processed data comprises data corresponding to a set second data type in the table region; Obtaining a final extraction result of text information of the to-be-processed drawing based on the first extraction information and the second extraction information, and outputting and displaying the final extraction result; The multi-modal algorithm model comprises an encoder composed of a Tranformer block module, a Mobile-ViT network, a PAN semantic segmentation network and a Concat feature fusion network, and a decoder composed of a Concat feature fusion network, a Bi-LSTM bidirectional long short-term memory network and a CRF layer; The encoder comprises two branches, one of which is composed of n Tranformer block modules connected in series; the other branch is composed of a Mobile-ViT network and a PAN semantic segmentation network; the outputs of the two branches are feature-fused and encoded through the Concat feature fusion network; The decoder sequentially processes the output of the Concat feature fusion network through K Bi-LSTM bidirectional long short-term memory networks and P CRF layers to obtain the second extraction information; K is a natural number greater than or equal to 2, P is a natural number greater than or equal to 2; K is equal to P.

2. The method of claim 1, wherein The model architecture of the NLP algorithm model comprises m LSTM long short-term memory networks, m Multi-Head Attention multi-head attention networks, m Layernorm layer normalization networks, m Conv1*1 convolution kernels and m ADD feature fusion networks; m is a natural number greater than or equal to 2.

3. The method of claim 2, wherein The model architecture of the NLP algorithm model is composed of m analysis modules connected in series, one of which comprises an LSTM long short-term memory network, a Multi-Head Attention multi-head attention network, a Layernorm layer normalization network and a Conv1*1 convolution kernel connected in series; and the output of the LSTM long short-term memory network is also transmitted to the Conv1*1 convolution kernel through an ADD feature fusion network; m∈[5, 15].

4. The method according to any one of claims 1 to 3, wherein The set first data type comprises a text data type, and the set second data type comprises a text data type and a table layout image data type.

5. The method according to any one of claims 1 to 3, wherein The final extraction result of text information of the to-be-processed drawing based on the first extraction information and the second extraction information comprises: For each question-answer pair in the second extraction information, determine the first cell coordinates corresponding to the question of the question-answer pair and the second cell coordinates corresponding to the answer; Calculate the row coordinate difference and column coordinate difference between the first cell coordinates and the second cell coordinates; Determine whether the sum of the row coordinate difference and the column coordinate difference is less than a set threshold value; If yes, determine that the question-answer pair is normal; if no, determine that the question-answer pair is abnormal, and mark the question-answer pair as abnormal; Divide the question-answer pairs in the first extraction information and the second extraction information into normal question-answer pairs and abnormal question-answer pairs to obtain the final extraction result of the text information of the to-be-processed drawing and output the final extraction result.

6. A drawing text information extraction apparatus characterized by comprising: Comprise: An acquisition module is configured to acquire a region division result of a to-be-processed drawing; the region comprises a text description region and a table region; An NLP algorithm model is configured to perform prediction processing on first to-be-processed data corresponding to the text description region, and output first extraction information containing a plurality of question-answer pairs; the first to-be-processed data comprises data corresponding to a set first data type in the text description region; A multi-modal algorithm model is configured to perform prediction processing on second to-be-processed data corresponding to the table region, and output second extraction information containing a plurality of question-answer pairs; the second to-be-processed data comprises data corresponding to a set second data type in the table region; The multi-modal algorithm model comprises an encoder composed of a Tranformer block module, a Mobile-ViT network, a PAN semantic segmentation network, and a Concat feature fusion network, and a decoder composed of a Concat feature fusion network, a Bi-LSTM bidirectional long short-term memory network, and a CRF layer; The encoder comprises two branches, one of which is composed of n Tranformer block modules connected in series; the other branch is composed of a Mobile-ViT network and a PAN semantic segmentation network; the outputs of the two branches are feature-fused and encoded through the Concat feature fusion network; The decoder performs prediction processing on the output of the Concat feature fusion network through K Bi-LSTM bidirectional long short-term memory networks and P CRF layers in sequence to obtain the second extraction information; K is a natural number greater than or equal to 2, P is a natural number greater than or equal to 2; K is equal to P; An integrated processing module is configured to obtain the final extraction result of the text information of the to-be-processed drawing based on the first extraction information and the second extraction information, and output the final extraction result.

7. A document information extraction apparatus characterized by comprising: A processor, a memory, and a computer program stored in the memory and executable on the processor are included, and the processor implements the drawing text information extraction method according to any one of claims 1 to 4 when executing the computer program.

Images