Method for realizing efficient semantic understanding of PDF (Portable Document Format) text by using deep learning

Abstract

The invention provides a method for realizing efficient semantic understanding of a PDF internal text by using deep learning, and relates to the technical field of semantic understanding, and the method comprises the steps: analyzing PDF multi-modal data of a building industry, learning modal structured information, and constructing a segmentation extraction module which is used for analyzing PDF modal distribution and carrying out structured information extraction; establishing a corresponding recognition relationship among the multi-modal structured information, and learning cross-modal semantic association features; and establishing a collaborative search relationship according to the cross-modal semantic association features, when any feature in the collaborative search relationship is triggered, carrying out PDF text search according to the collaborative search relationship, and carrying out integrated feedback on search results. Through the method and the device, the technical problem that the multi-modal information in the PDF document in the building industry is difficult to accurately analyze in the prior art can be solved, the technical target of accurately analyzing the multi-modal information in the PDF document is achieved, and the technical effect of improving the information search and extraction efficiency and accuracy is achieved.

Description

Technical Field

[0001] The present application relates to the technical field of semantic understanding, and in particular to a method for realizing efficient semantic understanding of PDF text using deep learning. Background Art

[0002] PDF documents in the construction industry usually contain a large amount of text, drawings, tables, annotations and other types of information, which makes the processing of PDF documents in the field of architecture and engineering design complex and challenging.

[0003] At present, traditional tools often encounter several major problems when parsing these documents. First, the information is distributed in an intertwined manner, and text, graphics, tables and other contents are mixed together, making it difficult to extract information. Secondly, for scanned or non-text PDF files, the accuracy of OCR recognition is low (in scanned PDF or non-text PDF, character recognition is difficult, which affects content understanding), and character recognition is difficult, which directly affects content understanding. Furthermore, traditional semantic understanding methods mostly rely on simple keyword matching or rules, and cannot identify deeper semantic relationships, resulting in inaccurate information search and extraction. Finally, it is difficult to extract structured information, especially to extract useful structured information from complex documents such as tables and drawing instructions, which requires stronger semantic parsing capabilities. The construction industry has a large number of exclusive terms and expressions, which further aggravates the difficulty of information extraction. Traditional tools cannot achieve efficient and accurate results when dealing with these fields.

[0004] To sum up, there are technical problems in the existing technology that are difficult to accurately parse multimodal information in PDF documents in the construction industry due to complex information distribution, low character recognition accuracy, shallow semantic understanding and difficulty in extracting structured information, which further affects the efficiency and accuracy of information search and extraction. Summary of the invention

[0005] The purpose of this application is to provide an efficient semantic understanding method for PDF text using deep learning, in order to solve the technical problem in the prior art that multimodal information in PDF documents in the construction industry is difficult to accurately parse due to complex information distribution, low character recognition accuracy, shallow semantic understanding, and difficulty in extracting structured information, further affecting the efficiency and accuracy of information search and extraction.

[0006] In view of the above problems, the present application provides an efficient semantic understanding method for PDF content using deep learning, including: parsing PDF multimodal data in the construction industry, learning modal structured information, and constructing a segmentation and extraction module, wherein the segmentation and extraction module is used to parse the PDF modal distribution and extract structured information; establishing a corresponding recognition relationship between multimodal structured information, and learning cross-modal semantic association features; establishing a collaborative search relationship based on the cross-modal semantic association features, and when any feature in the collaborative search relationship is triggered, performing a PDF content search based on the collaborative search relationship to obtain multimodal search results, and integrating the search results for feedback.

[0007] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0008] By parsing the multimodal data of PDF in the construction industry, learning the modal structured information, and constructing a segmentation and extraction module, the segmentation and extraction module is used to parse the PDF modal distribution and extract structured information; establish a corresponding recognition relationship between multimodal structured information, and learn cross-modal semantic association features; establish a collaborative search relationship based on the cross-modal semantic association features, and when any feature in the collaborative search relationship is triggered, perform a PDF text search based on the collaborative search relationship to obtain multimodal search results, and integrate the search results for feedback, so as to achieve the technical goal of accurately parsing multimodal information in PDF documents in the construction industry and achieve the technical effect of improving the efficiency and accuracy of information search and extraction.

[0009] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented according to the contents of the specification, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are specifically cited below. It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easy to understand through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0011] Figure 1 This is a flowchart of a method for implementing efficient semantic understanding of PDF content using deep learning in this application;

[0012] Figure 2 This is a flowchart of building a segmentation and extraction module in the method for efficient semantic understanding of PDF text using deep learning in this application. DETAILED DESCRIPTION

[0013] This application provides an efficient semantic understanding method for PDF content using deep learning, which solves the technical problem in the prior art that multimodal information in PDF documents of the construction industry is difficult to accurately parse due to complex information distribution, low character recognition accuracy, shallow semantic understanding, and difficulty in extracting structured information, further affecting the efficiency and accuracy of information search and extraction. The technical goal of accurately parsing multimodal information in PDF documents of the construction industry is achieved, and the technical effect of improving the efficiency and accuracy of information search and extraction is achieved.

[0014] Below, the technical solutions in the present application will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments of the present application. It should be understood that the present application is not limited to the example embodiments described herein. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application. It should also be noted that, for the convenience of description, only the parts related to the present application are shown in the accompanying drawings, rather than all of them.

[0015] Please see attached Figure 1 , this application provides an efficient semantic understanding method for PDF content using deep learning, which specifically includes the following steps:

[0016] Step 1: parse the PDF multimodal data of the construction industry, learn the modal structured information, and build a segmentation and extraction module, which is used to parse the PDF modal distribution and extract structured information.

[0017] Specifically, in the process of parsing PDFs in the construction industry, multimodal data refers to different types of information contained in PDF files, such as text, images, tables, etc. By learning the modal structured information in multimodal data, the relationship and organization of modal structured information are obtained, such as the relationship between columns and rows in a table, or how lines in an image form the floor plan of a building. Then, a segmentation and extraction module is constructed to effectively segment and extract information of different modes. Specifically, the segmentation and extraction module can extract the architectural design drawings from the PDF separately, or process the numerical values ​​and text in the table separately, and then parse the modal distribution of the PDF file, and can extract the structured information therein, so as to better extract useful structured information from complex architectural PDF files. Among them, the segmentation and extraction module adopts deep learning frameworks such as TensorFlow or PyTorch, and uses supervised learning to train with large-scale PDF data sets in the construction industry. The optimization parameters include learning rate, number of iterations, regularization coefficient, etc., to ensure that the model can accurately converge to the best extraction effect, and finally obtain a high-precision segmentation and extraction model. Model framework.

[0018] Step 2: Establish corresponding recognition relationships between multimodal structured information and learn cross-modal semantic association features.

[0019] Specifically, data from different modalities (such as text, images, tables, etc.) are matched and associated. For example, in a building PDF, the geometric dimensions on the drawing correspond to the dimension description in the text description, and the corresponding relationship is identified and established. Next, by learning cross-modal semantic association features, the semantic connection between different modal data is obtained, such as a line in the building drawing represents a wall, and the wall description in the text is related to the line. Through the learning process, information from different modalities is integrated. For example, "reinforced concrete" in the text is matched with the corresponding structure in the drawing to ensure that the semantic information of each modality corresponds to each other, thereby improving the accuracy of data understanding and processing. Among them, the process uses a cross-modal learning model for training. The training data includes multimodal data such as text, drawings and tables in the construction industry. The attention mechanism model (such as Transformer) is used to learn the semantic association between each modality. The training parameters include the depth and level of multimodal feature extraction. Through the incremental learning method, the model gradually converges to achieve the optimal state of cross-modal semantic association.

[0020] Step 3: Establish a collaborative search relationship based on the cross-modal semantic association features. When any feature in the collaborative search relationship is triggered, perform a PDF text search based on the collaborative search relationship to obtain multimodal search results, and integrate the search results for feedback.

[0021] Specifically, the semantic connection between different modal data is obtained based on the cross-modal semantic association features, and a collaborative search relationship is established based on the connection. When the features related to any modality in the document are triggered, the data of other modalities will also be automatically associated and searched. Then, the PDF content is searched according to the triggered features, and multi-modal search results are collected, including text descriptions, drawing elements, and table data. Finally, the search results from different modalities are integrated and fed back to the user to ensure that the user can obtain all relevant information at one time. For example, when a user searches for a building and enters "basement structural material requirements", the returned results include description text related to the basement structural materials (such as "the basement adopts reinforced concrete structure, the concrete grade is C30, and the steel bar model is HRB400"), tables of relevant materials extracted from the bill of materials table (such as steel bar specifications, usage, concrete ratio and strength, etc.), and drawings showing the structural annotation information of the basement (such as building size, material location, etc.). The collaborative search module is trained through a feature feedback mechanism based on multimodal data. The model uses pre-trained natural language processing models such as BERT as a basis to fine-tune specific terms and semantic relationships in the construction field. The training data includes a large number of industry terms and contextual relationships. The model framework can be optimized in combination with ElasticSearch to ensure search efficiency and accuracy.

[0022] The method of using deep learning to achieve efficient semantic understanding of PDF text can achieve the technical goal of accurately parsing multimodal information in PDF documents in the construction industry, and achieve the technical effect of improving the efficiency and accuracy of information search and extraction.

[0023] Further, if Figure 2 As shown, this application also includes:

[0024] Collect multi-type PDF sample sets from the construction industry, including multimodal data, perform uniformity identification on the sample sets, enhance scarce samples, and construct a training sample set; annotate the training sample set with a polygonal mask, transform the coordinates of the annotated area to generate a target area framework and mask data for supervised learning of the image segmentation model, segment the annotated sample set, and construct a model training set and a model test set; use a pre-trained model as a basis, and use the model training set and model test set to perform incremental training and testing on the pre-trained model to achieve a convergence target and obtain a segmentation extraction model; and construct the segmentation extraction module based on the segmentation extraction model.

[0025] Specifically, we collect multi-type PDF sample sets from the construction industry, including different types of data, such as text, images, tables, etc., as multimodal data. When processing the sample set, we identify the uniformity of the sample set to ensure that the differences between samples can be accurately reflected. For the scarce parts of the sample set, we increase the number of scarce samples through data enhancement technology, so as to ensure that the constructed training sample set is more complete and diversified, avoid the model biased towards categories with large data volumes, and improve the overall recognition effect.

[0026] Next, polygonal masks are used to annotate the constructed training sample set. Polygonal masks are a method used to mark specific areas in an image, which can more accurately identify objects of complex shapes. Through the annotation method of polygonal masks, the coordinates of the annotated area are transformed to generate the target area framework and corresponding mask data for supervised learning of the image segmentation model, further helping to identify and segment different areas in the image. Then, the annotated sample set is segmented, and finally a model training set and a model test set are generated to prepare for subsequent training and testing of the model. The model is trained using a deep learning framework such as TensorFlow or PyTorch. The parameters set during the training process include learning rate, number of iterations, regularization coefficient, etc., and supervised learning is used to optimize the model weights through back propagation to achieve the convergence goal.

[0027] Using the pre-trained model as the basis, the pre-trained model is incrementally trained and tested using the constructed model training set and model test set. A pre-trained model refers to a model that has been preliminarily trained. Through further training, the pre-trained model can better adapt to the tasks in the construction industry. Through incremental training, the pre-trained model gradually converges to the target effect, and finally obtains a segmentation and extraction model that can perform accurate segmentation and extraction. During the incremental training process, the Adam optimizer or the SGD optimizer is used for model optimization. The tuning parameters include batch size, step size, etc. to ensure the convergence of the pre-trained model on the construction industry data.

[0028] Finally, based on the segmentation and extraction model, a segmentation and extraction module for practical applications is constructed. The segmentation and extraction module is responsible for segmenting and extracting images and data from PDF files in the construction industry in practical applications, and realizing automatic data processing and classification.

[0029] By collecting, labeling, training and building models of multimodal PDF samples, we have achieved automated image segmentation and extraction of PDF data for the construction industry, ensuring accuracy and wide application. This enables more efficient processing and analysis of multiple types of data in the construction industry, thereby improving work efficiency.

[0030] Furthermore, the present application also includes:

[0031] The input layer uses the PDF page image and the annotated area as the model input; the convolution layer uses the convolutional neural network to extract features and extract image features of different areas; the region proposal network generates candidate areas in the image to locate different modal areas, and the region proposal network generates a series of candidate boxes, which include different modal contents; the segmentation head uses the segmentation head to segment each candidate box and generate a mask for each modal area; the classification head uses the segmentation head to segment each area and classify each modal type; wherein the input layer, convolution layer, region proposal network, segmentation head and classification head work together in a forward transmission process, when the PDF page image is input, the convolution layer extracts features, the region proposal network generates candidate areas, the segmentation head segments the candidate areas, and the classification head determines the area category, and finally outputs the image segmentation mask and the regional modal classification result.

[0032] Specifically, the role of the input layer is to use the image of the PDF page and the related annotated area as the input of the model, thereby providing a data basis for subsequent image processing. Next, the convolutional layer uses a convolutional neural network for feature extraction. Through multiple layers of convolution operations, it is responsible for extracting useful feature information from the input PDF page image. It can not only capture simple edges and textures, but also extract more complex abstract information through deeper convolutional layers. For example, the convolutional layer can extract features such as the morphology of text, the grid structure of tables, and the geometric features of drawings for subsequent processing. Deep learning uses hierarchical feature representation, gradually from low-level features (such as edges and lines) to high-level features (such as patterns of charts and tables), to understand complex image information in an automated and data-driven manner. Traditional manual feature extraction may be ineffective when facing complex PDF documents in the construction industry, while convolutional neural networks can automatically learn these features through a large amount of training data.

[0033] Then, a region proposal network is used to generate candidate regions in these images to determine the parts containing different modal information. Multiple candidate boxes are generated through the region proposal network to help the model locate areas that may contain different modal contents such as drawings, text or tables. The region proposal network generates candidate region boxes that may contain target objects based on feature maps. Unlike the sliding window method in traditional image processing, the region proposal network can find potential regions of interest in the image, and the regions of interest are automatically generated through learning, with high efficiency and accuracy. The region proposal network automatically generates candidate regions that match the actual target by learning the patterns of target objects in the image (such as text blocks and table structures), which can greatly reduce unnecessary searches and calculations and ensure the capture of important modal areas.

[0034] Next, each candidate box is segmented by the segmentation head to generate a mask for each modal area. The mask is a covering layer that helps identify the boundaries of each modal area, thereby accurately segmenting different parts of the image. The segmentation model refers to a segmentation head such as the one in Mask R-CNN, which can generate a detailed pixel mask based on each candidate area, and each pixel is accurately classified into different modal categories (such as text, tables, drawings). This is achieved through the contextual information and object edge information extracted by the convolutional layer of deep learning. Traditional segmentation methods often have difficulty handling complex image backgrounds, while deep learning models can automatically generate high-precision pixel-level segmentation results by learning object boundary information in large-scale data. In PDF documents in the construction industry, drawings, texts, and tables are often mixed together, which can effectively separate these contents and give accurate segmentation results.

[0035] Next, after segmentation using the classification head, each modal region is classified to determine the type of modality to which the region belongs. The classification head identifies the region as a specific type such as image, text, or table. When classifying the segmented regions, the deep learning classification network can semantically classify each region based on the contextual information learned from the convolutional features. The classifier can automatically learn the features of each modality from the training data, such as text, charts, tables, annotations, etc. Through end-to-end training of deep learning, the classification head can adaptively understand the features of different modalities, thereby correctly classifying each segmented region, thereby improving the classification accuracy and avoiding the limitations of manual rule definition.

[0036] Finally, the input layer, convolutional layer, region proposal network, segmentation head and classification head work together in the entire forward pass process. The input layer, convolutional layer, region proposal network, segmentation head and classification head are built together to form an end-to-end deep learning model, which works together in a forward pass process. When the image is input, the convolutional layer extracts features, the region proposal network generates candidate regions, the segmentation head performs fine segmentation, and the classification head determines the region category. Finally, the entire model outputs an accurately segmented image mask and region classification results. The training process of the model is adjusted according to the difference between the real annotated region and the model output through backpropagation. The loss function of the entire network considers both segmentation and classification tasks to optimize the performance of each module. After inputting the PDF page image, the convolutional layer extracts features, the region proposal network generates candidate regions, the segmentation head performs region segmentation, the classification head determines the type of each region, and finally outputs the image segmentation mask and the modal classification result of the region. Through the end-to-end learning process, the image of the PDF page is input and the fine segmentation mask and modal category are output. During the training process, the loss function is calculated through the back propagation algorithm, and the difference between the predicted result and the actual annotation is propagated back to each layer of the model, and the weight of the network is gradually adjusted to continuously optimize and improve the model during training. End-to-end training allows the model to automatically learn all the required features and rules without manually designing features or rules, so that the deep learning model can directly learn how to extract useful information from complex PDF documents.

[0037] By inputting data from PDF images, extracting features, locating candidate areas, segmenting image areas and classifying them, the author finally outputs accurate image segmentation results and modal classification results, thereby finely processing architectural PDF files.

[0038] Furthermore, the present application also includes:

[0039] A modal analysis model is constructed according to the modal type of each area, including a text recognition channel, a table analysis channel, and a drawing analysis channel. The text recognition channel has a built-in natural language processor to convert the image in the text area into readable text, perform semantic understanding on the extracted text, and convert the unstructured text into structured information; the table analysis channel has a built-in table analysis model to extract the structured information of the table information and integrate and arrange it according to the table layout; the drawing analysis channel has a built-in object monitoring model to identify the architectural elements and related geometric dimensions and annotation data in the drawings, and extract the structured geometric dimensions and annotation data; channel allocation is performed based on the regional modal classification results, and structured information of each modal type is extracted through each allocation channel, and the structured information of each modal area is output.

[0040] Specifically, different parsing models are constructed according to the modal type of each area, including text recognition channel, table parsing channel and drawing parsing channel. That is, after segmenting the text area, OCR technology (such as Tesseract combined with CNN) is used to convert the image in the text area into readable text. Next, the extracted text is semantically understood using NLP models (such as BERT, GPT) to extract specific information (such as design instructions, bill of materials, project requirements, etc.). Text recognition (OCR) and natural language processing (NLP) convert unstructured text in PDF into structured information, such as extracting material specifications, floor information, building numbers, etc. in the design instructions, and organizing and storing them logically. For example, fields such as name, specification, quantity, etc. in the bill of materials are extracted and stored in a predefined format, or the drawing annotation text is analyzed to extract descriptions related to a specific structure (such as a wall or window).

[0041] Then, the table parsing model built into the table parsing channel can recognize and extract the information of the table in the PDF. By segmenting the table area, the grid structure in the table is parsed using a table parsing model (such as TableNet or DeepTabStR), and the data of each cell is converted into a readable format. For tables with wrong rows and columns, the model automatically corrects them by learning the table pattern to ensure that each field corresponds to the correct data position, and then automatically converts information such as budget tables and material tables into database format. For example, each line of a bill of materials is parsed into structured data containing fields such as material name, quantity, and price to facilitate subsequent calculations and analysis. Deep learning can effectively deal with noise and misalignment problems in complex typesetting or scanned tables by learning table patterns.

[0042] Next, the drawing parsing channel uses object detection technology (such as YOLO, Faster R-CNN) to analyze the segmented drawing area and identify the architectural elements in the drawing (such as doors, windows, walls, etc.) and the associated dimensioning and annotations. The drawing information is converted into structured data by associating the detected objects with the annotations of the drawing. The complex drawing structure is converted into structured geometry and annotation data. For example, structured data such as building height, length, area, etc. are automatically extracted and associated with the corresponding building objects (such as floors, structures). The dimensional information in the drawing can be combined with the text information in the design description document to ensure complete structured information extraction.

[0043] Next, based on the regional modality classification results, each region will be assigned to the corresponding parsing channel. For example, the text region is sent to the text recognition channel, the table region is sent to the table parsing channel, and the drawing region is sent to the drawing parsing channel. By assigning channels, regional information of various modalities can be extracted as structured data.

[0044] By constructing different modal analysis channels, the text, table and drawing information in the architectural PDF file are analyzed and structured respectively. Each channel performs a specific analysis task according to the input data type, and finally outputs the corresponding structured information, thereby converting the architectural file into easy-to-use digital data for further processing and application.

[0045] Furthermore, the present application also includes:

[0046] The language information of the text type is converted into a semantic embedding vector; a corresponding and inclusive relationship library of architectural terms is constructed, and the semantic embedding vector is split and expanded using the corresponding and inclusive relationship library of architectural terms to generate text embedding, table embedding, and drawing embedding, so that data of different modalities can be compared and matched in the same semantic space; the text embedding, table embedding, and drawing embedding are feature aligned, and the corresponding recognition relationship of text, table, and drawing features is established through joint training; the cross-modal semantic association features are learned based on the corresponding recognition relationship of text embedding, table embedding, and drawing embedding.

[0047] Specifically, the language information of the text type is converted into a semantic embedding vector, and the text content is processed by the model into a vector representation that can be used for calculation. For example, the "concrete wall" in the architectural document is converted into a multi-dimensional numerical vector to reflect the semantic characteristics of the word, which is convenient for subsequent processing. Then, based on historical data or experience, the content of different building elements is integrated to record the vocabulary and their relationships in the architectural field, such as "wall" and "concrete" are inclusion relationships. The semantic embedding vector generated previously is split and expanded through the relationship library to generate text embedding, table embedding and drawing embedding. Different types of embedding enable data information from different modalities to be compared and matched in the same semantic space.

[0048] Then, the text embedding, table embedding, and drawing embedding are feature aligned to find common features in the same dimension. The corresponding recognition relationship between text, table, and drawing is established through joint training. For example, the geometric figures in the architectural drawing correspond to the numerical values ​​in the table, and the numerical values ​​are connected to the text description. Through joint training to understand the corresponding relationship, the cross-modal semantic association features are learned based on the embedded corresponding recognition relationship, and then the semantic connection between different modalities is obtained. For example, the "window" in the text corresponds to the specific design symbol in the drawing, and the material description in the table is associated with the structural elements in the drawing.

[0049] By converting text information into semantic embedding vectors and building an architectural vocabulary relationship library, we can generate embeddings of different modalities, compare and match modalities in the same semantic space, and obtain the correspondence and semantic connection of different modal data through feature alignment and joint training. For example, we can associate "door" with the rectangular symbol on the drawing and the size information in the table, thereby improving the matching accuracy between data and ensuring that data of different modalities are more consistent in architectural design and analysis.

[0050] Furthermore, the present application also includes:

[0051] Determine the hyponym and hyponym relationships of words based on the semantic embedding vector, and perform hyponym and hyponym mapping based on the split and expanded words; establish a multi-layer semantic embedding structure based on the contextual relationship, and project the semantic vector into a hierarchical structure according to the mapping relationship; based on the multi-layer semantic embedding structure, generate text embedding, table embedding, and drawing embedding with multi-layer recognition relationships, respectively, for layer-by-layer semantic recognition relationships from high-level semantic locking to low-level details.

[0052] Specifically, based on the semantic embedding vector, the hypernym relationship of the word is determined to understand whether the word is a representative of a broader category or a more specific instance. For example, "building" is a hypernym, while "house" is a hyponym. By analyzing the semantic embedding vector, a hierarchical relationship between words is established. Then, based on the results of word splitting and expansion, the words are mapped to the hypernym relationship. Specific words are mapped to the broader category they belong to, such as mapping "concrete" to the "building materials" category.

[0053] Then, based on the contextual relationship, a multi-layer semantic embedding structure is established. According to the meaning of the context, the semantic vectors are organized into a hierarchical structure. The semantic vectors are projected layer by layer, and the words are distributed in different semantic levels according to the upper and lower relationships. For example, in a building document, the higher-level semantics is "building", while the lower-level semantics are "window" or "wall".

[0054] Based on the multi-layer semantic embedding structure, text embedding, table embedding, and drawing embedding with multi-level identification relationships are generated respectively, which are used to lock the semantics from higher levels to lower levels of details. For example, the embedding of architectural drawings first identifies the "building", and then identifies specific elements such as "walls" and "windows" layer by layer. Text embedding and table embedding also gradually identify detailed information through multi-level semantic structures.

[0055] By determining the hyponym and hyponym relationships of words, establishing a multi-layer semantic embedding structure, and generating embeddings with multi-level recognition relationships, it is possible to gradually recognize high-level semantic concepts down to specific details, and ultimately parse and classify information more accurately in architectural documents or drawings.

[0056] Furthermore, the present application also includes:

[0057] An information specification library is established according to the building industry standards, which includes core data for each structure type; a semantic embedding vector is extracted according to the triggered features; the information specification library is searched using the semantic embedding vector as an index to obtain a standard building information composition; the search results are verified using the standard building information composition to determine the missing information composition.

[0058] Specifically, an information specification library is established to ensure data standardization. It is built according to the standards of the construction industry. The information specification library contains the core data required for each type of structure. For example, for wall structures, the information specification library records data such as material type, size, and strength. Then, when a trigger feature is detected, the semantic embedding vector related to the feature is extracted. For example, if the trigger feature is "reinforced concrete wall", the semantic vector related to "reinforced concrete wall" is extracted.

[0059] Then, using the extracted semantic embedding vector as an index, search in the information specification library to find the standard building information components corresponding to the features, and find all the necessary data about "reinforced concrete wall" in the construction industry standard, such as wall thickness, steel bar specifications, etc. Next, use standard building information to verify the composition of the search results to check whether the data is complete and meets the standards. If information is missing, identify and mark the missing parts. For example, if the wall thickness information is missing, mark and prompt to supplement the data.

[0060] By establishing a standard information specification library for the construction industry and combining it with semantic embedding vectors for search and verification, the integrity and standardization of construction information can be ensured. For example, when processing the design documents of a high-rise building, according to the extracted features, such as "reinforced concrete wall", the corresponding standard data is searched from the information specification library, and the information in the document is verified to be complete. If missing, prompts are given, thereby improving the standardization and accuracy of construction data and ensuring compliance with industry standards.

[0061] Furthermore, the present application also includes:

[0062] According to the multimodal search results, consistency verification is performed, including performing context semantic analysis on the multimodal search results to determine the context relationship; selecting the context semantic consistent relationship according to the context relationship to perform data alignment to obtain the missing amount and the deviation amount; determining the semantic relationship compensation difference amount according to the context semantic inconsistent relationship, performing data alignment compensation according to the semantic relationship compensation difference amount to obtain the missing amount and the deviation amount; when the multimodal search results have a multi-layer semantic embedding structure, performing layer-by-layer consistency verification according to the multi-layer semantic embedding structure to ensure that data from high level to low level remain consistent.

[0063] Specifically, we conduct consistency verification based on multimodal search results, and check the data results of different modes to ensure consistency at the semantic level. For example, the description in the text and the architectural elements in the drawings need to match in meaning. Then, we conduct contextual semantic analysis on the search results to determine the contextual relationship. For example, in a building file, "wall" and "door frame" have an adjacent relationship.

[0064] Then, according to the contextual relationship, select the part with consistent contextual semantics for data alignment to find the missing or deviated parts. For example, if the location of "window" is found in the table, but the relevant annotations are missing on the drawing, identify the missing amount or deviation. Then, for the places where the contextual semantics are inconsistent, adjust the inconsistent parts by determining the semantic relationship compensation difference. For example, although they are all basements, the original actual drawings mentioned above are the updated drawings, and the height is increased by 50cm through the adjustment below. The adjustment of 50cm is used as the compensation amount, and then the relationship of the context is verified through alignment compensation, so that the final data results are more accurate and the missing amount and deviation amount are found.

[0065] When there are multiple layers of semantic embedding structures in the multimodal search results, consistency verification is performed layer by layer, from high-level semantic concepts to low-level detailed information, all layers need to be consistent. For example, first verify the high-level semantic consistency of "building", and then verify the details such as "walls", "doors and windows" layer by layer to ensure that each layer matches.

[0066] By performing semantic analysis and consistency verification on multimodal search results, data is checked and aligned layer by layer to identify omissions and deviations, thereby improving data accuracy and completeness.

[0067] In summary, the method for realizing efficient semantic understanding of PDF content using deep learning provided by this application has the following technical effects:

[0068] By parsing the multimodal data of PDF in the construction industry, learning the modal structured information, and constructing a segmentation and extraction module, the segmentation and extraction module is used to parse the PDF modal distribution and extract structured information; establish a corresponding recognition relationship between multimodal structured information, and learn cross-modal semantic association features; establish a collaborative search relationship based on the cross-modal semantic association features, and when any feature in the collaborative search relationship is triggered, perform a PDF text search based on the collaborative search relationship to obtain multimodal search results, and integrate the search results for feedback, so as to achieve the technical goal of accurately parsing multimodal information in PDF documents in the construction industry and achieve the technical effect of improving the efficiency and accuracy of information search and extraction.

[0069] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0070] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of the present application and its equivalent technology, the present application is also intended to include these modifications and variations.

Claims

1. An efficient semantic understanding method for PDF content using deep learning, characterized by: Methods include: Parse the PDF multimodal data of the construction industry, learn the modal structured information, and build a segmentation and extraction module, which is used to parse the PDF modal distribution and extract structured information; Establish corresponding recognition relationships between multimodal structured information and learn cross-modal semantic association features; A collaborative search relationship is established based on the cross-modal semantic association features. When any feature in the collaborative search relationship is triggered, a PDF text search is performed based on the collaborative search relationship to obtain multimodal search results, and the search results are integrated and fed back.

2. The method for realizing efficient semantic understanding of PDF content by using deep learning as claimed in claim 1, characterized in that: The method of parsing the PDF multimodal data of the construction industry, learning the modal structured information, and building a segmentation and extraction module includes: Collect multiple types of PDF sample sets in the construction industry, including multimodal data, identify the uniformity of the sample sets, enhance the scarce samples, and construct a training sample set; The training sample set is annotated using a polygonal mask, the coordinates of the annotated area are transformed to generate a target area framework and mask data for supervised learning of an image segmentation model, and the annotated sample set is segmented to construct a model training set and a model test set; Using the pre-trained model as a basis, performing incremental training and testing on the pre-trained model using the model training set and the model test set, and achieving a convergence target to obtain a segmentation and extraction model; The segmentation extraction module is constructed based on the segmentation extraction model.

3. The method for realizing efficient semantic understanding of PDF content by using deep learning as claimed in claim 2, characterized in that: The pre-trained model includes: Input layer, which takes the PDF page image and the annotated area as model input; Convolutional layer, using convolutional neural network for feature extraction, extracting image features of different regions; A region proposal network generates candidate regions in the image to locate different modal regions. The region proposal network generates a series of candidate boxes, which include different modal contents. Segmentation head: Use the segmentation head to segment each candidate box and generate a mask for each modal area. The classification head, after segmentation by the segmentation head, classifies each region and determines the type of each modality; The input layer, convolution layer, region proposal network, segmentation head and classification head work together in a forward transmission process. When a PDF page image is input, the convolution layer extracts features, the region proposal network generates candidate regions, the segmentation head performs candidate region segmentation, and the classification head determines the region category, and finally outputs the image segmentation mask and regional modal classification results.

4. The method for realizing efficient semantic understanding of PDF content by using deep learning as claimed in claim 3, characterized in that: The pre-trained model also includes: A modal analysis model is constructed according to the modal type of each area, including a text recognition channel, a table analysis channel, and a drawing analysis channel. The text recognition channel has a built-in natural language processor to convert the image in the text area into readable text, perform semantic understanding on the extracted text, and convert the unstructured text into structured information; the table analysis channel has a built-in table analysis model to extract the structured information of the table information and integrate and arrange it according to the table layout; the drawing analysis channel has a built-in object monitoring model to identify the architectural elements and related geometric dimensions and annotation data in the drawings, and extract the structured geometric dimensions and annotation data; Channel allocation is performed based on the regional modal classification result, structural information of each modal type is extracted through each allocated channel, and structural information of each modal region is output.

5. The method for realizing efficient semantic understanding of PDF content by using deep learning as claimed in claim 1, characterized in that: Establish corresponding recognition relationships between multimodal structured information and learn cross-modal semantic association features, including: Convert the language information of text type into semantic embedding vector; Constructing a library of architectural word correspondence and inclusion relations, using the library of architectural word correspondence and inclusion relations to split and expand the semantic embedding vector, generating text embedding, table embedding, and drawing embedding, so that data of different modalities can be compared and matched in the same semantic space; Aligning the text embedding, table embedding, and drawing embedding, and establishing corresponding recognition relationships among text, table, and drawing features through joint training; The cross-modal semantic association features are learned based on the corresponding recognition relationships among the text embedding, table embedding, and drawing embedding.

6. The method for realizing efficient semantic understanding of PDF content by using deep learning as claimed in claim 5, characterized in that: The semantic embedding vector is split and expanded by using the architectural word correspondence and inclusion relationship library to generate text embedding, table embedding, and drawing embedding, including: Determine the hyponymy relationship of words according to the semantic embedding vector, and perform hyponymy mapping based on the split and expanded words; Establish a multi-layer semantic embedding structure according to the contextual relationship, and project the semantic vector into a hierarchical structure according to the mapping relationship; Based on the multi-layer semantic embedding structure, text embedding, table embedding, and drawing embedding with multi-level identification relationships are generated respectively, which are used for layer-by-layer semantic identification relationships from high-level semantic locking to low-level details.

7. The method for realizing efficient semantic understanding of PDF content by using deep learning as claimed in claim 6, characterized in that: After obtaining the multimodal search results and integrating the search results for feedback, the method further includes: Establish an information specification library, structured according to building industry standards, which includes the core data contained in each structure type; Extract semantic embedding vectors based on the triggered features; Searching the information specification library using the semantic embedding vector as an index to obtain standard building information components; The search results are compositionally verified using the standard building information composition to determine the missing information composition.

8. The method for realizing efficient semantic understanding of PDF content by using deep learning as claimed in claim 7, characterized in that: Also includes: Performing consistency verification based on the multimodal search results, including performing contextual semantic analysis on the multimodal search results to determine contextual relationships; According to the contextual relationship, the contextual semantic consistency relationship is selected to align the data and obtain the missing amount and deviation amount; Determine a semantic relationship compensation difference amount according to the context semantic inconsistency relationship, perform data alignment compensation according to the semantic relationship compensation difference amount, and obtain the missing amount and the deviation amount; When the multimodal search results have a multi-layer semantic embedding structure, layer-by-layer consistency verification is performed according to the multi-layer semantic embedding structure to ensure that data from high levels to low levels remain consistent.

Images