Method, system, device and storage medium for extracting structured data from engineering archives

Abstract

The invention relates to a method for extracting structured data of engineering archives, comprising the following steps: constructing an engineering archive rule base, wherein the engineering archive rule base comprises a plurality of rule meta-attributes; pre-training a text extraction model, collecting structured data of professional vocabulary of historical engineering archives as raw data, and performing iterative training by using unsupervised learning to obtain a text extraction model; obtaining input data from an engineering file, and performing pre-processing, inputting the pre-processed data into a pre-trained text extraction model to extract text vocabulary; performing feature association and data cleaning processing on the text vocabulary to obtain text metadata; performing character matching on the text metadata to obtain a plurality of character attributes in the text metadata; performing rule matching on the engineering archive rule base through the character attributes to determine the rule meta-attributes matched by the text metadata, thereby determining the entities associated with the text metadata; and generating structured data according to the entities associated with the text metadata.

Description

Technical Field

[0001] The present invention relates to a method, system, equipment and storage medium for extracting structured data of an engineering archive, and belongs to the technical field of digital processing of engineering archives. Background Art

[0002] With the rise and widespread adoption of artificial intelligence, repetitive tasks have been significantly reduced. Deep learning can now be used to train models to meet daily repetitive tasks, reducing manual input costs and significantly increasing the value of work. Artificial intelligence is widely used in archival work, often requiring the preservation of important historical data. Data storage methods often change dramatically with the changing technology of the time, from paper documents to digital data. Large amounts of data require manual analysis and documentation. Engineering archives are one such example, containing crucial information from the construction process and representing a significant archival resource.

[0003] Engineering archives serve as the basis for project completion, evidence of project quality, and reliable data for scientific research and infrastructure. They offer significant value in reuse, providing valuable insights, and promoting innovation. Engineering archives are typically stored and preserved as signed paper copies or electronic scans. These documents typically undergo an unstructured process, including project handover signatures, project file backup scans, and electronic storage of engineering archives in a database. With ever-increasing amounts of data, efficient organization and retrieval of engineering archives are often required, often requiring significant manual effort and cost. With the increasing maturity of artificial intelligence (AI), the automated cataloging of unstructured data in engineering archives using AI has become a viable solution.

[0004] Existing technologies such as the invention patent with application number "202110289665.9" disclose a method for assisting the generation of official documents for government agencies, which involves the field of natural language generation technology. This method utilizes the computer's large-capacity storage, fast processing, and convenient human-computer interaction capabilities to construct a corpus-based computer-assisted writing system. Through human-computer interaction, it recommends sentence patterns and examples from a real corpus to users in real time, provides assistance for the core activity of sentence making, fills the technical gap in corpus-based computer-assisted writing systems, and solves the problems of inaccurate and inefficient information provided by existing technologies, and inadequate and imperfect writing assistance services by adopting this one-stop intelligent document writing assistance method, thereby fully meeting writing needs. However, this existing technology is only a semi-automatic assisted writing method. By recommending relevant sentence patterns and examples to the author, and the author makes modifications, it still requires a lot of manual work to refine the words and sentences.

[0005] For example, the invention patent with application number "201811548852.9" discloses a method for automatic writing of official documents. According to the target title input by the user, the method matches a title template with the same article genre and core content from the article template library, and parses the target title according to the core content to generate a target title template, and obtains candidate title templates similar to the target title template. The candidate title templates are then evaluated based on information such as keywords of the target title template and the candidate title templates, and the article template corresponding to the candidate title template with the highest evaluation is selected as the target article template corresponding to the target title. Finally, the article is generated based on the target title and the target article template. However, this existing technology requires manual preparation of a large number of template libraries in advance, and the template library is selected only by simple keyword matching, which makes it difficult to avoid the situation where the template library is not applicable. Summary of the Invention

[0006] In order to solve the problems existing in the above-mentioned prior art, the present invention proposes a method for extracting structured data from engineering archives, constructs an engineering archive rule library, and combines text extraction and feature association on the basis of the engineering archive rule library to effectively associate the required text entities from the engineering files, and then performs entity association storage through engineering archive management rules to realize the extraction of unstructured data from engineering archives, thereby greatly improving the efficiency of archive digitization.

[0007] The technical solutions of the present invention are as follows:

[0008] In one aspect, the present invention provides a method for extracting structured data from an engineering archive, comprising the following steps:

[0009] Constructing an engineering archive rule base based on historical engineering archives and engineering archive management methods, wherein the engineering archive rule base includes a number of rule meta-attributes;

[0010] Pre-training the text extraction model: collecting structured data of professional vocabulary from historical engineering archives as raw data, using data mining technology to extract rules from the raw data to form a baseline model. The extracted rules are then used for unsupervised learning. The baseline model is iteratively trained with data outside the rules to obtain a text extraction model.

[0011] Get input data from the project file, preprocess the input data, and input the preprocessed data into the pre-trained text extraction model to extract text vocabulary;

[0012] Perform feature association and data cleaning on the extracted text words to obtain text metadata;

[0013] Perform character matching on text metadata to obtain multiple character attributes in the text metadata; perform rule matching through character attributes to the engineering archive rule library to determine the rule meta-attributes matched by the text metadata, thereby determining the entity associated with the text metadata; generate structured data based on the entity associated with the text metadata.

[0014] As a preferred embodiment, the method for performing feature association on the extracted text words is specifically as follows:

[0015] Get word vectors for text vocabulary;

[0016] Calculate the cross entropy of word vectors;

[0017] The cross entropy of word vectors is input into the bidirectional LSTM neural network for feature extraction;

[0018] A softmax function is set after the output layer of the bidirectional LSTM neural network to perform feature concatenation to obtain the context feature vector of the word vector.

[0019] As a preferred embodiment, the method for performing data cleaning processing on the extracted text vocabulary is specifically as follows:

[0020] Use text regularization to filter out data that does not conform to the rules in the text vocabulary.

[0021] As a preferred embodiment, the method of obtaining input data from the engineering file is specifically as follows:

[0022] Perform OCR recognition on the engineering file and obtain the image data of the engineering file as input data;

[0023] The method for preprocessing the input data is specifically as follows:

[0024] The image data is processed in sequence by Gaussian filtering, mean blurring, hue adjustment, contrast enhancement, image edge processing, and Gaussian noise processing.

[0025] In another aspect, the present invention provides a system for generating structured data of an engineering archive, comprising:

[0026] An engineering archive rule base construction module is used to construct an engineering archive rule base based on historical engineering archives and engineering archive management methods, wherein the engineering archive rule base includes a number of rule meta-attributes;

[0027] The text extraction model training module is used to collect structured data of professional vocabulary in historical engineering archives as raw data, use data mining technology to extract rules from the raw data, form a baseline model, and use the extracted rules for unsupervised learning. The baseline model is iteratively trained with data outside the rules to obtain a text extraction model.

[0028] The input module is used to obtain input data from the project file, preprocess the input data, and input the preprocessed data into the pre-trained text extraction model to extract text vocabulary;

[0029] An association module is used to associate features of the extracted text words;

[0030] The cleaning module is used to perform data cleaning on the extracted text words and obtain text metadata;

[0031] The matching module is used to perform character matching on the text metadata and obtain multiple character attributes in the text metadata; perform rule matching on the engineering archive rule library through the character attributes, determine the rule meta-attribute that matches the text metadata, and thus determine the entity associated with the text metadata;

[0032] The data generation module is used to generate structured data according to the entities associated with the text metadata.

[0033] As a preferred embodiment, the method in which the association module performs feature association on the extracted text words is specifically as follows:

[0034] Get word vectors for text vocabulary;

[0035] Calculate the cross entropy of word vectors;

[0036] The cross entropy of word vectors is input into the bidirectional LSTM neural network for feature extraction;

[0037] A softmax function is set after the output layer of the bidirectional LSTM neural network to perform feature concatenation to obtain the context feature vector of the word vector.

[0038] As a preferred embodiment, the method in which the cleaning module performs data cleaning processing on the extracted text words is specifically as follows:

[0039] Use text regularization to filter out data that does not conform to the rules in the text vocabulary.

[0040] As a preferred embodiment, the method for the input module to obtain input data from the project file is specifically as follows:

[0041] Perform OCR recognition on the engineering file and obtain the image data of the engineering file as input data;

[0042] The method for the input module to preprocess the input data is specifically as follows:

[0043] The image data is processed in sequence by Gaussian filtering, mean blurring, hue adjustment, contrast enhancement, image edge processing, and Gaussian noise processing.

[0044] On the other hand, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for extracting structured data from engineering archives as described in any embodiment of the present invention is implemented.

[0045] On the other hand, a computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method for extracting structured data from engineering archives as described in any embodiment of the present invention.

[0046] The present invention has the following beneficial effects:

[0047] The present invention provides a method for extracting structured data from engineering archives. It constructs an engineering archive rule library. Based on the engineering archive rule library, it combines text extraction and feature association to effectively associate the required text entities from engineering files. Then, it performs entity association storage through engineering archive management rules, realizes the extraction of unstructured data from engineering archives, and greatly improves the efficiency of archive digitization. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0049] Figure 2 This is an example diagram of feature association of text words in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0051] It should be understood that the step numbers used herein are only for convenience of description and are not intended to limit the order in which the steps are to be executed.

[0052] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0053] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0054] The term "and / or" refers to and includes any and all possible combinations of one or more of the associated listed items.

[0055] Example 1:

[0056] See also Figure 1 , a method for extracting structured data from engineering archives, comprising the following steps:

[0057] An engineering archive rule base is constructed based on historical engineering archives and engineering archive management methods. In this embodiment, the engineering archives are specifically the engineering project archives of the State Grid Corporation of China. Therefore, historical power grid engineering archives are collected and the engineering archive rule base is constructed based on the power grid engineering project management method and power grid management method. Structured data is extracted from a large amount of historical data in the power grid engineering archives. Through manual verification and multiple rounds of iteration, a large number of rule meta-attributes are generated.

[0058] Pre-training the text extraction model: collecting structured data of professional vocabulary from historical engineering archives as raw data, using data mining technology to extract rules from the raw data to form a baseline model. The extracted rules are then used for unsupervised learning. The baseline model is iteratively trained with data outside the rules to obtain a text extraction model.

[0059] Get input data from the project file, preprocess the input data, input the preprocessed data into the pre-trained text extraction model to extract text words, and then retrieve the extracted text words in the format of common messages;

[0060] Perform feature association and data cleaning on the extracted text words to obtain text metadata;

[0061] Perform character matching on text metadata and divide the text metadata into multiple character attributes; perform rule matching through character attributes to the engineering archive rule library, and determine the rule meta-attributes that match the text metadata based on the characteristics of the rule meta-attributes, thereby determining the entities associated with the text metadata; generate structured data based on the entities associated with the text metadata.

[0062] See Figure 2 As a preferred implementation of this embodiment, the method for performing feature association on the extracted text words is specifically as follows:

[0063] Get word vectors for text vocabulary;

[0064] Calculate the cross entropy of word vectors;

[0065] The cross entropy of the word vector is input into the input layer of the bidirectional LSTM neural network for feature extraction;

[0066] A softmax function is set after the output layer of the bidirectional LSTM neural network to perform feature concatenation to obtain the context feature vector of the word vector.

[0067] As a preferred implementation of this embodiment, the method for performing data cleaning processing on the extracted text vocabulary is specifically as follows:

[0068] Use text regularization to filter out data that does not conform to the rules in the text vocabulary; using text regularization can ensure that the data is more professional and accurate, and avoid some dirty data (garbled characters, symbols, etc.) on the basis of accurate recognition.

[0069] As a preferred implementation of this embodiment, the method of obtaining input data from the project file is specifically as follows:

[0070] Perform OCR recognition on the engineering file and obtain the image data of the engineering file as input data;

[0071] The method for preprocessing the input data is specifically as follows:

[0072] The image data is processed in sequence by Gaussian filtering, mean blurring, hue adjustment, contrast enhancement, image edge processing, and Gaussian noise processing.

[0073] Example 2:

[0074] This embodiment provides a system for generating structured data of engineering archives, including:

[0075] An engineering archive rule base construction module is used to construct an engineering archive rule base based on historical engineering archives and engineering archive management methods, wherein the engineering archive rule base includes a number of rule meta-attributes;

[0076] The text extraction model training module is used to collect structured data of professional vocabulary in historical engineering archives as raw data, use data mining technology to extract rules from the raw data, form a baseline model, and use the extracted rules for unsupervised learning. The baseline model is iteratively trained with data outside the rules to obtain a text extraction model.

[0077] The input module is used to obtain input data from the project file, preprocess the input data, and input the preprocessed data into the pre-trained text extraction model to extract text vocabulary;

[0078] An association module is used to associate features of the extracted text words;

[0079] The cleaning module is used to perform data cleaning on the extracted text words and obtain text metadata;

[0080] The matching module is used to perform character matching on the text metadata and obtain multiple character attributes in the text metadata; perform rule matching on the engineering archive rule library through the character attributes, determine the rule meta-attribute that matches the text metadata, and thus determine the entity associated with the text metadata;

[0081] The data generation module is used to generate structured data according to the entities associated with the text metadata.

[0082] As a preferred embodiment, the method in which the association module performs feature association on the extracted text words is specifically as follows:

[0083] Get word vectors for text vocabulary;

[0084] Calculate the cross entropy of word vectors;

[0085] The cross entropy of word vectors is input into the bidirectional LSTM neural network for feature extraction;

[0086] A softmax function is set after the output layer of the bidirectional LSTM neural network to perform feature concatenation to obtain the context feature vector of the word vector.

[0087] As a preferred embodiment, the method in which the cleaning module performs data cleaning processing on the extracted text words is specifically as follows:

[0088] Use text regularization to filter out data that does not conform to the rules in the text vocabulary.

[0089] As a preferred embodiment, the method for the input module to obtain input data from the project file is specifically as follows:

[0090] Perform OCR recognition on the engineering file and obtain the image data of the engineering file as input data;

[0091] The method for the input module to preprocess the input data is specifically as follows:

[0092] The image data is processed in sequence by Gaussian filtering, mean blurring, hue adjustment, contrast enhancement, image edge processing, and Gaussian noise processing.

[0093] Example 3:

[0094] The present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for extracting structured data from an engineering archive as described in any embodiment of the present invention is implemented.

[0095] Example 4:

[0096] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method for extracting structured data from engineering archives as described in any embodiment of the present invention.

[0097] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can represent: a, b, c, a and b, a and c, b and c or a and b and c, where a, b, c can be single or multiple.

[0098] Those skilled in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented using a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0099] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0100] In the several embodiments provided in this application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of this application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory; hereinafter referred to as: ROM), random access memory (Random Access Memory; hereinafter referred to as: RAM), magnetic disk or optical disk, etc. Various media that can store program code. The above description is only an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for extracting structured data from engineering archives, characterized in that: The following steps are involved: Constructing an engineering archive rule base according to historical engineering archives and an engineering archive management method, wherein the engineering archive rule base includes a number of rule meta-attributes; Pre-training text extraction model, collecting structured data of professional vocabulary of historical engineering archives as raw data, using data mining technology to extract rules from the raw data, forming a benchmark model, and using the extracted rules for unsupervised learning, marking the data outside the rules to iteratively train the benchmark model to obtain the text extraction model; Obtain input data from the project file, preprocess the input data, and input the preprocessed data into the pre-trained text extraction model to extract text vocabulary; Perform feature association and data cleaning on the extracted text words to obtain text metadata; Perform character matching on text metadata to obtain multiple character attributes in the text metadata; perform rule matching on the engineering archive rule library through the character attributes to determine the rule meta-attributes matched by the text metadata, thereby determining the entity associated with the text metadata; generate structured data according to the entity associated with the text metadata; The method of performing feature association on the extracted text words is specifically as follows: Get word vectors for text vocabulary; Calculate the cross entropy of word vectors; The cross entropy of word vectors is input into the bidirectional LSTM neural network for feature extraction; A softmax function is set after the output layer of the bidirectional LSTM neural network to perform feature concatenation and obtain the context feature vector of the word vector.

2. A method for extracting structured data from engineering archives according to claim 1, characterized in that: The method for performing data cleaning processing on the extracted text vocabulary is specifically as follows: Use text regularization to filter out data that does not conform to the rules in the text vocabulary.

3. The method for extracting structured data from engineering archives according to claim 1, characterized in that: The method for obtaining input data from a project file is specifically as follows: Perform OCR recognition on the engineering file and obtain the image data of the engineering file as input data; The method for preprocessing the input data is specifically as follows: The image data is processed in sequence by Gaussian filtering, mean blurring, tone adjustment, contrast enhancement, image edge processing, and Gaussian noise processing.

4. A system for extracting structured data from engineering archives, characterized in that: include: An engineering archive rule base construction module is used to construct an engineering archive rule base according to historical engineering archives and an engineering archive management method, wherein the engineering archive rule base includes a number of rule meta-attributes; The text extraction model training module is used to collect structured data of professional vocabulary of historical engineering archives as raw data, extract rules from the raw data using data mining technology, form a benchmark model, and use the extracted rules for unsupervised learning, annotate the data outside the rules to iteratively train the benchmark model, and obtain the text extraction model; An input module is used to obtain input data from a project file, preprocess the input data, and input the preprocessed data into a pre-trained text extraction model to extract text vocabulary; An association module, used to associate features of the extracted text words; A cleaning module is used to perform data cleaning on the extracted text words and obtain text metadata; A matching module is used to perform character matching on text metadata and obtain multiple character attributes in the text metadata; By matching character attributes to the engineering archive rule base, the rule meta-attributes matching the text metadata are determined, thereby determining the entity associated with the text metadata; A data generation module, used for generating structured data according to entities associated with the text metadata; The method in which the association module performs feature association on the extracted text words is specifically as follows: Get word vectors for text vocabulary; Calculate the cross entropy of word vectors; The cross entropy of word vectors is input into the bidirectional LSTM neural network for feature extraction; A softmax function is set after the output layer of the bidirectional LSTM neural network to perform feature concatenation and obtain the context feature vector of the word vector.

5. A system for extracting structured data from engineering archives according to claim 4, characterized in that: The method in which the cleaning module performs data cleaning processing on the extracted text words is specifically as follows: Use text regularization to filter out data that does not conform to the rules in the text vocabulary.

6. A system for extracting structured data from engineering archives according to claim 4, characterized in that: The method for the input module to obtain input data from the project file is specifically as follows: Perform OCR recognition on the engineering file and obtain the image data of the engineering file as input data; The method for the input module to preprocess the input data is specifically as follows: The image data is processed in sequence by Gaussian filtering, mean blurring, tone adjustment, contrast enhancement, image edge processing, and Gaussian noise processing.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for extracting structured data from engineering archives as described in any one of claims 1 to 3 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for extracting structured data from engineering archives as described in any one of claims 1 to 3 is implemented.

Images