Industrial document-oriented multi-modal information extraction method and system

Abstract

The application provides a kind of industrial document-oriented multimodal information extraction method and system;The method comprises the following steps: obtaining semi-structured first text information and unstructured second text information from industrial documents;Text features are extracted based on the first text information;Image features are extracted based on the second text information;Multimodal fusion features are obtained according to the text features and image features;Overall view extraction is performed on industrial documents based on multimodal fusion features;By fusing text features and image features, the overall view of the industrial document can be extracted, and the accuracy of the view extraction is improved;No additional manual operation is required during the entire process, reducing labor costs and improving production efficiency;In addition, the industrial document-oriented multimodal information extraction method provided by the application has certain transferability and can be applied to industrial documents in different vertical fields to solve the problem of industrial document extraction.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information processing, in particular to the processing technology of industrial documents, and specifically to a multi-modal information extraction method and system for industrial documents. BACKGROUND

[0002] Industrial documents are files and materials formed and archived by industries in production design, processing and manufacturing activities, and contain a large amount of knowledge related to production and operation.

[0003] At present, enterprises mainly use manual review to find useful information for production and operation, but the industrial document data is large in amount and complex in structure, and the manual review method is low in efficiency. The existing document extraction technology lacks semantic understanding of text and pictures, and it is difficult to process pictures containing key information, or it does not consider the interaction between text and pictures. At the same time, if the existing technology wants to get good results in a specific industrial document, it needs to perform statistics or model training on the specific industrial document, or needs to manually design preprocessing work such as word segmentation and data cleaning for the specified document library. It is difficult to migrate models between different corpora, and the generalization is poor.

[0004] How to efficiently and accurately extract multi-modal industrial document data is a key problem to be solved in the process of building a multi-modal industrial knowledge base. SUMMARY

[0005] The purpose of the present application is to provide a multi-modal information extraction method and system for industrial documents, which solves the problems mentioned in the background.

[0006] In a first aspect, the present application provides a multi-modal information extraction method for industrial documents, comprising the following steps: obtaining semi-structured first text information and unstructured second text information from industrial documents; extracting text features based on the first text information; extracting image features based on the second text information; obtaining multi-modal fusion features according to the text features and the image features; and performing overall view extraction on the industrial documents based on the multi-modal fusion features.

[0007] In the present application, by fusing text features and image features, the overall view of the industrial document can be effectively extracted, and the accuracy of view extraction is improved. In the informationization process of the industrial document, no additional manual operation is needed, the labor cost is reduced, and the production efficiency is improved. In addition, the multi-modal information extraction method for industrial documents provided by the present application has a certain migration property and can be applied to industrial documents of different vertical fields to solve the problem of industrial document extraction.

[0008] In an implementation form of the first aspect, the obtaining the semi-structured first text information and the unstructured second text information from the industrial document comprises: analyzing the industrial document to divide the industrial document into three parts of structured information, semi-structured information and unstructured information; screening the semi-structured information and the unstructured information respectively to obtain the first text information and the second text information.

[0009] In an implementation form of the first aspect, the obtaining the semi-structured first text information and the unstructured second text information from the industrial document further comprises: extracting the structured information based on an industrial knowledge graph and an industrial concept library.

[0010] In an implementation form of the first aspect, the extracting the text feature based on the first text information comprises: performing entity relationship extraction on the first text information by using an entity enhancement algorithm based on an industrial concept library to obtain the text feature.

[0011] In an implementation form of the first aspect, the extracting the image feature based on the second text information comprises: extracting fine-grained semantic information in the second text information based on an image encoder and a pre-trained language model; and outputting the image feature related to the fine-grained semantic information.

[0012] In an implementation form of the first aspect, the obtaining the multi-modal fusion feature according to the text feature and the image feature comprises: mapping the text feature and the image feature to the same semantic space to obtain an image semantic vector group and a text semantic vector group; calculating semantic similarity between the image semantic vector group and the text semantic vector group; and performing feature fusion between the text feature and the image feature according to the semantic similarity to obtain the multi-modal fusion feature.

[0013] In an implementation form of the first aspect, the performing overall view extraction on the industrial document based on the multi-modal fusion feature comprises: obtaining a view segment according to the multi-modal fusion feature to realize overall view extraction on the industrial document based on the view segment.

[0014] In a second aspect, the present application provides a multi-modal information extraction system for industrial documents, comprising: a first obtaining module configured to obtain semi-structured first text information and unstructured second text information from an industrial document; a first extracting module configured to extract a text feature based on the first text information; a second extracting module configured to extract an image feature based on the second text information; a second obtaining module configured to obtain a multi-modal fusion feature according to the text feature and the image feature; and a view extraction module configured to perform overall view extraction on the industrial document based on the multi-modal fusion feature.

[0015] In a third aspect, the present application provides a storage medium having stored thereon a computer program which, when executed by a processor, implements the industrial document-oriented multi-modal information extraction method described above.

[0016] In a fourth aspect, the present application provides an electronic device comprising: a processor and a memory; the memory is configured to store a computer program; the processor is configured to execute the computer program stored in the memory, so that the electronic device executes the industrial document-oriented multi-modal information extraction method described above.

[0017] As described above, the industrial document-oriented multi-modal information extraction method, system, medium and electronic device of the present application have the following beneficial effects:

[0018] Compared with the prior art, the present application develops a set of industrial document-oriented multi-modal information extraction method from the aspects of efficient processing of document types, deep mining of potential rules, and high condensation of key information, thereby improving the accuracy of viewpoint extraction. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 FIG. 1 shows a structural schematic diagram of the industrial document-oriented multi-modal information extraction system of the present application in an embodiment.

[0020] Figure 2 FIG. 2 shows a working principle diagram of the industrial document analysis subsystem of the present application in an embodiment.

[0021] Figure 3 FIG. 3 shows a working principle diagram of the semi-structured data extraction subsystem of the present application in an embodiment.

[0022] Figure 4 FIG. 4 shows a working principle diagram of the entity enhancement algorithm based on the industrial concept library of the present application in an embodiment.

[0023] Figure 5 FIG. 5 shows a working principle diagram of the unstructured data extraction subsystem of the present application in an embodiment.

[0024] Figure 6 FIG. 6 shows a working principle diagram of the multi-modal feature fusion subsystem of the present application in an embodiment.

[0025] Figure 7 FIG. 7 shows a working principle diagram of the document viewpoint extraction subsystem of the present application in an embodiment.

[0026] Figure 8 FIG. 8 shows a flowchart of the industrial document-oriented multi-modal information extraction method of the present application in an embodiment.

[0027] Figure 9A flow chart showing an embodiment of the present application for obtaining semi-structured first text information and unstructured second text information from an industrial document.

[0028] Figure 10 A flow chart showing an embodiment of the present application for extracting image features based on the second text information.

[0029] Figure 11 A flow chart showing an embodiment of the present application for obtaining multi-modal fusion features according to text features and image features.

[0030] Figure 12 A structural schematic diagram showing another embodiment of the present application for a multi-modal information refining system for industrial documents. DETAILED DESCRIPTION

[0031] The present application can be implemented or applied in other different specific embodiments, and the details in the specification can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0032] It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner, and only show the components related to the present application, not the number, shape and size of the components when actually implemented. The actual implementation of each component may be a random change in type, number and proportion, and the component layout pattern may be more complex.

[0033] Referring to Figures 1 to 12 The following embodiments of the present application provide a multi-modal information refining method, system, medium and electronic device for industrial documents. Compared with the prior art, the present application develops a multi-modal information refining method for industrial documents from the aspects of efficient processing of document types, deep mining of potential rules, and high condensation of key information, thereby improving the accuracy of viewpoint extraction.

[0034] The multi-modal information refining method for industrial documents of the present application will be explained and described below through specific embodiments.

[0035] In the prior art, the core challenge of the overall viewpoint extraction task for industrial documents is how to utilize the interaction information between multi-modal data to improve the accuracy of extraction.

[0036] Interaction between modal data refers to the meaningful connection between different modalities, which is an important way for multi-modal data to complement each other. The industrial document view extraction of multi-modal fusion needs to find an effective way to fuse information of each modality, capture semantic information between different modalities, and maximize the interaction information between each modality and each modality.

[0037] To solve the above problems, in an embodiment, the present application provides an industrial document-oriented multi-modal information extraction system using the above industrial document-oriented multi-modal information extraction method. Specifically, first, the input document is preliminarily analyzed and classified according to the structure, content, etc. of the industrial document; second, for the semi-structured information in the document, the I-BERT model is used to extract engineering-related entities and relationships, and for the unstructured information in the document (mainly images), the CLIP model and GPT2 model are used to extract keywords from the unstructured content; then, the feature vectors extracted from the structured information in the document and the feature vectors extracted from the unstructured information are mapped to the same semantic space to eliminate the differences in feature dimensions and realize multi-modal feature fusion; finally, based on the document name, multi-modal fusion features, etc. of the industrial document text data, the overall view of the document is extracted. This method does not require manual word segmentation, reduces the amount of manual work, and has good transferability in different industrial documents to solve the problem of industrial document extraction.

[0038] As shown in Figure 1 , the industrial document-oriented multi-modal information extraction system of the present application comprises:

[0039] 1) Industrial document analysis subsystem 11, which will quickly analyze the input industrial document, classify the information content in the document according to structured, semi-structured and unstructured, extract structured information, and screen semi-structured and unstructured information.

[0040] The specific implementation of the preliminary analysis and classification of the industrial document is as shown in Figure 2 , and the specific description is as follows:

[0041] The industrial document analysis subsystem is composed of content classification and content screening modules. The system can be applied in actual industrial production scenarios, including but not limited to: quickly and effectively processing massive documents, improving the processing efficiency and accuracy of document information, shortening the document processing time and reducing the labor cost, etc.

[0042] Step (1.1), content classification module. For the decoded content of the document, analyze the format, font size, identifier, etc. of the entire document, and divide the document into structured, semi-structured and unstructured three parts for further information processing and analysis.

[0043] Step (1.2), content screening module. Based on large-scale data pre-trained neural network model and industrial knowledge graph, concept library, structured, semi-structured and unstructured information is screened.

[0044] For structured information, the system uses a text similarity matching algorithm and uses an engineering concept tree as a matching template to capture explicit features such as file name, title, directory relationship tree, title relationship tree, coding format (including security level) from document structured content, facilitating further optimization and improving document information processing efficiency and accuracy.

[0045] For semi-structured information, the system uses a content relevance analysis algorithm to extract text content and tables and other information within the document that are strongly related to the industry, thereby further improving document information processing and analysis.

[0046] For unstructured information (images), the system uses a ResNet model fine-tuned on industrial image data to screen images. Specifically, the ResNet-50 image classification model is selected as the base model, which is fine-tuned based on industrial domain data. Based on the set of industrial image categories, the structure and parameters of the original model are updated to make the model more suitable for the current data distribution. During the fine-tuning process, dynamic pruning is used for fine-tuning training, and 5-fold cross-validation is used to test the model to ensure the generalization performance of the model. Finally, the fine-tuned ResNet model is used to screen images, and only strongly related real scene images, flowcharts and design drawings are retained, thereby improving the accuracy and efficiency of document information processing.

[0047] 2) Semi-structured data extraction subsystem 12, for semi-structured text information within the document, the subsystem uses BERT as the base model, and optimizes the entity relationship extraction model based on the industrial concept library to extract the entity and relationship triples and text features in the text content that are strongly related to engineering.

[0048] For semi-structured text information within the document, the specific implementation of extracting engineering-related entities and relationships based on the BERT model is as shown in Figure 3 , which is described as follows:

[0049] Step (2.1), definition of industrial concept library. The industrial concept library is an information structure that systematically organizes and expresses relevant knowledge and data in the industrial field. It consists of three basic elements: entity categories, entity attributes, and entity relationships. Entity categories are categories of objects with the same characteristics or functions in the industrial field, such as products, equipment, materials, processes, etc. Entity attributes are information used to describe the characteristics or status of entity categories, such as name, model, specifications, parameters, etc. Entity relationships refer to the connection or dependency between different entity categories, such as belonging to, composition, production, consumption, etc. The construction of the industrial concept library requires reference to expert experience, industrial flow charts, product concept trees and other materials. Through multiple iterations and adjustments, the information structure is optimized and improved so that it can effectively support decision-making and innovation in various links such as industrial design, engineering, and management.

[0050] Step (2.2), Entity Enhancement Algorithm Based on Industrial Concept Library. Secondly, this system uses the entity enhancement algorithm based on the industrial concept library to extract entity relationships, and adopts the entity relationship extraction mode of extracting entities first and then extracting relationships. The input of the entity extraction model is:

[0051] X=(x1,x2,...,x n );

[0052] Among them, x i Represents the i-th sentence in the semi-structured text.

[0053] In the process of entity extraction, the industrial-specific BERT base model (I-BERT) is used to encode the characters in the document into tokens. Given a semi-structured text X, its encoding embedding representation F t The formula is as follows:

[0054] F t =bert_industry(X);

[0055] In order to solve the problem of nested entities, a span-based entity extraction mode is adopted to extract all possible span arrangements S, and the entity type of each span is judged by SoftMax. i ∈S, span represents h e (s i ) is defined as:

[0056]

[0057] Among them, x start(i) is the span s i Embedding of the start character, x end(i) Indicates span s i End character embedding, is the embedding representation of the span width feature.

[0058] In order to improve the accuracy of entity extraction, the system proposes an entity enhancement algorithm based on the concept library, as shown in the following formula: Figure 4 As shown in the formula, according to the industrial concept library, common sense library, entity length distribution statistics, industrial field expert experience, etc., the adaptive entity span is realized. Through the text similarity matching algorithm, the similarity between the current span and the related concept in the industrial concept library is calculated, and the entity extraction is further improved, and the accuracy of entity extraction is improved.

[0059] In relation classification, the category information of the entity is introduced into the input layer, and the entity boundary and type are added to the front and back of the entity Span as identifiers, and then they are used as the input of the relation model, and the format is introduced as follows:

[0060] <s:pn>and< / s:pn> : represents the head entity of the entity type Product, <s>is the first token of the head entity span,< / s> is the last token.

[0061] <o:pn>and< / o:pn> : represents the tail entity of the entity type Product, <o>is the first token of the tail entity span,< / o> is the last token.

[0062] The input of the relation model is represented as follows:

[0063] rel_input=...<S:e i >,x start(i) ,...,x end(i) ,< / S:e i >,

[0064] ...<O:e i >,x start(i) ,...,x end(i) ,< / O:e i >,....

[0065] In relation extraction, I-BERT is used to encode rel_input with entity type into token, and the encoding of the first token in each entity pair is spliced, and then SoftMax classification is performed. In order to learn the different context representations of entities and relations, the entity model and the relation model adopt two independent pre-training models for encoding (without sharing parameters).

[0066] Step (2.3), loss function of entity and relation. The loss function of extracting engineering-related entities and relations based on the industrial specialized BERT base model is as follows:

[0067]

[0068]

[0069] L=αL e +βL r ;

[0070] in, Indicates the training set s i The real entity type, Represents the span pair s in the training data i , s j The true relationship type, L is the loss of the entire model, L e represents the loss of entity classification, L r represents the loss of relationship classification, α and β are the weight coefficients of entity classification loss and relationship classification loss, respectively, which are obtained through prior experience knowledge in professional fields and subsequent intelligent learning.

[0071] 3) Unstructured data extraction subsystem 13: For real-life images in documents, the subsystem extracts fine-grained semantic information from images based on the CLIP model and the GPT2 model, and outputs relevant keywords and image features.

[0072] For the unstructured information in the document, the subsystem extracts semantic information of different granularities based on the CLIP model and the GPT2 model to extract keywords. Figure 5 As shown in the figure, for flowcharts and design drawings in the document, this system uses them as attributes in the document information structure tree. For real-life images, the unstructured data extraction subsystem extracts keywords based on the CLIP model and the GPT2 model. The steps are as follows:

[0073] Step (3.1), image encoder. CLIP visual encoder is used, which is an advanced image encoder based on Transformer structure. It can perform self-supervised learning on large-scale image-text datasets to extract language-related visual image representations from images. Visual image features refer to attributes that can reflect information about image content, style, emotion, etc., such as color, shape, texture, and edge. The specific formula is as follows:

[0074]

[0075] Where g i Represents image i.

[0076] Step (3.2), feature mapping network. Use a multi-layer perceptron as a bridge between the CLIP and GPT2 models. Input the image visual features obtained by the CLIP model into the MLP mapping network (multi-layer fully connected layer) and map them into a visual prefix prompt vector p. The mapping network formula is as follows:

[0077]

[0078] where p represents the embedding vector of the image vector after passing through the MLP mapping network, and the dimension of each vector is the same as the text embedding.

[0079] Step (3.3), text decoder. The visual prefix prompt vector p generated in step (3.2) is taken as the prefix of the generated image description, and the GPT2 model is used to generate the keywords described by the image.

[0080] In the training stage, in order to make the model lightweight, only the intermediate feature mapping network is trained, and the original parameters of the CLIP and GPT2 models are kept. In the prediction stage, the visual prefix of the input image is extracted using the CLIP encoder and the mapping network, the keywords conditioned on the visual prefix are generated based on the visual prefix using the text decoder, and the next token is predicted one by one according to the language model output.

[0081] 4) Multimodal feature fusion subsystem 14, which maps the text features extracted from semi-structured information and the image features extracted from unstructured information to the same semantic space, generates a feature mask according to the semantic similarity, and fuses the features of the two modalities to provide support for the viewpoint extraction subsystem.

[0082] For the feature vectors extracted from the structured information in the document and the feature vectors extracted from the unstructured information, the multimodal data fusion is realized. Considering that directly mapping the image representation to the text representation space will lose a lot of texture information and geometric information, the application discards the multimodal fusion method of using image representation to enhance text representation. According to the assumption that the semantic information of different forms of representation extracted from the same target object is the same, the application uses a semantic space as a mapping plane, and simultaneously maps the text representation and the image representation to the same semantic space for fusion of different modal features, as shown in the specific implementation manner Figure 6 as shown in the specific implementation manner

[0083] Step (4.1), multimodal representation alignment mapping. The self-organizing mapping algorithm of unsupervised learning is used in the application to cluster the image representation extracted by the CLIP model in step 3) to obtain an image semantic vector group Specifically, the self-organizing network initializes the weight vector of the mapping node with a Gaussian distribution, and for a certain image feature First, the similarity between the real value vector and the weight vector of the mapping node is calculated according to the Euclidean distance function, and the node with the smallest distance is classified as the best matching unit, which will decrease with the number of iterations and the distance. The update formula of the weight W v (s) is:

[0084]

[0085] where s represents the step size, u represents the best matching unit, a(s) represents the learning coefficient, and Θ(u, v, s) represents the proximity function of neurons u and v at step size s. After training, similar elements in the image representation are considered to be the same semantic information, and the image features of the real-valued vector are converted into an image semantic vector group of similar semantic information Similarly, the text features extracted by the Bert model in step 2) are projected into the semantic space of the image semantic vector group to form a text semantic vector group

[0086] Step (4.2), multi-modal representation alignment and fusion. In order to fuse the features of similar semantics in different modalities, the present application first calculates the cosine distance in the feature vector groups of the two modalities to find the modal features with similar similarity. Specifically, for any feature vector k i in the text semantic vector group i , the cosine distance between each feature vector in the image semantic vector group is calculated, and the feature vector with the smallest cosine distance [k j , r n ] is considered to be approximately semantic and is fused into m i = [k j , r n ], and the final multi-modal fusion feature group M = {m1, m2, m3…m i} is obtained, and the formula is as follows:

[0087]

[0088] where Sim(*) is a similarity measure formula, and for any k j and r t , when the similarity of the two is the highest among all vector combinations, the feature fusion is performed.

[0089] 5) Document opinion extraction subsystem 15, based on multiple factors such as the name of the industrial document data, the multi-modal features in the document, etc., to realize the overall opinion extraction of the document.

[0090] Based on multiple factors such as multi-modal fusion features and industrial document text data, the overall opinion of the document is extracted, and the specific implementation is as shown in Figure 7 , and the specific description is as follows:

[0091] ​​Step (5.1), opinion fragment screening module. The present application realizes this module by adopting the method of classification task, and constructs a classifier for all (sentence, document) pairs. Specifically, given the multi-modal fusion feature M or the text feature F of a certain chapter z of a document t , which is input into the Sigmoid classifier for calculation to determine whether it contains the opinion of the document.

[0092]

[0093] wherein m is the multi-modal fusion feature, F t is the text feature, and k is the probability of the sentence containing the opinion of the document.

[0094] The present application sets τ as the opinion tolerance, and regards the sentence with k≤τ as the content that may represent the opinion of the document, and concatenates all the continuous opinion sentences to form an opinion fragment.

[0095] Step (5.2), document-oriented opinion extraction module. The present application realizes this module by adopting the span sorting method, directly lists all the spans in the document name n given the opinion fragment u, and selects the best span related to the opinion fragment as the target argument of the opinion fragment u. Specifically, take the span a in the document name / large chapter title / small chapter title n, and take the opinion fragment u and the span a as the input of the model, encode them in the opinion target extraction BERT (bert_concept), and get the span a as the score of the opinion fragment u through the Sigmoid classifier, and take the span with the highest score as the target argument of the opinion fragment u.

[0096] Through step 5) and the above description, the overall opinion extraction of the document is realized. Given an industrial document, the input document is quickly analyzed by the industrial document parsing subsystem, and the contents in the document are classified according to the structured, semi-structured and unstructured. For the structured information in the document, the explicit features such as the file name, the title, the directory relationship tree, the title relationship tree, the coding format (including the security level) in the document are captured. For the semi-structured content in the document, the strongly related text paragraphs and tables and other information are retained, and the strongly related entities and relationships in the semi-structured content are extracted by the semi-structured data extraction subsystem. For the unstructured content in the document, the strongly related real scene pictures, flowcharts and design drawings and other information are retained, and the corresponding image descriptions are generated by the unstructured data extraction subsystem. Based on the feature vectors extracted from the structured information of the document and the feature vectors extracted from the unstructured information, the multi-modal data fusion is realized. Based on multiple factors such as the multi-modal fusion feature and the text data of the industrial document, the overall opinion extraction of the document is realized, and the document structure tree is finally obtained by combining the extracted document opinion with the output of other subsystems, which provides support for subsequent downstream applications (document retrieval, graph construction).

[0097] In the above embodiments, the present application aims to provide a multi-modal information extraction system for industrial documents. The system uses a semi-structured data extraction subsystem to extract industrial-related entities and relationships, and uses an unstructured extraction subsystem to extract key information from images. By fusing text features and image features, the system can effectively extract the overall view of the document. This method does not require additional manual operation in the document informatization process, reducing labor costs and improving production efficiency. In addition, the system has a certain migration ability and can be applied to industrial documents in different vertical fields to solve the problem of industrial document extraction.

[0098] The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings of the embodiments of the present application.

[0099] As shown in the figure, in an embodiment, the multi-modal information extraction method for industrial documents of the present application comprises the following steps: Figure 8

[0100] Step S1, obtaining semi-structured first text information and unstructured second text information from industrial documents.

[0101] Specifically, according to the structure, content, etc. of the industrial document, the input document is quickly analyzed, the structured information is extracted, and the semi-structured and unstructured information is screened.

[0102] In an embodiment, for the decoded information of the document, first, the entire document is divided into structured, semi-structured and unstructured three parts of content according to the format, attribute and identifier, etc.

[0103] Secondly, based on the industrial knowledge graph and the industrial concept library, the structured information is extracted, based on the correlation analysis, the semi-structured information is screened, and based on the pre-trained convolutional neural network model, the unstructured information is screened. For structured information, a similarity matching algorithm is used, and an engineering concept tree is used as a matching template to capture explicit features such as file name, title, directory relationship tree, title relationship tree, coding format (including security level) from the structured content of the document. For semi-structured information, a content correlation analysis algorithm is used to extract text content and table information that are strongly related to the industry within the document. For unstructured information (images), the ResNet model fine-tuned by industrial image data is used to classify the images, and the strongly related real scene images, flowcharts and design diagrams, etc. are retained.

[0104] As shown in the figure, in an embodiment, the method for obtaining semi-structured first text information and unstructured second text information from industrial documents comprises: Figure 9

[0105] ​​Step S11, analyzing the industrial document to divide the industrial document into three parts of structured information, semi-structured information and unstructured information.

[0106] Step S12, screening the semi-structured information and the unstructured information respectively to obtain the first text information and the second text information.

[0107] In an embodiment, the obtaining of the semi-structured first text information and the unstructured second text information from the industrial document further comprises:

[0108] Step S13, extracting the structured information based on an industrial knowledge graph and an industrial concept library.

[0109] Step S2, extracting text features based on the first text information.

[0110] Specifically, for the semi-structured text information in the document, the subsystem uses BERT as the base model, optimizes the base model according to the industrial concept library, extracts the triplets and text features in the text that are strongly related to engineering.

[0111] In an embodiment, first, the entity categories, entity attributes and relationships between entities are defined based on expert experience, industrial flowcharts and product concept trees. Multiple iterations and adjustments are made during the definition, and optimization and improvement are made according to actual conditions.

[0112] Secondly, the entity relationship extraction algorithm based on the concept library is used to extract the entity relationship. In the process of entity extraction, the industrial specialized BERT base model (I-BERT) is introduced to encode the text information into tokens. In the training, the industrial text data is used to fine-tune the conventional pre-training BERT model, so that the base model is more suitable for the data distribution of industrial text when predicting. In order to solve the problem of nested entities, the entity relationship extraction mode based on span is adopted to extract all possible fragments and arrange them. Through SoftMax, the entity type of each Span is judged. In the definition of span, the adaptive entity span is realized according to the industrial concept library, common sense library, entity length distribution statistics and industrial domain expert experience.

[0113] In the entity extraction stage, the similarity between the current span and the related concept in the industrial concept library is calculated by a text similarity matching algorithm to further improve entity extraction and improve the accuracy of entity extraction. In the relationship classification, the category information of the entity is introduced into the input layer, that is, the entity boundary and type are added as identifiers before and after the entity Span, and then used as the input of the relationship model. I-BERT is used to encode the input into tokens. The encoding of the first token in each entity pair is spliced, and SoftMax is used for classification. In order to learn the different context representations of entities and relationships, the entity model and the relationship model use two independent pre-training models for encoding (without sharing parameters).

[0114] In an embodiment, the extracting text features based on the first text information comprises: performing entity relationship extraction on the first text information by using an entity enhancement algorithm based on an industrial concept library to obtain the text features.

[0115] Step S3, extracting image features based on the second text information.

[0116] Specifically, for unstructured information in documents such as real scene graphs and flowcharts, the subsystem extracts fine-grained semantic information in unstructured information based on the CLIP model and the GPT2 model, and outputs related keywords and image features.

[0117] In an embodiment, first, the CLIP model is used to extract deep representation information of the input image and further encode it to obtain an image representation vector; second, a mapping network is used to map the image representation vector to a text feature space, and the image representation vector is converted into a visual prefix prompt vector by aligning the distribution of the image representation vector and the text representation vector; finally, the GPT2 model is used to generate keywords describing the image according to the visual prefix prompt vector.

[0118] As shown in Figure 10 In an embodiment, the extracting image features based on the second text information comprises:

[0119] Step S31, extracting fine-grained semantic information in the second text information based on an image encoder and a pre-trained language model.

[0120] Step S32, outputting the image features related to the fine-grained semantic information.

[0121] Step S4, obtaining multi-modal fusion features according to the text features and the image features.

[0122] Specifically, the feature vectors extracted from the structured information and the feature vectors extracted from the unstructured information in the document are mapped to the same semantic space to eliminate the differences in feature dimensions and realize multimodal feature fusion.

[0123] In one embodiment, based on the assumption that the semantic information of different forms of representation extracted from the same target object is the same, the present invention uses the semantic space as the mapping plane, and simultaneously maps the text representation and the image representation to the same semantic space to fuse the features of different modalities. The self-organizing map algorithm of unsupervised learning is used to extract the image representation in the unstructured data extraction subsystem. Clustering is performed through competitive learning to obtain image semantic vector groups Similarly, the text features extracted by the I-BERT model in the semi-structured data extraction subsystem Using the same algorithm, it is projected into the semantic space of and to form a text semantic vector group

[0124] In order to fuse features of different modalities with similar semantics, the present invention first calculates the cosine distance between the feature vector groups of the two modalities and searches for modal features with similar similarity. Specifically, for the text semantic vector group Any eigenvector k in i , calculate the cosine distance between each feature vector in the image semantic vector group, and take the feature vector with the smallest cosine distance [k i ,r j ] is considered as approximate semantics and fused into m n =[k i ,r j ], and obtain the final multimodal fusion feature group M.

[0125] It should be noted that, for text features and image features, the present invention maps the two into the same semantic space based on semantic relevance, so as to eliminate the feature differences brought by heterogeneous multi-source data, fuse different modal features, and provide support for the document extraction subsystem.

[0126] like Figure 11 As shown, in one embodiment, obtaining multimodal fusion features based on the text features and the image features includes:

[0127] Step S41: Map the text features and the image features to the same semantic space to obtain an image semantic vector group and a text semantic vector group.

[0128] Step S42: Calculate the semantic similarity between the image semantic vector group and the text semantic vector group.

[0129] Step S43, feature fusion is performed between the text features and the image features according to the semantic similarity, and the multi-modal fusion features are obtained.

[0130] Step S5, overall view extraction is performed on the industrial document based on the multi-modal fusion features.

[0131] Specifically, in combination with the extracted structured data and the multi-modal features extracted by the subsystem in the industrial document, the overall view of the document is extracted, the information is finally integrated, the content hierarchical relationship tree of the document is generated according to the contribution degree of the extracted content, and support is provided for subsequent downstream applications (document retrieval, graph construction).

[0132] The document name and the multi-modal fusion features in step S4 are input into the document recognition view segment module, and the sentences related to the overall view of the document are recognized through the multi-modal fusion features. In the target view extraction module of the view segment, the bert_concept model is obtained by fine-tuning BERT using the data of the target view extraction of the view segment, the encoding of the input is completed, and the (view segment, argument) pair is obtained through the classifier, wherein the view segment is a sentence or a plurality of continuous sentences in the document, and the argument is a subsequence of the document name.

[0133] In an embodiment, the overall view extraction on the industrial document based on the multi-modal fusion features comprises: obtaining a view segment according to the multi-modal fusion features, so as to realize the overall view extraction on the industrial document based on the view segment.

[0134] The multi-modal information extraction method for industrial documents provided by the application receives an industrial document, quickly analyzes the document, classifies the content in the document according to structured, semi-structured and unstructured, retains strong related text paragraphs and tables and other information in the semi-structured content in the document, extracts the strong related entities and relationships in the semi-structured content through the semi-structured data extraction subsystem, retains strong related real scene pictures, flowcharts and design diagrams and other information in the unstructured content in the document, generates corresponding image descriptions through the unstructured data extraction subsystem, realizes multi-modal data fusion based on the feature vectors extracted from the structured information in the document and the feature vectors extracted from the unstructured information, realizes the overall view extraction on the document based on the multi-modal fusion features, industrial document text data and other factors, and improves the accuracy of the view extraction.

[0135] It should be noted that the protection scope of the multi-modal information extraction method for industrial documents is not limited to the execution order of the steps listed in the embodiment, and any scheme realized by adding, replacing or replacing steps of the prior art according to the principle of the application is included in the protection scope of the application.

[0136] As Figure 12 shown in one embodiment, the industrial document-oriented multi-modal information extraction system of the present application comprises:

[0137] A first acquisition module 121 is configured to acquire semi-structured first text information and unstructured second text information from an industrial document.

[0138] A first extraction module 122 is configured to extract text features based on the first text information.

[0139] A second extraction module 123 is configured to extract image features based on the second text information.

[0140] A second acquisition module 124 is configured to acquire multi-modal fusion features according to the text features and the image features.

[0141] An opinion extraction module 125 is configured to perform overall opinion extraction on the industrial document based on the multi-modal fusion features.

[0142] It should be noted that the structures and principles of the first acquisition module 121, the first extraction module 122, the second extraction module 123, the second acquisition module 124 and the opinion extraction module 125 correspond to the steps (steps S1-S5) in the above industrial document-oriented multi-modal information extraction method one by one, and thus will not be described here.

[0143] It should be noted that the division of each module of the above system is only a logical functional division, and all or part of them can be integrated into one physical entity, or can be physically separated. These modules can all be implemented in the form of software called by a processing element; they can all be implemented in the form of hardware; or some modules can be implemented in the form of software called by a processing element, and some modules can be implemented in the form of hardware. For example, the x module can be a separately established processing element, or it can be integrated into a chip of the above system, in addition, it can also be stored in the form of program code in the memory of the above system, and the function of the above x module is called and executed by a processing element of the above system. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be independently implemented. The processing element mentioned here can be an integrated circuit with signal processing capability. In the implementation process, each step of the above method or each module can be completed by the integrated logic circuit of hardware or the instruction of software in the processing element.

[0144] For example, the modules above can be one or more integrated circuits configured to implement the methods above, such as one or more Application Specific Integrated Circuits (ASICs), or one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs), etc. For another example, when a certain module above is implemented in the form of a processing element scheduling code, the processing element can be a general purpose processor, such as a Central Processing Unit (CPU) or other processor that can invoke code. For another example, the modules can be integrated together to implement in the form of a System-On-a-Chip (SOC).

[0145] It should be noted that the industrial document-oriented multi-modal information extraction system of the present application can implement the industrial document-oriented multi-modal information extraction method of the present application, but the implementation device of the industrial document-oriented multi-modal information extraction method of the present application includes but is not limited to the structure of the industrial document-oriented multi-modal information extraction system listed in the present embodiment, and any modification and replacement of the existing technology according to the principle of the present application is included in the protection scope of the present application.

[0146] The storage medium of the present application stores a computer program, which is executed by a processor to implement the above-mentioned industrial document-oriented multi-modal information extraction method. The storage medium includes various media that can store program codes, such as Read-Only Memory (ROM), Random Access Memory (RAM), magnetic disk, U disk, memory card, optical disk, etc.

[0147] Any combination of one or more computer readable medium can be utilized. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium can be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0148] A computer readable signal medium can include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal can take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium can be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0149] Program code embodied on a computer readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0150] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In an embodiment, the present application is directed to computer program code embodied in a computer readable medium for execution by a computer processing unit.

[0151] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0152] These computer program instructions can also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to operate in a specific manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks.

[0153] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0154] The electronic device of the present application comprises a processor and a memory.

[0155] The memory is used for storing computer programs; preferably, the memory comprises a ROM, a RAM, a disk, a U disk, a memory card, or various media capable of storing program codes such as an optical disk.

[0156] The processor is connected with the memory, and is used for executing the computer programs stored in the memory, so as to enable the electronic device to execute the above-mentioned industrial document-oriented multi-modal information extraction method.

[0157] Preferably, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0158] To sum up, the industrial document-oriented multi-modal information extraction method, system, medium and electronic equipment of the present application develop a set of industrial document-oriented multi-modal information extraction method from the aspects of efficient processing of document types, deep mining of potential rules, and high condensation of key information, improve the accuracy of viewpoint extraction; therefore, the present application effectively overcomes the various shortcomings in the prior art and has high industrial utilization value.

[0159] The above embodiments only exemplarily illustrate the principles and effects of the present application, and are not used to limit the present application. Any person skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the present application. Therefore, all equivalent modifications or changes completed by those skilled in the art without departing from the spirit and technical thought disclosed by the present application should be covered by the claims of the present application.

Claims

1. A multimodal information extraction method for industrial documents, characterized by: The method comprises the following steps: obtaining semi-structured first text information and unstructured second text information from an industrial document; extracting text features based on the first text information; the text feature extraction based on the first text information comprises: optimizing an entity relationship extraction model according to an industrial concept library, so as to perform entity relationship extraction on the first text information based on the entity relationship extraction model, and realize that in the entity relationship extraction process, the characters in the first text information are encoded into the text features by introducing an industrial specialized BERT base model; extracting image features based on the second text information; extracting image features based on the second text information specifically includes: adopting a CLIP visual encoder, thereby extracting visual image representations related to language in the image ; obtaining multi-modal fusion features according to the text features and the image features; specifically comprising: Multi-modal representation alignment mapping: using unsupervised learning self-organizing mapping algorithm, image representation extracted by CLIP model , clustering by competitive learning, getting image semantic vector group , self-organizing network initializes the weight vector of the node in the mapping with Gaussian distribution, for a certain image feature , first calculate the similarity between the real value vector and the weight vector of the mapping node according to the Euclidean distance function, the node with the smallest distance is classified as the best matching unit, which will decrease with the number of iterations and distance; the update formula of the weight is: ； Where, represents the step size, u represents the best matching unit, represents the learning coefficient, Represents neurons and In step length After training, similar elements in the image representation are regarded as representations of the same semantic information, and the image features of the real-valued vector are Converted into multiple image semantic vector groups with similar semantic information , similarly, the text features extracted by the Bert model Using the same algorithm, project them into the same semantic space to form a text semantic vector group ; Multimodal representation alignment fusion: In order to fuse the features of different modalities with similar semantics, we first calculate the cosine distance between the feature vector groups of the two modalities and find modal features with similar similarity. Any eigenvector in , calculate the cosine distance between each feature vector in the image semantic vector group, and take the feature vector with the smallest cosine distance Viewed as similar semantics, and fused into , and obtain the final multimodal fusion feature set , its formula is as follows: ； In the formula, Sim(*) is a similarity measure formula, and for any one and When the similarity of the two is the highest among all vector combinations, feature fusion is performed. extracting the overall view of the industrial document based on the multi-modal fusion features; Opinion snippet screening: Specifically, the multimodal fusion features or text features for a section z of a given document are input into a sigmoid classifier to determine whether it contains an opinion about the document; ； wherein M is a multi-modal fusion feature, is a text feature, is a probability that a sentence contains a document opinion; Setting For the viewpoint tolerance, the sentence with ≤ is regarded as the content that may represent the viewpoint of the document, and all continuous viewpoint sentences are concatenated to form a viewpoint segment. Document-oriented opinion extraction: Given an opinion snippet u, directly list out the document name Among all spans, select the best span related to the opinion snippet as the target argument of the opinion snippet u, take the document name / section title / subsection title In span a, take the opinion snippet u and span a as the input of the model together, input into the opinion target extraction BERT for encoding, get the span a as the score of the opinion snippet u through the Sigmoid classifier, and take the span with the highest score as the target argument of the opinion snippet u. 2.The industrial document oriented multi-modal information abstraction method of claim 1, wherein, the step of obtaining semi-structured first text information and unstructured second text information from an industrial document comprises: analyzing the industrial document to divide the industrial document into structured information, semi-structured information and unstructured information; screening the semi-structured information and the unstructured information respectively to obtain the first text information and the second text information. 3.The industrial document oriented multi-modal information abstraction method of claim 2, wherein, the step of obtaining semi-structured first text information and unstructured second text information from an industrial document further comprises: extracting the structured information based on an industrial knowledge graph and an industrial concept library. 4.The industrial document oriented multi-modal information abstraction method of claim 1, wherein, the step of extracting image features based on the second text information comprises: extracting fine-grained semantic information in the second text information based on an image encoder and a pre-trained language model; outputting the image features related to the fine-grained semantic information.

5. The industrial document oriented multi-modal information abstraction method of claim 1, wherein, the step of obtaining multi-modal fusion features according to the text features and the image features comprises: mapping the text features and the image features to the same semantic space to obtain an image semantic vector group and a text semantic vector group; calculating the semantic similarity between the image semantic vector group and the text semantic vector group; performing feature fusion between the text features and the image features according to the semantic similarity to obtain the multi-modal fusion features.

6. The industrial document oriented multi-modal information abstraction method of claim 1, wherein, the step of extracting the overall view of the industrial document based on the multi-modal fusion features comprises: obtaining a view segment according to the multi-modal fusion features, so as to realize the extraction of the overall view of the industrial document based on the view segment. comprises:

7. A multimodal information extraction system for industrial documents implemented using the method of claim 1, characterized in that: a first obtaining module for obtaining semi-structured first text information and unstructured second text information from an industrial document; a first extracting module for extracting text features based on the first text information; the text feature extraction based on the first text information comprises: optimizing an entity relationship extraction model according to an industrial concept library, so as to perform entity relationship extraction on the first text information based on the entity relationship extraction model, and realize that in the entity relationship extraction process, the characters in the first text information are encoded into the text features by introducing an industrial specialized BERT base model; a second extracting module for extracting image features based on the second text information; a second obtaining module for obtaining multi-modal fusion features according to the text features and the image features; a view extraction module for extracting the overall view of the industrial document based on the multi-modal fusion features. ​ 8. A storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the industrial document-oriented multi-modal information extraction method in any one of claims 1 to 6.

9. An electronic device, comprising: Comprising: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory, so that the electronic device executes the industrial document-oriented multi-modal information extraction method in any one of claims 1 to 6.

Images