A method and system for intelligent retrieval and review of multimodal power engineering documents

Abstract

The present invention relates to a method and system for intelligent retrieval and review of multimodal electric power engineering documents, which belongs to the field of natural language processing technology and solves the problems of poor relevance and incomplete review of existing multimodal document retrieval results. The method includes: dividing each electric power engineering document into logical areas, extracting multimodal fusion vectors of each logical area and aggregating them to obtain document feature vectors; calculating the multi-level similarity between the received query terms and each electric power engineering document, and obtaining multiple electric power engineering documents with the highest similarity as retrieval documents; dynamically expanding each rule in the rule base, and using the expanded rules to conduct compliance review of each retrieved document according to the document feature vector of the retrieved document, and generating a review report. Efficient and accurate retrieval of multimodal electric power engineering documents is achieved, and the comprehensiveness and flexibility of the review are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method and system for intelligent retrieval and review of multimodal electric power engineering documents. Background Art

[0002] As power engineering projects continue to expand, the types and volumes of documents involved are increasing. These include multimodal documents such as design specifications, drawings, and material lists. These documents often contain complex and interrelated information in a variety of formats, including text, images, tables, and drawings. Traditional manual retrieval and review processes are inefficient and prone to omissions and errors. This is particularly true in compliance reviews, where the lack of a systematic approach leads to delayed or incomplete problem discovery, making them incapable of meeting the demands of large-scale, complex projects.

[0003] Most existing algorithms are optimized for specific tasks within a single modality (e.g., processing only text or images). They are unable to handle the interdependencies among multimodal information (e.g., text, tables, drawings, etc.) found in power engineering documents, resulting in incomplete information extraction and insufficient contextual understanding. A small number of algorithms lack effective cross-modal feature fusion and contextual association modeling when processing multimodal documents, making it difficult to capture the semantic relationships between the various information forms within the document.

[0004] In terms of document retrieval, existing methods often rely on static indexes or simple keyword matching and lack dynamic adjustment mechanisms, resulting in poor relevance of retrieval results. Especially when faced with complex query requirements, it is impossible to achieve efficient and accurate document positioning and sorting.

[0005] In compliance reviews, existing technologies mostly rely on static matching based on fixed rule bases, making them difficult to adapt to the dynamic changes in power engineering specifications. Furthermore, the rule matching process lacks intelligent semantic extension capabilities, resulting in limited review coverage and the tendency to miss important issues. They also fail to effectively detect contradictions or inconsistencies between different forms of content within a document. For example, it can be difficult to detect inconsistencies between parameters in a design specification and annotations in a drawing. Summary of the Invention

[0006] In view of the above analysis, the embodiments of the present invention aim to provide a method and system for intelligent retrieval and review of multimodal power engineering documents, so as to solve the existing problems of low efficiency in multimodal document recognition, poor relevance of retrieval results and incomplete review.

[0007] In one aspect, an embodiment of the present invention provides a method for intelligent retrieval and review of multimodal power engineering documents, comprising the following steps:

[0008] Each power engineering document is divided into logical areas, and the multimodal fusion vectors of each logical area are extracted and aggregated to obtain the document feature vector;

[0009] Calculate the multi-level similarity between the received query word and each electric power engineering document, and obtain multiple electric power engineering documents with the highest similarity as retrieval documents;

[0010] Each rule in the rule base is dynamically expanded, and the expanded rules are used to perform compliance review on each retrieved document according to the document feature vector of the retrieved document, and a review report is generated.

[0011] Based on the further improvement of the above method, the method further includes:

[0012] According to the document feature vectors and the expanded rules of the electric power engineering document, a matching rule set of each electric power engineering document is obtained; according to the matching rule set and the feature vectors of each mode in the electric power engineering document, a consistency review is performed on each electric power engineering document to obtain a review result.

[0013] Based on the further improvement of the above method, each power engineering document is divided into logical areas, including:

[0014] After pre-processing, each power engineering document is divided into regions to obtain multiple initial blocks;

[0015] The context feature value is calculated according to the semantic similarity and visual similarity between any two initial blocks, and the initial blocks are merged and split according to all the context feature values ​​to obtain the adjusted logical regions.

[0016] Based on the further improvement of the above method, the multimodal fusion vectors of each logical region are extracted and aggregated to obtain the document feature vector using a multimodal visual language model;

[0017] The multimodal visual language model is an improvement on the multimodal ViT-BERT model. The ViT model and BERT model are used to extract the feature vectors of each modality in each logical area, and multimodal weights are introduced to weightedly concatenate the feature vectors of each modality to obtain the multimodal fusion vectors of each logical area. The self-attention mechanism is then used to calculate the attention score and weight the multimodal fusion vectors of each logical area to obtain the document feature vector.

[0018] Based on the further improvement of the above method, the multi-level similarity between the received query word and each power engineering document is calculated, including:

[0019] Extracting a query feature vector of the same length as the multimodal fusion vector from the received query word;

[0020] Calculate the matching score of each logical region in each power engineering document based on the query feature vector and the multimodal fusion vector of each logical region in each power engineering document, and then weight them to obtain the single document matching score of each power engineering document;

[0021] Based on the single-document matching score of each electric power engineering document, the collaborative weight between each electric power engineering document and each other electric power engineering document is calculated, and then the cross-document enhancement score of each electric power engineering document is calculated by weighted calculation;

[0022] According to the single document matching score and cross-document enhancement score of each power engineering document, the multi-level similarity of each power engineering document is obtained by weighting.

[0023] Based on the further improvement of the above method, each rule in the rule base is dynamically expanded based on the weight between any two rules in the rule base. For each rule, the semantic embedding model is used to extract the embedding vectors of all rules in the rule base and perform weighted summation according to the corresponding weights to obtain the feature vector of the expanded rule.

[0024] Based on the further improvement of the above method, the compliance review of each retrieved document is performed using the expanded rules according to the document feature vector of the retrieved document, including:

[0025] The similarities between the document feature vector of each retrieved document and the feature vector of each expanded rule are calculated in sequence, and the similarities are used as multiple pending matching degrees. The sum of all pending matching degrees of each retrieved document is calculated to obtain the total matching degree of each retrieved document.

[0026] If the total matching degree is not less than the first matching threshold, the corresponding retrieval document is a compliant retrieval document; otherwise, the corresponding retrieval document is a non-compliant retrieval document, and then the expanded rules corresponding to the matching degree to be reviewed that is less than the second matching threshold in the non-compliant retrieval document are obtained to obtain the non-compliant items of the non-compliant retrieval document.

[0027] Based on the further improvement of the above method, the matching rule set of each electric power engineering document is obtained according to the document feature vector of the electric power engineering document and the expanded rules. The similarity between the document feature vector of each electric power engineering document and the feature vector of each expanded rule is calculated, and the expanded rule corresponding to the similarity greater than the trigger threshold is used as the matching rule set of the corresponding electric power engineering document.

[0028] Based on the further improvement of the above method, each power engineering document is reviewed for consistency according to the matching rule set and the feature vector of each mode in the power engineering document. The review results include:

[0029] Calculate the sum of similarities between the document feature vector of the power engineering document and the feature vector of each rule in the matching rule set to obtain the rule consistency score;

[0030] Calculate the similarity and consistency weight between the feature vectors of any two modes in the power engineering document, and obtain the modal consistency score by weighted summation;

[0031] The rule consistency score and modal consistency score are weighted and summed according to their respective importance weights to obtain the total consistency score; when the total consistency score is greater than or equal to the review threshold, the power engineering document passes the review; otherwise, the power engineering document fails the review.

[0032] Based on the further improvement of the above method, the consistency weight is calculated using the softmax function according to the similarity between the feature vectors of any two modalities.

[0033] Based on the further improvement of the above method, each power engineering document is preprocessed, including:

[0034] Convert all power engineering documents into PDF documents;

[0035] Extract the image from the PDF document, perform noise reduction processing using the image processing library, and then reinsert the noise-reduced image into the corresponding position in the PDF document.

[0036] Based on a further improvement of the above method, the initial block is obtained by detecting the chapter numbers and titles in the electric power engineering document and using them as chapter markers, and then dividing the electric power engineering document into regions according to the chapter markers.

[0037] Based on a further improvement of the above method, the initial block includes: content text and / or table and / or engineering drawing;

[0038] The context feature values ​​are calculated based on the semantic similarity and visual similarity between any two initial blocks, including:

[0039] Extracting text and engineering drawings from each initial block, the text including content text and / or text in a table;

[0040] Use the natural language processing model to obtain text feature vectors, and use the image vision model to obtain the image feature vectors corresponding to the engineering drawings;

[0041] Calculate the similarity between the text feature vectors of any two initial blocks as semantic similarity, and the similarity between the image feature vectors as visual similarity;

[0042] The context feature value between the corresponding two initial blocks is obtained by weighted summation of semantic similarity and visual similarity.

[0043] Based on the further improvement of the above method, the initial blocks are merged and split according to all context feature values ​​to obtain the adjusted logical regions, including:

[0044] Identify each context feature value in turn, and merge the two initial blocks corresponding to the context feature values ​​greater than the merging threshold into the first region; otherwise, the initial block is directly used as the first region;

[0045] The gradient mean of each first region is identified in turn, and the first region corresponding to the gradient mean greater than the splitting threshold is split into two second regions; otherwise, the first region is directly used as the second region; all the second regions are the adjusted logical regions.

[0046] Based on the further improvement of the above method, the merging threshold is the sum of the mean and standard deviation of all context feature values; the splitting threshold is the weighted sum of the gradient mean and gradient standard deviation of all first regions.

[0047] Based on a further improvement of the above method, the first region corresponding to the gradient mean greater than the split threshold is split into two second regions, including:

[0048] Calculate the edge strength of each pixel in the first area according to the Sobel operator;

[0049] According to the edge strength of each pixel point, the edge density of each column and each row is calculated, and the column and row with the largest edge density are obtained. If the edge density of the largest column is greater than the edge density of the largest row, the largest column is split into two left and right sub-regions as the second region. Otherwise, the largest row is split into two upper and lower sub-regions as the second region.

[0050] Based on a further improvement of the above method, the edge strength of each pixel in the first region is the gradient modulus calculated based on the horizontal gradient component and the vertical gradient component of each pixel; the horizontal gradient component and the vertical gradient component of each pixel are calculated based on the Sobel operator; and the edge density of each column and each row is obtained by summing up the edge strength of the pixels in each column and each row respectively.

[0051] Based on a further improvement of the above method, the matching score of each logical area in each power engineering document is calculated according to the query feature vector and the multimodal fusion vector of each logical area in each power engineering document. The matching score is obtained by calculating the Hadamard product of the normalized query feature vector and the normalized multimodal fusion vector of each logical area and taking the maximum component value therefrom.

[0052] In another aspect, an embodiment of the present invention provides a system for intelligent retrieval and review of multimodal power engineering documents, comprising:

[0053] The document feature extraction module is used to divide each power engineering document into logical areas, extract the multimodal fusion vectors of each logical area, and aggregate them to obtain the document feature vector;

[0054] A multimodal retrieval module is used to calculate the multi-level similarity between the received query term and each power engineering document, and obtain multiple power engineering documents with the highest similarity as retrieval documents;

[0055] The compliance review module is used to dynamically expand each rule in the rule base, perform compliance review on each retrieved document using the expanded rules according to the document feature vector of the retrieved document, and generate a review report.

[0056] Based on the further improvement of the above system, the system also includes:

[0057] The collaborative review module is used to obtain the matching rule set of each power engineering document based on the document feature vector and the expanded rules of the power engineering document; based on the matching rule set and the feature vector of each mode in the power engineering document, each power engineering document is reviewed for consistency to obtain the review result.

[0058] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0059] 1. It realizes intelligent retrieval and automated review of power engineering documents without human intervention, avoids the risks of omissions and tampering caused by manual operations, and improves the reliability and legality of the review work.

[0060] 2. Through dynamic logical area division, multimodal feature extraction and multi-level similarity retrieval, efficient and accurate processing of multimodal documents is achieved; through dynamic expansion rules and compliance review, the integrity and flexibility of the entire process are ensured, significantly improving the efficiency and quality of power engineering document management and review.

[0061] 3. On the one hand, based on an in-depth analysis of the contextual features of document regions, the boundaries of each region are dynamically adjusted in real time and with high precision. This process focuses on the aggregate analysis of dynamic regional features to achieve a refined division and definition of document regions. On the other hand, multimodal feature information of each logical region is deeply extracted and then aggregated to obtain a comprehensive and rich document feature vector. This process focuses on the fusion of multimodal context and fully explores the correlation and complementarity between different modal information in the document. The combination of these two methods significantly improves the information extraction accuracy of power engineering documents, making them more accurately meet the needs of practical applications and providing more reliable support for document processing and analysis in the power engineering field.

[0062] 4. Based on user queries, each power engineering document is compared one by one to achieve accurate matching; dynamic triggering of cross-document collaborative search captures the semantic relevance between multiple documents, greatly improving the comprehensiveness and relevance of the retrieval.

[0063] 5. By expanding the rule base through semantic association, the rule coverage is increased to better meet the ever-changing engineering specifications of power engineering projects; by adjusting the expansion weight, the contribution of rules with high semantic relevance to the expansion rules is enhanced, thereby improving the generalization ability of the model.

[0064] 6. Dynamically select review rules based on document content to optimize review efficiency; by verifying the contextual consistency of multimodal content, ensure semantic consistency between data in multiple modes such as charts, text, and tables, and through the rule triggering mechanism, dynamically adjust the review process according to real-time needs to achieve a more efficient and flexible review process and ensure the comprehensiveness and accuracy of the review results.

[0065] In the present invention, the above-mentioned technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of the present invention will be described in the following description, and some advantages will become apparent from the description or be learned through practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. Throughout the drawings, the same reference symbols denote the same components.

[0067] Figure 1 This is a flow chart of a method for intelligent retrieval and review of multimodal power engineering documents in Example 1 of the present invention;

[0068] Figure 2 This is a schematic diagram of the system structure for intelligent retrieval and review of multimodal power engineering documents in Example 2 of the present invention. DETAILED DESCRIPTION

[0069] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and, together with the embodiments of the present invention, are used to illustrate the principles of the present invention and are not used to limit the scope of the present invention. It should be understood that the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0070] Those skilled in the art should be aware that the following specific embodiments or implementations are a series of optimized configurations listed in the present invention to further explain the specific content of the invention, and these configurations can be combined or used in conjunction with each other, unless the present invention explicitly states that some or a specific embodiment or implementation cannot be combined or used in conjunction with other embodiments or implementations. At the same time, the following specific embodiments or implementations are only intended to be optimized configurations and are not to be understood as limiting the scope of protection of the present invention.

[0071] Example 1

[0072] A specific embodiment of the present invention discloses a method for intelligent retrieval and review of multimodal power engineering documents, such as Figure 1 As shown, the following steps are included:

[0073] S1. Divide each power engineering document into logical areas, extract the multimodal fusion vectors of each logical area, and aggregate them to obtain the document feature vector.

[0074] It should be noted that power engineering documents usually contain multiple information forms such as text, tables, and engineering drawings. These parts are related to each other. This step is broken down into:

[0075] Step S11 divides each power engineering document into logical regions, conducts in-depth analysis of the document region contextual features, and dynamically adjusts the region boundaries in real time and accurately, focusing on the aggregate analysis of dynamic region features to achieve refined division and definition of each logical region of the document;

[0076] Step S12 deeply extracts the multimodal feature information of each logical area, and then aggregates it to obtain a comprehensive and rich document feature vector, focusing on multimodal context fusion, fully exploring the correlation and complementarity between different modal information in the document, and significantly improving the information extraction accuracy of power engineering documents.

[0077] Step S11 and step S12 are described in detail below.

[0078] S11. Divide each power engineering document into logical areas.

[0079] It should be noted that this step includes:

[0080] (1) After preprocessing, each power engineering document is divided into regions to obtain multiple initial blocks.

[0081] The preprocessing of each power engineering document mainly involves format standardization and noise removal, including:

[0082] Convert all power engineering documents into PDF documents;

[0083] Extract the image from the PDF document and use an image processing library, such as OpenCV, to perform median filtering and Gaussian filtering on the image to reduce noise, remove isolated black spots or noise specks, and improve the clarity of the image area in the document and reduce noise interference.

[0084] After the noise reduction process, the noise-reduced image is reinserted into the corresponding position in the PDF document.

[0085] Preferably, a region filtering method is used to remove interfering contents such as headers, footers, and page numbers in the PDF document.

[0086] Furthermore, the pre-processed power engineering document is preliminarily divided into regions to obtain a plurality of initial blocks.

[0087] The initial blocks of this step are obtained by detecting the chapter numbers and titles in the electric power engineering document and using them as chapter marks, and then dividing the electric power engineering document into regions according to the chapter marks.

[0088] Specifically, by parsing the PDF documents of power engineering, text information including text, font, font size and position is obtained.

[0089] Since titles are usually larger than body paragraphs, the font threshold can be set to distinguish between titles and body text. It is also possible to identify chapter numbers in text information according to the power engineering document specifications, and then divide the PDF document into regions according to the positions of titles and chapter numbers to obtain multiple initial blocks.

[0090] For example, by using a PDF parsing tool such as Python's PyMuPDF library to parse the power engineering document, text, tables, images, and vector graphics in the PDF can be extracted, and vector graphics can be converted into images.

[0091] (2) Calculate the context feature value based on the semantic similarity and visual similarity between any two initial blocks, merge and split the initial blocks according to all the context feature values ​​to obtain the adjusted logical region.

[0092] 1) Extract contextual feature values

[0093] It should be noted that each initial block obtained may include multiple modal information: content text and / or table and / or engineering drawing; among them, content text includes title and paragraph body, table includes table structure and table text, engineering drawing includes engineering drawing in image format and vector format, content text and table text are collectively referred to as text.

[0094] Furthermore, this step considers the similarity between the initial blocks from the two dimensions of text semantics and image vision, extracts context feature values ​​as the basis for dynamic region division, and then adjusts the region boundaries of the initial blocks.

[0095] It should be noted that the context feature value is calculated based on the semantic similarity and visual similarity between any two initial blocks, including:

[0096] ① Extract the text and engineering drawings in each initial block, where the text includes content text and / or text in a table.

[0097] It should be noted that there are many methods for extracting text and engineering drawings from the initial block, and this embodiment does not limit the method used.

[0098] For example, for text, use Python's PyMuPDF library to extract the text, or convert the initial block into an image and use OCR technology to recognize the text;

[0099] For engineering drawings, we determine whether the text contains keywords such as "drawing number", "legend" and "scale" to further determine whether there is an engineering drawing. If so, we use Python's PyMuPDF library to extract the image and vector graphic elements. If the engineering drawing is a vector graphic, we further obtain the bounding rectangle of the engineering drawing by identifying the line segments in the vector graphics and their closed relationships; alternatively, we can extract the engineering drawing area in the initial block by training a neural network.

[0100] Furthermore, the text in each initial block is preprocessed, including: removing stop words, punctuation marks and special symbols; using Chinese word segmentation tools, such as Jieba, to convert the text into a word sequence.

[0101] The engineering drawings in each initial block are preprocessed, including: converting to image formats of the same size and normalizing pixel values.

[0102] ② Use the natural language processing model to obtain text feature vectors, and use the image vision model to obtain the image feature vectors corresponding to the engineering drawings.

[0103] Use trained natural language processing models, such as the Bert model and Word2Vec model, to extract text feature vectors from the preprocessed text.

[0104] Use trained image vision models, such as the ViT model and the ResNet model, to extract image feature vectors from engineering drawings.

[0105] ③ Calculate the similarity between the text feature vectors of any two initial blocks as semantic similarity, and the similarity between the image feature vectors as visual similarity.

[0106] It should be noted that the similarity between text feature vectors and the similarity between image feature vectors are obtained by calculating the cosine similarity or dot product between the vectors.

[0107] ④ The weighted sum of semantic similarity and visual similarity is used to obtain the context feature value between the corresponding two initial blocks.

[0108] The context feature value between any two initial blocks is calculated using the following formula:

[0109] ,

[0110] in, Indicates the initial block and the initial block The context feature values ​​between ;

[0111] and Represent the weights of semantic similarity and visual similarity respectively;

[0112] and Represents the initial block and the initial block The semantic similarity and visual similarity between them.

[0113] 2) Dynamic region boundary adjustment

[0114] It should be noted that the initial blocks are merged and split according to all context feature values ​​to obtain the adjusted logical regions, including:

[0115] Identify each context feature value in turn, and merge the two initial blocks corresponding to the context feature values ​​greater than the merging threshold into the first region; otherwise, the initial block is directly used as the first region;

[0116] The gradient mean of each first region is identified in turn, and the first region corresponding to the gradient mean greater than the splitting threshold is split into two second regions; otherwise, the first region is directly used as the second region; all the second regions are the adjusted logical regions.

[0117] Specifically, when merging, the merge threshold is the mean of all contextual feature values and standard deviation The sum of the two is as follows:

[0118] ,

[0119] in, Indicates the number of initial blocks to merge.

[0120] when , indicating the initial block and the initial block Very similar, with strong contextual relevance, should be merged.

[0121] when , indicating the initial block and the initial block Insufficient similarity, weak contextual association, and remain independent.

[0122] For example, a document has 4 initial blocks, and the following context feature matrix is ​​constructed based on the context feature values ​​between any two initial blocks:

[0123]

[0124] According to the above formula, we can calculate , , , the diagonal values ​​do not participate in the merging operation.

[0125] therefore, , merge the initial blocks 1 and 2 into the first region; , the initial blocks 3 and 4 are not merged and are respectively used as the first regions.

[0126] Furthermore, all the first regions obtained after merging are split, and the split threshold is the mean gradient of all first regions and the standard deviation of the gradient The weighted sum of is as follows:

[0127] ,

[0128] in, The weight of the gradient standard deviation, usually set to 1 or 1.5.

[0129] It should be noted that, in order to improve calculation efficiency, the gradient of each pixel in the first region is the sum of the absolute value of the horizontal gradient component and the absolute value of the vertical gradient component.

[0130] The horizontal gradient component and the vertical gradient component are calculated based on the Sobel operator. The Sobel operator approximates the gradient by calculating the weighted difference of pixel values ​​in a 3×3 neighborhood, which can better suppress noise.

[0131] Furthermore, the gradient mean of all pixels in each first region is identified in turn. If it is greater than the split threshold, it means that the gradient inside the first region changes dramatically and needs to be split into two second regions. If it is less than or equal to the split threshold, it means that the gradient inside the first region changes smoothly and the first region remains unchanged and is directly used as the second region.

[0132] For example, a document has four first regions, and the mean gradient values ​​of each first region are [5.2, 8.0, 2.5, 4.3]. According to the above formula, we can calculate , ,exist When set to 1, the split threshold Since 8.0>7.2, the second first area needs to be split, and the other first areas remain unchanged.

[0133] Furthermore, a splitting method and a splitting boundary are obtained by edge detection, and the first region corresponding to the gradient mean value greater than the splitting threshold is split into two second regions, including:

[0134] Calculate the edge strength of each pixel in the first area according to the Sobel operator;

[0135] According to the edge strength of each pixel point, the edge density of each column and each row is calculated, and the column and row with the largest edge density are obtained. If the edge density of the largest column is greater than the edge density of the largest row, the largest column is split into two left and right sub-regions as the second region. Otherwise, the largest row is split into two upper and lower sub-regions as the second region.

[0136] It should be noted that the edge strength of each pixel in the first region is the gradient modulus of each pixel calculated based on the horizontal gradient component and the vertical gradient component of each pixel, and is used to reflect the value of the edge strength. The formula is as follows:

[0137] ,

[0138] in, Represents pixel points edge strength;

[0139] and Represents pixel points The horizontal and vertical gradient components at .

[0140] The edge density of each column and each row in the first region is calculated using the following formula:

[0141] ,

[0142] in, and Represents the first area height and width;

[0143] Indicates the edge density of the column;

[0144] Indicates the The edge density of the row.

[0145] Get the column and row with the largest edge density using the following formula:

[0146] ,

[0147] in, Indicates the column index with the largest edge density in the vertical direction;

[0148] Indicates the row index with the highest edge density in the horizontal direction.

[0149] If the edge density of the largest column is greater than the edge density of the largest row, indicating that the vertical edge is more obvious, the first area is divided into Split into left subregion and right subregion , the region boundary is expressed as:

[0150] .

[0151] If the edge density of the largest row is greater than the edge density of the largest column, indicating that the horizontal edge is more obvious, the first area is divided into Split into upper sub-areas and the lower sub-area , the region boundary is expressed as:

[0152] .

[0153] Finally, the sub-regions obtained by splitting and the first region that does not need to be split are both used as the second region, that is, the adjusted logical region.

[0154] S12. Extract the multimodal fusion vectors of each logical region and aggregate them to obtain the document feature vector.

[0155] It should be noted that each obtained logical area may also include multiple modal information: content text and / or tables and / or engineering drawings; therefore, the modal types contained in each logical area are first identified, and then the relevant attributes of each mode are extracted separately, so as to facilitate the extraction of the feature vector of the corresponding mode based on each attribute.

[0156] For the content text in each logical area, the extracted attributes include: each character content and character visual features, and the character visual features include but are not limited to: font, font size and width.

[0157] For example, the text information is extracted using Python's PyMuPDF library.

[0158] For the table in each logical area, the extracted attributes include: table text and table structural features. The table structural features include but are not limited to: the number of rows and columns in the table, the row number and column number of each cell in the table, and the relationship between cells.

[0159] For example, Python's PyMuPDF library is used to extract horizontal and vertical lines by analyzing the closed relationship of the lines; or Python's Camelot library is used to extract table information; or the structured parsing model TableNet is used to extract table information.

[0160] For the engineering drawings in each logical area, the engineering drawings of uniform size are obtained in the same manner as that in the initial block in step S11 .

[0161] Furthermore, in this embodiment, a multimodal visual language model is used to extract the multimodal fusion vectors of each logical region and aggregate them to obtain the document feature vector.

[0162] It should be noted that the multimodal visual language model is an improvement on the multimodal ViT-BERT model. The ViT model and BERT model are used to extract the feature vectors of each modality in each logical area, and multimodal weights are introduced to weightedly concatenate the feature vectors of each modality to obtain the multimodal fusion vectors of each logical area. The self-attention mechanism is then used to calculate the attention score and weight the multimodal fusion vectors of each logical area to obtain the document feature vector.

[0163] Specifically, the multimodal visual language model consists of an input feature extraction layer, a multimodal fusion layer, and a self-attention layer in sequence.

[0164] For each power engineering document, the content text, tables, and engineering drawing information extracted from each logical area are integrated and passed into the multimodal visual language model as document data;

[0165] The input feature extraction layer extracts the feature vectors of each modality in each logical area from the received document data through the ViT model and the BERT model, and passes them to the multimodal fusion layer;

[0166] The multimodal fusion layer introduces multimodal weights to perform weighted concatenation on the feature vectors of each modality to obtain the multimodal fusion vectors of each logical region and pass them to the self-attention layer;

[0167] The self-attention mechanism layer calculates the attention score through the self-attention mechanism and weights the multimodal fusion vector of each logical area to obtain the document feature vector.

[0168] In the input feature extraction layer:

[0169] The character visual features are encoded, and the BERT model is used to generate a semantic vector for each word in the content text. The semantic vector is then concatenated with the corresponding encoded character visual features to obtain the feature vector of each word, and then pooled to form the final text feature vector.

[0170] The structured features of the table cells are encoded. After using the BERT model to generate a semantic vector for the text of each cell in the table, it is concatenated with the corresponding encoded cell structured features and then pooled to form the final table feature vector.

[0171] The ViT model is used on engineering drawings to obtain image feature vectors.

[0172] In the multimodal fusion layer:

[0173] Multimodal weights are feature weights guided by domain rules. Based on the expert knowledge, experience, and common sense of the power industry, corresponding modal weights are set for the three modalities of content text, tables, and engineering drawings. The modal fusion vector after weighted splicing of each logical area is obtained through the following formula:

[0174] ,

[0175] in, Indicates the The first power engineering document Modal fusion vector of logical regions;

[0176] 、 and represent the modal weights of content text, tables, and engineering drawings, respectively;

[0177] 、 and Respectively represent The first power engineering document The text feature vector, table feature vector and image feature vector of each logical region.

[0178] In the self-attention layer:

[0179] The relationship between the modal fusion vectors of each logical region is calculated through the self-attention mechanism to obtain the attention score, and then the fusion vector of each logical region is weighted again to highlight the important logical region features and generate the document feature vector.

[0180] S2. Calculate the multi-level similarity between the received query term and each electric power engineering document, and obtain multiple electric power engineering documents with the highest similarity as retrieval documents.

[0181] It should be noted that this embodiment implements multi-level similarity retrieval when searching based on the query feature vector: it not only compares each power engineering document one by one to achieve accurate matching, but also captures the semantic relevance between multiple documents through cross-document collaborative search, thereby improving the comprehensiveness and flexibility of the retrieval.

[0182] Specifically, the multi-level similarity between the received query term and each power engineering document is calculated, including:

[0183] ① Extract the query feature vector with the same length as the multimodal fusion vector from the received query word.

[0184] It should be noted that the received query words may be multiple keywords input by the user, or may be obtained by segmenting the query text input by the user.

[0185] Furthermore, the trained Transformer model with self-attention mechanism is used for query words to dynamically adjust the importance of different words according to the context, thereby improving the expressive power of the output query feature vector.

[0186] ② Based on the query feature vector and the multimodal fusion vector of each logical region in each power engineering document, the matching score of each logical region in each power engineering document is calculated, and then the single document matching score of each power engineering document is obtained by weighting.

[0187] It should be noted that the matching score is obtained by calculating the Hadamard product of the query feature vector and the multimodal fusion vector of each logical area in each power engineering document, and taking the maximum component value from them; that is, the query feature vector and the multimodal fusion vector are multiplied element by element according to the feature dimension, and the maximum value across all feature dimensions is taken to obtain the matching score. The formula is as follows:

[0188] ,

[0189] in, Represents the query feature vector and the The first power engineering document Matching score of each logical region; , , Indicates the total number of power engineering documents; Indicates the The total number of logical areas divided into power engineering documents;

[0190] Indicates that the query feature vector is in Numerical representation on dimension; , Indicates the number of dimensions of the feature vector;

[0191] Indicates the The first power engineering document The multimodal fusion vector of the logical region is Numerical representation of the dimension.

[0192] Furthermore, the matching scores of the logical regions are weighted and summed to obtain the single document matching score. The formula is as follows:

[0193] ,

[0194] in, Indicates the Single document matching scores for power engineering documents;

[0195] Indicates the The first power engineering document Normalized weight of each logical region;

[0196] Represents the query feature vector and the The first power engineering document The matching score of each logical region.

[0197] ③ Based on the single document matching score of each electric power engineering document, calculate the collaborative weight between each electric power engineering document and each other electric power engineering document, and then calculate the cross-document enhancement score of each electric power engineering document.

[0198] It should be noted that the collaboration weight is the similarity weight between power engineering documents, which is calculated using the following formula:

[0199] ,

[0200] in, Indicates the The first power engineering document and the The collaborative weight between power engineering documents.

[0201] Then, the cross-document enhancement score of each power engineering document is calculated by weighting using the following formula:

[0202] ,

[0203] in, Indicates the Cross-document enhancement scores for power engineering documents.

[0204] ④ According to the single document matching score and cross-document enhancement score of each power engineering document, the multi-level similarity of each power engineering document is obtained by weighting.

[0205] Specifically, the multi-level similarity of each power engineering document is calculated using the following formula:

[0206] ,

[0207] in, Indicates the Multi-level similarity of power engineering documents;

[0208] and are hyperparameters, representing the weights of the single-document matching score and the cross-document enhancement score.

[0209] Finally, each electric power engineering document is sorted from high to low according to its multi-level similarity, and multiple electric power engineering documents with the highest similarity are obtained, that is, the top K electric power engineering documents are taken as the retrieval results.

[0210] Compared with the existing technology, this step utilizes a delayed interaction mechanism and a cross-document collaborative search strategy to calculate similarities layer by layer from logical areas in a single document to cross-documents, and returns the documents most relevant to the query content, thereby achieving efficient and accurate retrieval of multimodal power engineering documents.

[0211] S3. Dynamically expand each rule in the rule base, perform compliance review on each retrieved document using the expanded rules according to the document feature vector of the retrieved document, and generate a review report.

[0212] It should be noted that the rule base is built through domain expert knowledge, industry standards and historical data, including equipment parameter verification rules, layout design rules, bill of materials rules, etc., to cover common power engineering document review needs.

[0213] The rule base is represented as .

[0214] in, Indicates device parameter verification rules, such as voltage, current, and power factor. Device parameter verification rules are used to check whether the device parameters in the document meet the standards. For example, the transformer input voltage must meet the requirement of 10kV ± 5%.

[0215] Indicates layout design rules, such as conductor spacing, grounding requirements, and load distribution. Layout design rules are used to check whether the spatial arrangement of lines and equipment is reasonable. For example, the cable trench width must meet the requirement of being no less than 50 cm and no more than 100 cm.

[0216] Indicates bill of materials rules, such as cable specifications, fire resistance ratings, and current carrying capacity limits. Bill of materials rules are used to check whether electrical equipment selection complies with specifications. For example, the cross-sectional area of ​​copper conductors must be no less than 25 square millimeters and no more than 50 square millimeters.

[0217] Considering that different terms may be used to express the same rule in power engineering documents, in order to improve the generalization ability of the rules, the semantic association expansion of each rule in the rule base is performed to increase the rule coverage and better meet the ever-changing engineering specifications of power engineering.

[0218] Specifically, the dynamic expansion of each rule in the rule base is based on the weight between any two rules in the rule base. For each rule, the semantic embedding model is used to extract the embedding vectors of all rules in the rule base and perform weighted summation according to the corresponding weights to obtain the feature vector of the expanded rule.

[0219] Exemplarily, the semantic embedding model uses BERT or Word2Vec to convert each original rule in the rule base into an embedding vector.

[0220] This approach allows for combining multiple rules to accommodate different representations. By adjusting the weight of each extension in the rule base corresponding to each rule, the contribution of rules with high semantic relevance to the extension rule is enhanced, thereby improving the model's generalization capabilities.

[0221] It should be noted that the weight matrix between each rule and all other rules in the rule base is first established. , and then utilize the parallelism and efficiency of matrix operations to quickly complete the entire expansion process, and can be well integrated into the deep learning framework.

[0222] The following formula is used to express how the feature vector of each expanded rule is obtained:

[0223] ,

[0224] in, Indicates the The feature vector after the rule expansion;

[0225] Indicates the number of original rules in the rule base;

[0226] Indicates the Semantic embedding vector of rules;

[0227] It is used Rule extension The expansion weight of each rule can be set based on expert experience.

[0228] The feature vectors of the expanded rules can be used with trained generative language models (such as the GPT series and T5), using the feature vectors as input or guidance for the model to generate the corresponding text. Generative language models, by learning from large amounts of text data, have mastered the grammar and semantics of natural language and are able to generate semantically logical text based on the input feature vectors.

[0229] For example, a device parameter verification rule is "voltage: 10kV±5%" before expansion and becomes "voltage: 9.5Kv-10.5KV, rated voltage: 10KV, transformer allowable error ±5%" after expansion.

[0230] Furthermore, the expanded rules are used to perform compliance review on each retrieved document based on the document feature vector of the retrieved document, including:

[0231] The similarities between the document feature vector of each retrieved document and the feature vector of each expanded rule are calculated in sequence, and the similarities are used as multiple pending matching degrees. The sum of all pending matching degrees of each retrieved document is calculated to obtain the total matching degree of each retrieved document.

[0232] If the total matching degree is not less than the first matching threshold, the corresponding retrieval document is a compliant retrieval document; otherwise, the corresponding retrieval document is a non-compliant retrieval document, and then the expanded rules corresponding to the matching degree to be reviewed that is less than the second matching threshold in the non-compliant retrieval document are obtained to obtain the non-compliant items of the non-compliant retrieval document.

[0233] Specifically, the similarity between the document feature vector of each retrieved document and the feature vector of each expanded rule is obtained by calculating the cosine similarity or vector dot product method, which is used as the matching degree to be reviewed. The total matching degree of each retrieved document is then obtained by summarizing the calculated matching degrees to be reviewed.

[0234] When the total matching degree of the retrieved document is less than the first matching threshold, it means that there are rule mismatches in the retrieved document. Then, it is further identified whether each matching degree to be reviewed is less than the second matching threshold. If it is less, it means that the retrieved document does not match the expanded rule corresponding to the matching degree to be reviewed. These mismatched rules are organized into a review report for the retrieved document to help engineering personnel efficiently discover problems.

[0235] It should be noted that, considering that different modal information pits in the power engineering document may store contradictory information, the method of this embodiment further includes:

[0236] S4. Obtain a matching rule set for each power engineering document based on the document feature vector and the expanded rules of the power engineering document; perform a consistency review on each power engineering document based on the matching rule set and the feature vector of each modality in the power engineering document to obtain a review result.

[0237] It should be noted that before conducting a consistency review of power engineering documents, review rules are dynamically selected from the expanded rules based on the content of each document to form their own matching rule sets, avoiding redundant calculations and ensuring that the review is carried out on relevant content, optimizing review efficiency, and achieving a more efficient and flexible review process.

[0238] Specifically, based on the document feature vector of the power engineering document and the expanded rules, the matching rule set of each power engineering document is obtained by calculating the similarity between the document feature vector of each power engineering document and the feature vector of each expanded rule, and taking the expanded rule corresponding to the similarity greater than the trigger threshold as the matching rule set of the corresponding power engineering document.

[0239] It should be noted that the similarity between the document feature vector of each power engineering document and the feature vector of each expanded rule is obtained by calculating the cosine similarity using the following formula:

[0240] ,

[0241] in, Indicates the The document feature vector of the first power engineering document is The similarity between the feature vectors of the expanded rules;

[0242] Indicates the The document feature vector of the power engineering document is Numerical representation on dimension;

[0243] Indicates the The feature vector of the expanded rule is Numerical representation of the dimension.

[0244] It should be noted that The value range is [-1,1]. The higher the value, the higher the matching degree. The denominator is normalized during calculation to eliminate the order of magnitude effect of different documents and rule vectors.

[0245] Furthermore, if , then for the When conducting consistency review of power engineering documents, The expanded rule is triggered. ;

[0246] if , then for the When conducting consistency review of power engineering documents, The expanded rule is not triggered. .

[0247] Finally, the matching rule set for each power engineering document is:

[0248] ,

[0249] in, Indicates the A matching rule set for power engineering documents.

[0250] During implementation, each power engineering document is reviewed for consistency based on the matching rule set and the feature vector of each mode in the power engineering document. Review results are obtained, including:

[0251] ① Calculate the sum of the similarities between the document feature vector of the power engineering document and the feature vector of each rule in the matching rule set to obtain the rule consistency score.

[0252] The rule consistency score is calculated using the following formula:

[0253] ,

[0254] in, Indicates the Rule consistency scores for power engineering documents;

[0255] Indicates the The document feature vector of the power engineering document and the first The similarity between the feature vectors of the expanded rules.

[0256] ② Calculate the similarity and consistency weight between the feature vectors of any two modes in the power engineering document, and obtain the modal consistency score by weighted summation.

[0257] It should be noted that the feature vector of each mode in the power engineering document is obtained by performing maximum or average pooling on the feature vectors of the same mode in each logical area.

[0258] Each power engineering document may include multiple modal information. A power engineering document includes U modes, among which: The eigenvector of this mode is expressed as: ;No. The eigenvector of this mode is expressed as: .

[0259] The similarity between the feature vectors of the two modalities is calculated by calculating the cosine similarity using the following formula:

[0260] ,

[0261] in, Indicates the The first power engineering document The first mode and the The similarity between the feature vectors of the modalities;

[0262] Indicates the The first power engineering document The eigenvector of the mode is Numerical representation on dimension;

[0263] Indicates the The first power engineering document The eigenvector of the mode is Numerical representation of the dimension.

[0264] Furthermore, based on the similarity between the feature vectors of any two modalities, the softmax function is used to calculate the consistency weight between any two modalities. The formula is as follows:

[0265] ,

[0266] in, Indicates the The first power engineering document The first mode and the The consistency weight between the modalities;

[0267] Indicates the The first power engineering document The first mode and the The similarity between the feature vectors of the two modes.

[0268] This embodiment uses consistency weights to amplify the impact of high similarity between modalities and reduce the impact of low similarity between modalities.

[0269] Finally, the modal consistency score of the entire power engineering document is calculated using the following formula:

[0270] ,

[0271] in, Indicates the Modal consistency scores of power engineering documents.

[0272] ③ The rule consistency score and modal consistency score are weighted and summed according to their respective importance weights to obtain the total consistency score; when the total consistency score is greater than or equal to the review threshold, the power engineering document passes the review; otherwise, the power engineering document fails the review.

[0273] It should be noted that the total consistency score is calculated using the following formula:

[0274] ,

[0275] in, Indicates the The total consistency score of the power engineering documents;

[0276] and are hyperparameters, representing the weights of the rule consistency score and the modality consistency score respectively.

[0277] When Total consistency score of power engineering documents Greater than or equal to the review threshold When, The first power engineering document is reviewed, otherwise, the The power engineering documents failed the review and need to be manually reviewed.

[0278] Compared with the prior art, the intelligent retrieval and review method for multimodal power engineering documents provided in this embodiment has the following beneficial effects:

[0279] 1. It realizes intelligent retrieval and automated review of power engineering documents without human intervention, avoids the risks of omissions and tampering caused by manual operations, and improves the reliability and legality of the review work.

[0280] 2. Through dynamic logical area division, multimodal feature extraction and multi-level similarity retrieval, efficient and accurate processing of multimodal documents is achieved; through dynamic expansion rules and compliance review, the integrity and flexibility of the entire process are ensured, significantly improving the efficiency and quality of power engineering document management and review.

[0281] 3. On the one hand, based on an in-depth analysis of the contextual features of document regions, the boundaries of each region are dynamically adjusted in real time and with high precision. This process focuses on the aggregate analysis of dynamic regional features to achieve a refined division and definition of document regions. On the other hand, multimodal feature information of each logical region is deeply extracted and then aggregated to obtain a comprehensive and rich document feature vector. This process focuses on the fusion of multimodal context and fully explores the correlation and complementarity between different modal information in the document. The combination of these two methods significantly improves the information extraction accuracy of power engineering documents, making them more accurately meet the needs of practical applications and providing more reliable support for document processing and analysis in the power engineering field.

[0282] 4. Based on user queries, each power engineering document is compared one by one to achieve accurate matching; dynamic triggering of cross-document collaborative search captures the semantic relevance between multiple documents, greatly improving the comprehensiveness and relevance of the retrieval.

[0283] 5. By expanding the rule base through semantic association, the rule coverage is increased to better meet the ever-changing engineering specifications of power engineering projects; by adjusting the expansion weight, the contribution of rules with high semantic relevance to the expansion rules is enhanced, thereby improving the generalization ability of the model.

[0284] 6. Dynamically select review rules based on document content to optimize review efficiency; by verifying the contextual consistency of multimodal content, ensure semantic consistency between data in multiple modes such as charts, text, and tables, and through the rule triggering mechanism, dynamically adjust the review process according to real-time needs to achieve a more efficient and flexible review process and ensure the comprehensiveness and accuracy of the review results.

[0285] Example 2

[0286] Another embodiment of the present invention discloses a system for intelligent retrieval and review of multimodal power engineering documents, thereby implementing the method for intelligent retrieval and review of multimodal power engineering documents in Example 1. The specific implementation of each module is described in the corresponding description of Example 1.

[0287] like Figure 2 As shown, the system includes: a document feature extraction module 101, a multimodal retrieval module 102 and a compliance review module 103.

[0288] The document feature extraction module 101 is used to divide each power engineering document into logical regions, extract the multimodal fusion vectors of each logical region and aggregate them to obtain a document feature vector;

[0289] The multimodal retrieval module 102 is configured to calculate the multi-level similarity between the received query term and each electric power engineering document, and obtain multiple electric power engineering documents with the highest similarity as retrieval documents;

[0290] The compliance review module 103 is used to dynamically expand each rule in the rule base, perform compliance review on each retrieved document using the expanded rules according to the document feature vector of the retrieved document, and generate a review report.

[0291] Furthermore, the system also includes: a collaborative review module.

[0292] The collaborative review module is used to obtain the matching rule set of each power engineering document based on the document feature vector and the expanded rules of the power engineering document; based on the matching rule set and the feature vector of each mode in the power engineering document, each power engineering document is reviewed for consistency to obtain the review result.

[0293] Since the system for intelligent retrieval and review of multimodal power engineering documents in this embodiment and the method for intelligent retrieval and review of multimodal power engineering documents described above share similarities and can be mutually referenced, the descriptions herein are redundant and will not be repeated here. Since the principles of this system embodiment and the aforementioned method embodiment are the same, this system embodiment also has the corresponding technical effects of the aforementioned method embodiment.

[0294] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0295] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A method for intelligent retrieval and review of multimodal power engineering documents, characterized in that: The following steps are involved: Each power engineering document is divided into logical areas, and the multimodal fusion vectors of each logical area are extracted and aggregated to obtain the document feature vector; Calculate the multi-level similarity between the received query word and each electric power engineering document, and obtain multiple electric power engineering documents with the highest similarity as retrieval documents; Each rule in the rule base is dynamically expanded, and the compliance of each retrieved document is reviewed using the expanded rules according to the document feature vector of the retrieved document to generate a review report; the dynamic expansion of each rule in the rule base is based on the weight between any two rules in the rule base, and the embedding vectors of all rules in the rule base are extracted for each rule using a semantic embedding model and weighted summed according to the corresponding weights to obtain the feature vector of the expanded rule.

2. The method for intelligent retrieval and review of multimodal power engineering documents according to claim 1, characterized in that: The method further comprises: According to the document feature vectors and the expanded rules of the electric power engineering document, a matching rule set of each electric power engineering document is obtained; according to the matching rule set and the feature vectors of each mode in the electric power engineering document, a consistency review is performed on each electric power engineering document to obtain a review result.

3. The method for intelligent retrieval and review of multimodal power engineering documents according to claim 1, characterized in that: The logical division of each power engineering document includes: After pre-processing, each power engineering document is divided into regions to obtain multiple initial blocks; The context feature value is calculated according to the semantic similarity and visual similarity between any two initial blocks, and the initial blocks are merged and split according to all the context feature values ​​to obtain the adjusted logical regions.

4. The method for intelligent retrieval and review of multimodal power engineering documents according to claim 1, characterized in that: The multimodal fusion vectors of each logical region are extracted and aggregated to obtain the document feature vector using a multimodal visual language model; The multimodal visual language model is an improvement on the multimodal ViT-BERT model. The ViT model and BERT model are used to extract the feature vectors of each modality in each logical area, and multimodal weights are introduced to perform weighted concatenation on the feature vectors of each modality to obtain the multimodal fusion vectors of each logical area. The self-attention mechanism is then used to calculate the attention score and weight the multimodal fusion vectors of each logical area to obtain the document feature vector.

5. The method for intelligent retrieval and review of multimodal power engineering documents according to claim 1, characterized in that: The calculating of the multi-level similarity between the received query word and each power engineering document includes: Extracting a query feature vector of the same length as the multimodal fusion vector from the received query word; Calculate the matching score of each logical region in each power engineering document based on the query feature vector and the multimodal fusion vector of each logical region in each power engineering document, and then weight them to obtain the single document matching score of each power engineering document; Based on the single-document matching score of each electric power engineering document, the collaborative weight between each electric power engineering document and each other electric power engineering document is calculated, and then the cross-document enhancement score of each electric power engineering document is calculated by weighted calculation; According to the single document matching score and cross-document enhancement score of each power engineering document, the multi-level similarity of each power engineering document is obtained by weighting.

6. The method for intelligent retrieval and review of multimodal power engineering documents according to claim 1, characterized in that: The method of performing compliance review on each retrieved document using the expanded rules based on the document feature vector of the retrieved document includes: The similarities between the document feature vector of each retrieved document and the feature vector of each expanded rule are calculated in sequence, and the similarities are used as multiple pending matching degrees. The sum of all pending matching degrees of each retrieved document is calculated to obtain the total matching degree of each retrieved document. If the total matching degree is not less than the first matching threshold, the corresponding retrieval document is a compliant retrieval document; otherwise, the corresponding retrieval document is a non-compliant retrieval document, and then the expanded rules corresponding to the matching degree to be reviewed that is less than the second matching threshold in the non-compliant retrieval document are obtained to obtain the non-compliant items of the non-compliant retrieval document.

7. The method for intelligent retrieval and review of multimodal power engineering documents according to claim 2, characterized in that: The method of obtaining the matching rule set for each electric power engineering document based on the document feature vector and the expanded rules of the electric power engineering document is carried out by calculating the similarity between the document feature vector of each electric power engineering document and the feature vector of each expanded rule, and taking the expanded rule corresponding to the similarity greater than the trigger threshold as the matching rule set of the corresponding electric power engineering document.

8. The method for intelligent retrieval and review of multimodal power engineering documents according to claim 2, characterized in that: The consistency review of each power engineering document is performed based on the matching rule set and the feature vector of each mode in the power engineering document to obtain the review result, including: Calculate the sum of similarities between the document feature vector of the power engineering document and the feature vector of each rule in the matching rule set to obtain the rule consistency score; Calculate the similarity and consistency weight between the feature vectors of any two modes in the power engineering document, and obtain the modal consistency score by weighted summation; The rule consistency score and modal consistency score are weighted and summed according to their respective importance weights to obtain a total consistency score; when the total consistency score is greater than or equal to the review threshold, the power engineering document passes the review, otherwise, the power engineering document fails the review.

9. The method for intelligent retrieval and review of multimodal power engineering documents according to claim 8, characterized in that: The consistency weight is calculated using the softmax function based on the similarity between the feature vectors of any two modalities.

10. The method for intelligent retrieval and review of multimodal power engineering documents according to claim 3, characterized in that: The pre-processing of each power engineering document includes: Convert all power engineering documents into PDF documents; Extract the image from the PDF document, perform noise reduction processing using the image processing library, and then reinsert the noise-reduced image into the corresponding position in the PDF document.

11. The method for intelligent retrieval and review of multimodal power engineering documents according to claim 3, characterized in that: The initial blocks are obtained by detecting chapter numbers and titles in the electric power engineering document and using them as chapter marks, and then dividing the electric power engineering document into regions according to the chapter marks.

12. The method for intelligent retrieval and review of multimodal power engineering documents according to claim 3, characterized in that: The initial block includes: content text and / or table and / or engineering drawing; The calculating of the context feature value based on the semantic similarity and visual similarity between any two initial blocks includes: Extracting text and engineering drawings from each initial block, wherein the text includes content text and / or text in a table; Use the natural language processing model to obtain text feature vectors, and use the image vision model to obtain the image feature vectors corresponding to the engineering drawings; Calculate the similarity between the text feature vectors of any two initial blocks as semantic similarity, and the similarity between the image feature vectors as visual similarity; The context feature value between the corresponding two initial blocks is obtained by weighted summation of semantic similarity and visual similarity.

13. The method for intelligent retrieval and review of multimodal power engineering documents according to claim 3, characterized in that: The initial blocks are merged and split according to all context feature values ​​to obtain adjusted logical regions, including: Identify each context feature value in turn, and merge the two initial blocks corresponding to the context feature values ​​greater than the merging threshold into the first region; otherwise, the initial block is directly used as the first region; The gradient mean of each first region is identified in turn, and the first region corresponding to the gradient mean greater than the splitting threshold is split into two second regions; otherwise, the first region is directly used as the second region; all the second regions are the adjusted logical regions.

14. The method for intelligent retrieval and review of multimodal power engineering documents according to claim 13, characterized in that: The merging threshold is the sum of the mean and standard deviation of all context feature values; the splitting threshold is the weighted sum of the gradient mean and gradient standard deviation of all first regions.

15. The method for intelligent retrieval and review of multimodal power engineering documents according to claim 13, characterized in that: The step of splitting the first region corresponding to the gradient mean greater than the split threshold into two second regions includes: Calculate the edge strength of each pixel in the first area according to the Sobel operator; According to the edge strength of each pixel point, the edge density of each column and each row is calculated, and the column and row with the largest edge density are obtained. If the edge density of the largest column is greater than the edge density of the largest row, the largest column is split into two left and right sub-regions as the second region. Otherwise, the largest row is split into two upper and lower sub-regions as the second region.

16. The method for intelligent retrieval and review of multimodal power engineering documents according to claim 15, characterized in that: The edge strength of each pixel point in the first area is the gradient modulus calculated based on the horizontal gradient component and the vertical gradient component of each pixel point; the horizontal gradient component and the vertical gradient component of each pixel point are calculated based on the Sobel operator; the edge density of each column and each row is obtained by summarizing the edge strength of the pixel points in each column and each row respectively.

17. The method for intelligent retrieval and review of multimodal power engineering documents according to claim 5, characterized in that: The matching score of each logical area in each power engineering document is calculated based on the query feature vector and the multimodal fusion vector of each logical area in each power engineering document, which is obtained by calculating the Hadamard product of the normalized query feature vector and the normalized multimodal fusion vector of each logical area and taking the maximum component value therefrom.

18. A system for intelligent retrieval and review of multimodal power engineering documents, characterized in that: include: The document feature extraction module is used to divide each power engineering document into logical areas, extract the multimodal fusion vectors of each logical area, and aggregate them to obtain the document feature vector; A multimodal retrieval module is used to calculate the multi-level similarity between the received query term and each power engineering document, and obtain multiple power engineering documents with the highest similarity as retrieval documents; The compliance review module is used to dynamically expand each rule in the rule base, perform compliance review on each retrieved document using the expanded rules based on the document feature vector of the retrieved document, and generate a review report; the dynamic expansion of each rule in the rule base is based on the weight between any two rules in the rule base, and uses the semantic embedding model to extract the embedding vectors of all rules in the rule base for each rule and perform weighted summation according to the corresponding weights to obtain the feature vector of the expanded rule.

19. The system for intelligent retrieval and review of multimodal power engineering documents according to claim 18, characterized in that: The system further comprises: The collaborative review module is used to obtain the matching rule set of each power engineering document based on the document feature vector and the expanded rules of the power engineering document; based on the matching rule set and the feature vector of each mode in the power engineering document, each power engineering document is reviewed for consistency to obtain the review result.

Images