Nuclear power safety report data extraction method and system based on multi-modal feature fusion

Abstract

This invention proposes a method and system for extracting nuclear power safety report data based on multimodal feature fusion, belonging to the field of data processing technology. The method includes: acquiring a nuclear power safety report document; performing OCR recognition and paragraph segmentation on the report document to obtain first text data; using a dynamic template matching and semantic-driven extraction mechanism to perform text semantic understanding on the first text data to obtain text feature vectors; segmenting the report document into pages and identifying fonts and sizes to generate first layout data; using a cross-page table reconstruction algorithm to detect cells in the first layout data and concatenating cross-page tables to obtain layout feature vectors; aligning the text feature vectors and layout feature vectors and inputting them into a large-scale pre-trained language model to obtain semantic vectors; fusing the text feature vectors, layout feature vectors, and semantic vectors to form a composite expression unit, and intelligently extracting structured data based on the composite expression unit. This invention effectively identifies semantic entities and structural relationships in nuclear power reports.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, and in particular relates to a method and system for extracting nuclear power safety report data based on multimodal feature fusion. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of the nuclear power industry, nuclear safety management, as a core component of ensuring the safe operation of nuclear facilities and public health, has received increasing attention from regulatory agencies and nuclear power companies. In actual operation and supervision, nuclear power plants generate a large amount of nuclear safety-related reporting data annually, such as operation event reports, safety analysis reports, equipment failure reports, and regulatory inspection reports. These reports typically exist in a mixed format of structured and unstructured data, encompassing multimodal information such as text, tables, images, and diagrams. The content is complex, the formats diverse, the information redundant, and difficult to standardize. Traditional nuclear safety report data processing methods largely rely on manual reading and extraction, which is inefficient, costly, and highly subjective, failing to meet the demands of modern nuclear power's digital and intelligent development.

[0004] Currently, with the rapid development of artificial intelligence technology, intelligent information extraction technologies based on natural language processing (NLP), computer vision (CV), and deep learning have achieved remarkable results in fields such as finance, healthcare, and law. However, directly applying such technologies to the nuclear power sector still faces many challenges:

[0005] First, nuclear power reports are highly specialized and use complex terminology, making them difficult to understand and recognize accurately using general semantic models.

[0006] Second, a large amount of information in nuclear power reports is presented in the form of images, drawings and tables, and relying solely on text extraction methods cannot achieve comprehensive information extraction.

[0007] Third, there are issues such as upstream and downstream references, semantic connections, and content duplication among the reports, resulting in serious information redundancy and increasing the complexity of extraction.

[0008] Finally, regulatory compliance requirements have set higher standards for the completeness, accuracy, and traceability of reported content, and existing technologies are significantly lacking in the reliability and interpretability of information extraction from reports. Summary of the Invention

[0009] To overcome the shortcomings of existing technologies, this invention proposes a method and system for extracting nuclear power safety report data based on multimodal feature fusion. This aims to solve technical challenges in the nuclear power industry, such as low efficiency, poor accuracy, significant differences in wording, and difficulty in recognizing cross-page tables in unstructured regulatory reports. By integrating multimodal features such as text, images, and semantics, a unified data extraction framework is established, enhancing the structured, standardized, and intelligent processing capabilities of key information in nuclear power documents. This provides a high-quality, high-precision data foundation for nuclear safety situation analysis and regulatory support. This invention not only effectively alleviates the pressure of manual processing and improves the automation level of data processing, but also helps to promote the nuclear power industry towards digital and intelligent regulation and operation, possessing significant engineering application value and broad market prospects.

[0010] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0011] In a first aspect, the present invention discloses a method for extracting nuclear power safety report data based on multimodal feature fusion, comprising:

[0012] Obtain nuclear power safety report documents;

[0013] The report document is subjected to OCR recognition and paragraph segmentation to obtain the first text data; a dynamic template matching and semantic-driven extraction mechanism is used to perform text semantic understanding on the first text data to obtain text feature vectors;

[0014] The report document is segmented into pages and font size is identified to generate first layout data; a cross-page table reconstruction algorithm is used to detect cells in the first layout data and concatenate the cross-page table to obtain a layout feature vector;

[0015] The text feature vector and the layout feature vector are aligned and input into a large pre-trained language model to obtain the semantic vector.

[0016] The text feature vector, layout feature vector, and semantic vector are fused to form a composite expression unit, and structured data is intelligently extracted based on the composite expression unit.

[0017] Secondly, this invention discloses a nuclear power safety report data extraction system based on multimodal feature fusion, comprising:

[0018] The data acquisition module is used to acquire nuclear power safety report documents;

[0019] The text parsing module is used to perform OCR recognition and paragraph segmentation on the report document to obtain the first text data; and to perform text semantic understanding on the first text data using a dynamic template matching and semantic-driven extraction mechanism to obtain a text feature vector.

[0020] The layout analysis module is used to segment the report document into pages and identify fonts and sizes to generate first layout data; a cross-page table reconstruction algorithm is used to detect cells in the first layout data and concatenate the cross-page table to obtain a layout feature vector;

[0021] The semantic analysis module is used to perform feature alignment between the text feature vector and the layout feature vector and input them into a large pre-trained language model to obtain semantic vectors;

[0022] The intelligent extraction module is used to fuse the text feature vector, layout feature vector and semantic vector to form a composite expression unit, and intelligently extract structured data based on the composite expression unit.

[0023] Thirdly, the present invention discloses an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when run by the processor, complete the steps of the above-mentioned nuclear power safety report data extraction method based on multimodal feature fusion.

[0024] Fourthly, the present invention discloses a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps of the above-mentioned nuclear power safety report data extraction method based on multimodal feature fusion.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0026] This invention overcomes the reliance of traditional information extraction on single modalities (such as text or images) by constructing an analytical matrix that fuses three-dimensional features of text, layout, and semantics. This significantly improves the depth of understanding and accuracy of content extraction in complex document structures. Particularly in nuclear power safety reports with mixed text and graphics, diverse title styles, and complex table structures, it can more effectively identify semantic entities and structural relationships, significantly reducing the risk of misidentification and omission.

[0027] The cross-page table continuity reconstruction mechanism proposed in this invention can automatically detect boundaries, model cell associations, and stitch cross-page data when dealing with multi-page extended tables. This greatly improves the completeness and consistency of table information extraction and solves the long-standing technical bottlenecks such as table breaks and field misalignments in existing technologies.

[0028] In terms of semantic-driven content extraction, this invention designs a dynamic template matching mechanism and an automatic prompt word generation method based on a nuclear power professional knowledge base. Combined with the contextual understanding capabilities of a large language model, it achieves accurate identification of complex, flexible, and non-standard expressions. Specifically, when extracting key fields such as experimental conclusions, risk levels, and rectification measures, the system can adaptively generate constrained instructions based on the context, significantly improving its compatibility and generalization ability for diverse expressions.

[0029] To address the issues of content ambiguity and structural confusion in reports, this invention introduces a layout feature compensation and semantic confidence judgment mechanism. This enables the system to maintain a high accuracy rate and robustness in information extraction even when faced with complex situations such as semantic ambiguity, overlapping fields, and abnormal layout, ensuring that the extraction results have good engineering practical value and interpretability.

[0030] This invention possesses excellent system integration capabilities and engineering deployability. It is suitable for deployment in internal document processing systems of nuclear power plants and design institutes, as well as for integration into compliance review systems of nuclear safety regulatory agencies. It can broadly support the digital transformation and intelligent regulatory upgrade of the nuclear power industry, providing solid support for building an efficient, transparent, and secure nuclear safety data processing system.

[0031] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0032] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0033] Figure 1 This is a flowchart of the nuclear power safety report data extraction method based on multimodal feature fusion as described in Embodiment 1 of the present invention.

[0034] Figure 2 This is a flowchart of the dynamic template matching and semantic-driven extraction mechanism described in Embodiment 1 of the present invention.

[0035] Figure 3 This is a flowchart of the cross-page table reconstruction algorithm described in Embodiment 1 of the present invention. Detailed Implementation

[0036] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0037] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0038] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0039] Example 1

[0040] In one or more embodiments, a method for extracting nuclear power safety report data based on multimodal feature fusion is disclosed, which constructs a multimodal parsing engine and integrates three subsystems: an OCR engine, a native PDF parsing engine, and a semantic modeling engine, to achieve comprehensive and accurate capture of report information.

[0041] This embodiment constructs a method and system for intelligent extraction of nuclear power safety report data that integrates multimodal features and possesses text understanding, image recognition, and semantic modeling capabilities. It processes various data formats in reports, including text, tables, and images; automatically identifies and extracts key nuclear safety information, such as event time, location, related equipment, scope of impact, root cause, and handling measures; improves extraction accuracy and robustness through model fusion mechanisms; and combines knowledge graphs and other technologies to achieve semantic association and visualization of the extraction results, thereby significantly improving the efficiency of nuclear power safety data processing and contributing to the construction of nuclear safety decision support and knowledge management systems. Figure 1 As shown, it includes the following steps:

[0042] Step S1: Obtain the nuclear power safety report document; the nuclear power safety report is usually presented in PDF format, and contains a large amount of text and graphics mixed with nested structures and dense terminology. Its content data is unstructured.

[0043] Step S2: Use the text parsing module to perform OCR recognition and paragraph segmentation on the report document to obtain the first text data; use dynamic template matching and semantic-driven extraction mechanism to perform text semantic understanding on the first text data to obtain text feature vectors;

[0044] Step S2-1: The text parsing module uses an OCR engine to parse the scanned report. It extracts text, row and column structure, font features and image boundaries through a deep learning image recognition model, performs OCR recognition and paragraph segmentation on the report document, and obtains the first text data.

[0045] Step S2-2: Apply a dynamic template matching and semantic-driven extraction mechanism to the first text data, such as... Figure 2 As shown, by combining regular expressions, domain knowledge, semantic similarity, and language model prompting techniques (Prompt Engineering), high-precision extraction of key content is achieved:

[0046] Step S2-2-1: Identify the title hierarchy structure of the first text data: Construct a "Nuclear Safety Report Title Rule Base", which includes common chapter numbering patterns (such as "1.", "1.2.3", "III.", etc.) and use regular expressions combined with semantic similarity models (such as SBERT) to identify multi-level title relationships and construct a chapter tree structure.

[0047] Step S2-2-2: Verify the chapter titles and evaluate semantic similarity of the constructed chapter tree structure to obtain an updated template, including the title table of contents and title content; based on the updated template, select a dynamic template to construct a semantic prompt word extraction model;

[0048] The process includes verifying chapter titles by matching the title table of contents with regular expressions and checking the semantic correctness of the title content based on hierarchical context; evaluating semantic similarity by setting appropriate thresholds to calculate the semantic coherence and reasonableness of the title content within its context; and dynamic template selection by using template weight calculations to ensure the use of the latest templates and by using domain ontology constraint injection to improve the accuracy of title extraction.

[0049] The semantic prompt word extraction model automatically generates Prompt templates with constraints for different field types (such as equipment name, test conclusion, risk level), such as "extract all equipment status paragraphs containing the words 'operating failure'" or "exclude rows containing 'planned maintenance' and only keep the equipment data table that has experienced failures".

[0050] Step S2-2-3: Introduce a two-layer filtering mechanism to the semantic prompt word extraction model. First, perform initial screening by title keyword matching. This initial screening includes inputting the title text of the nuclear power safety report and a preset keyword library. Based on the TF-IDF weighted cosine similarity algorithm, calculate the matching degree between the title text and the keyword library, and output the title library with a similarity threshold > 0.7. Then, perform contextual semantic judgment on the text content. Specifically, use a pre-trained BERT model to bidirectionally encode the title and its associated paragraph text after the initial screening to output semantic vectors. Output titles and associated content with a semantic confidence score > 0.85 to eliminate ambiguous expressions and obtain the initially screened title content, thereby reducing the impact of misidentification and semantic ambiguity.

[0051] Step S2-2-4: Perform layout feature compensation on the PDF document to obtain the correct text information. Layout feature compensation includes font feature analysis, typesetting feature analysis and decision correction algorithm.

[0052] Specifically, font feature analysis involves: recognizing text using CV technology, analyzing stroke thickness and font type, and extracting text information using a CNN model. Layout feature analysis involves: analyzing the document's global layout structure and local distribution patterns, and using edge detection and connected component analysis to locate the physical position and logical hierarchy of headings, body text, and lists.

[0053] The decision correction algorithm is as follows: based on font and layout features, the extraction results are dynamically adjusted, higher confidence weights are assigned to high-frequency fonts or specific layout areas, and the spatial relationship between the title and the body text is checked.

[0054] When there is semantic ambiguity, a layout feature decision mechanism is activated, such as comparing font differences, indentation distance, and vertical spacing between the title and the body text, to strengthen the determination of paragraph belonging.

[0055] Step S2-2-5: Perform data alignment and standardization on the title and content after layout compensation to output a structured text feature vector; wherein, the structured text feature vector includes chapter hierarchy relationships and metadata. In this embodiment, the metadata includes text content such as device records.

[0056] Specifically, the extracted results undergo data format standardization processing, including field unit conversion, time format standardization, numerical precision standardization, and field label alignment, to ensure information consistency and comparability.

[0057] This embodiment proposes a dynamic template extraction strategy of "structural rule base + large model semantic matching + automatic prompt word generation", including chapter identification and structural modeling based on title number and layout features; semantic filtering of content blocks using nuclear power domain knowledge base and semantic similarity model; and automatic generation of instructions with constraints based on the extraction target field using the prompt word engineering method to achieve semantic-driven extraction of complex content; which significantly improves the adaptability to diverse semantic expressions.

[0058] Nuclear power plant documentation is highly template-driven, with chapter titles typically following certain rules, but the wording varies considerably. Furthermore, documents contain numerous structurally different but semantically similar content blocks (such as event descriptions and hazard mitigation measures), limiting the effectiveness of traditional template extraction methods. This embodiment utilizes a dynamic template matching and semantic-driven extraction mechanism to accurately extract high-value information such as "equipment status change descriptions," "risk management recommendations," and "regulatory recommendations" from complex, varied, and multimodal document environments. It also adapts to changes in report structure, demonstrating good portability and versatility.

[0059] Step S3: Use the layout analysis module to segment the report document into pages and identify fonts and sizes to generate the first layout data; use the cross-page table reconstruction algorithm to detect cells in the first layout data and concatenate the cross-page tables to obtain the layout feature vector;

[0060] Step S3-1: The layout analysis module uses the PDF native parsing engine to process the report document, segment the page and extract its embedded text stream, page layout information, font format (such as bold, italic, font size) and paragraph structure data to generate the first layout data.

[0061] Step S3-2: Apply a cross-page table reconstruction algorithm to the first format data, such as... Figure 3As shown, the intelligent modeling and reconstruction of table structures based on multi-feature fusion and deep learning networks includes the following key steps:

[0062] Step S3-2-1: Detect table lines and identify physical boundaries: Use image processing methods and deep edge detection models (such as Canny or CNN-based Line Detector) to detect all table boundary line segments in the document and vectorize them.

[0063] Step S3-2-2: Determine cell distribution and page break positions for vectorized tables: Based on page size and the vertical extension trend of the table, detect possible page break positions; by comparing the similarity of table structures (number of rows and columns, column width, etc.) on the front and back pages, preliminarily determine whether they are the same table, and extract the page break table, including cell text sequence and position features.

[0064] Step S3-2-3: Construct a Bi-LSTM row and column continuity model to determine the continuity relationship of cross-page tables: Construct a bidirectional long short-term memory network, input the cell text sequence and position features of the table before and after the cross-page, capture its contextual relevance, determine the row and column continuity relationship of the cells, and output the complete content of the cross-page table.

[0065] Step S3-2-4: For the header and content of the cross-page table, a cross-page reconstruction and completion mechanism is adopted. Multimodal features of the cross-page cells are merged and integrated through a self-attention mechanism. Cross-page breakpoints are identified and the header is separated and marked. The missing borders are automatically spliced ​​and logically continuous column header information is completed by encoding the table content and associating the header with the content, so as to realize the automatic restoration of the cross-page table structure. After cross-page table splicing, header inheritance relationship annotation and table semantic completion, a structured layout feature vector is obtained.

[0066] Tables spanning multiple pages are a common occurrence in nuclear power safety reports. Conventional text parsing methods struggle to determine whether paginated tables maintain the same structural continuity, leading to information breaks, field misalignments, or missing content. This embodiment utilizes a cross-page table reconstruction algorithm to significantly improve the completeness and semantic consistency of information extraction from complex tables, exhibiting exceptionally high structural restoration accuracy, particularly for data fields in event-related tables (such as "time," "responsible department," and "status").

[0067] This embodiment constructs a reconstruction algorithm based on table line detection, cell distribution similarity analysis, Bi-LSTM row and column continuity modeling, and a self-attention mechanism. This method can automatically identify data segments belonging to the same logical table across multiple pages and perform structural reconstruction, field completion, and boundary restoration, providing a reliable guarantee for the structured extraction of multi-page tables in complex reports.

[0068] In steps S2 and S3, the OCR module and the native PDF parsing module work together to complete multi-source information collection and preliminary structured processing. The document is parsed into multiple semantic blocks, each containing: the original text content, page number and location information (XY coordinates), font, paragraph structure and other layout features, and a semantic vector representation of the text context. Finally, all semantic blocks are integrated into a multi-dimensional matrix, which serves as the basic input for subsequent extraction models.

[0069] Step S4: Align the text feature vector with the layout feature vector and input them into a large pre-trained language model to obtain semantic vectors;

[0070] In this embodiment, the semantic modeling engine uses large pre-trained language models (such as BERT, LLM, etc.) to generate contextual semantic vectors of text fragments to capture potential logical connections in semantics. The input of the large pre-trained language model is a feature vector obtained by aligning and fusing text feature vectors and layout feature vectors. The text feature vectors and layout feature vectors are mapped to the same semantic space through a linear projection layer, and the weights are adjusted through a cross-modal attention mechanism to align the two in terms of dimension and distribution.

[0071] By applying a "dual-engine parallel strategy" of OCR and native parsing to the same document, multi-source heterogeneous information is obtained. Then, a coordinate mapping algorithm is used to achieve spatial alignment of the two types of data on the page, ensuring the consistency and integrity of text, structure and semantic information in the spatial dimension.

[0072] Step S5: The text feature vector, layout feature vector and semantic vector are fused to form a composite expression unit, and structured data is intelligently extracted based on the composite expression unit.

[0073] Step S5-1: This embodiment uses a three-dimensional feature matrix modeling method to fuse text feature vectors, layout feature vectors and semantic vectors to provide a structured semantic foundation for subsequent information extraction.

[0074] Among them, the text feature vector represents the text dimension information: extracting the original text content in the document and labeling its page number, paragraph position, and context range;

[0075] Layout feature vectors represent layout dimension information: constructing a spatial structure model of the document, including paragraph spacing, font size, bold / underline status, border segment distribution (for table detection), etc.

[0076] Semantic vectors represent semantic dimension information: a contextual semantic vector is generated for each paragraph or sentence through a language model, representing its deep semantic features and binding it to structural position.

[0077] The 3D feature matrix modeling method specifically involves a gated attention fusion network that dynamically adjusts the weight contributions of three types of modal features (text content, layout features, and semantic vectors) in different scenarios.

[0078] In the table reconstruction task, the system assigns higher weight to layout features for boundary recognition and structure normalization.

[0079] In the text paragraph comprehension task, the contribution of semantic vectors is dynamically enhanced for context-related reasoning.

[0080] In the title recognition task, structural and semantic features are given equal importance, and a robust recognition mechanism is formed by combining title rules and vector semantic judgment.

[0081] After the fusion of three-dimensional features, a "semantic-format-content" composite expression unit of the document is formed. Based on this, hierarchical content extraction, rule matching and entity recognition are further carried out, providing high-quality feature support for downstream intelligent extraction models.

[0082] Step S5-2: Intelligent extraction of structured data based on composite expression units, including two dimensions of target extraction and content alignment, and two levels of structure-level extraction and content-level extraction:

[0083] The structural extraction process specifically involves: identifying the document structure based on heading levels and semantic indicators, extracting units such as chapters, sub-chaps, tables, and paragraphs, and constructing a document tree structure;

[0084] Content-level extraction specifically involves: extracting field-level data from the target content based on dynamic templates and semantic prompt word generators, including key fields such as event time, cause analysis, rectification measures, and verification conclusions;

[0085] The extracted results will be cleaned and standardized by the post-processing module, including: table break repair, unit conversion (such as MWe, MWd), numerical deduplication and alignment, and expression conversion (such as "analysis suggests serious consequences" → "high-risk conclusion").

[0086] Specifically, table break repair involves restoring table continuity by deleting page breaks, enabling cross-page row break attributes, or merging cells / inserting rows. After detecting cross-page tables, section breaks are automatically deleted or independent tables are manually cut and pasted to merge, and column width alignment is verified.

[0087] Unit conversion: Create a unit conversion dictionary, parse strings with units, extract the numerical value and unit, convert according to rules, and output the standardized result;

[0088] Numerical deduplication and alignment: Hash tables store unique values, group by column / field type, fill missing values ​​with NULL or the mean, and ensure consistent row and column dimensions;

[0089] Expression transformation: The NLP model extracts the core components, removes modifiers, and generates a concise expression. The final output can be used as a structured data table, a knowledge graph node, or for data analysis and decision-making in downstream AI systems.

[0090] Preferably, this embodiment provides the following engineering implementation scheme, including:

[0091] (1) Construction of nuclear power layout feature library: Collect and label a large number of nuclear power regulatory report samples, and train layout classification model to identify differences in title style, paragraph style and table structure.

[0092] (2) Development of a dynamic prompt word generator: Automatically generate prompt word combinations based on field extraction type and context scenario to adapt to different expression methods of large language models.

[0093] (3) Cross-page table detection and splicing service: Develop an independent service component for the document preprocessing stage to identify and uniformly merge tables with continuity.

[0094] (4) Adapter fine-tuning mechanism: Based on the nuclear power professional corpus, the large language model is lightly fine-tuned to improve its ability to understand industry language such as terminology, abbreviations, and metaphors.

[0095] (5) Embedded deployment and data security strategy: Provides a localized deployment version that supports use in the intranet environment of regulatory agencies and nuclear power companies, meeting the data confidentiality and auditing requirements.

[0096] This embodiment designs an engineering implementation plan based on key implementation elements and deployment scenarios to ensure the deployability of this method in a real nuclear power enterprise documentation environment.

[0097] This embodiment fully integrates various technologies such as natural language processing, computer vision, structural analysis, and semantic modeling to systematically solve the key challenges in extracting unstructured document information in the nuclear power industry. Compared with existing shallow extraction methods that mainly rely on OCR recognition or template rules, this invention achieves several technological breakthroughs in multimodal analysis, table reconstruction, semantic understanding, and contextual linkage.

[0098] Example 2

[0099] In one or more embodiments, a nuclear power safety report data extraction system based on multimodal feature fusion is disclosed, specifically including:

[0100] The data acquisition module is used to acquire nuclear power safety report documents;

[0101] The text parsing module is used to perform OCR recognition and paragraph segmentation on the report document to obtain the first text data; and to perform text semantic understanding on the first text data using a dynamic template matching and semantic-driven extraction mechanism to obtain a text feature vector.

[0102] The layout analysis module is used to segment the report document into pages and identify fonts and sizes to generate first layout data; a cross-page table reconstruction algorithm is used to detect cells in the first layout data and concatenate the cross-page table to obtain a layout feature vector;

[0103] The semantic analysis module is used to perform feature alignment between the text feature vector and the layout feature vector and input them into a large pre-trained language model to obtain semantic vectors;

[0104] The intelligent extraction module is used to fuse the text feature vector, layout feature vector and semantic vector to form a composite expression unit, and intelligently extract structured data based on the composite expression unit.

[0105] Example 3

[0106] This embodiment provides an electronic device, including a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the computer instructions are executed by the processor, they complete the steps of the above-described method for extracting nuclear power safety report data based on multimodal feature fusion.

[0107] Example 4

[0108] This embodiment provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps of the above-described method for extracting nuclear power safety report data based on multimodal feature fusion.

[0109] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0110] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0111] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment, whereby a series of operational steps are performed to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0112] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0113] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for nuclear power safety report data extraction based on multi-modal feature fusion, characterized in that, include: Obtain nuclear power safety report documents; The report document is subjected to OCR recognition and paragraph segmentation to obtain the first text data; A dynamic template matching and semantic-driven extraction mechanism is used to perform text semantic understanding on the first text data to obtain text feature vectors; The report document is segmented into pages and font size is identified to generate first layout data; a cross-page table reconstruction algorithm is used to detect cells in the first layout data and concatenate the cross-page table to obtain a layout feature vector; The text feature vector and the layout feature vector are aligned and input into a large pre-trained language model to obtain the semantic vector. The text feature vector, layout feature vector, and semantic vector are fused to form a composite expression unit, and structured data is intelligently extracted based on the composite expression unit; The dynamic template matching and semantic-driven extraction mechanism includes: The title hierarchy structure of the first text data is identified, a nuclear safety report title rule base is constructed, and a multi-level title relationship is identified using regular expressions combined with a semantic similarity model to construct a chapter tree structure. The constructed chapter tree structure is validated, chapter titles are evaluated, semantic similarity is assessed, and a semantic cue word extraction model is built. A two-layer filtering mechanism is introduced into the semantic prompt word extraction model to perform initial screening of keyword matching and contextual semantic judgment; The first text information is obtained by performing layout feature compensation on the document. The first text information after layout feature compensation is then aligned and standardized to output a structured text feature vector. The cross-page table reconstruction algorithm includes: Detect table lines, identify physical boundaries, and vectorize them; determine cell distribution and page spanning positions for the vectorized table. A Bi-LSTM row and column continuity model is constructed to determine the continuity relationship of cross-page tables. A cross-page reconstruction and padding mechanism is adopted, and a self-attention mechanism is used to merge and fuse multimodal features of cross-page cells to restore the cross-page table structure and obtain a structured layout feature vector.

2. The nuclear power safety report data extraction method based on multimodal feature fusion as described in claim 1, characterized in that, The layout feature compensation includes font feature analysis, typesetting feature analysis, and decision correction algorithm; Specifically, text is identified using CV technology, stroke thickness and font type are analyzed, and features are extracted using a CNN model to obtain font features; the global layout structure and local distribution patterns of the document are analyzed, and edge detection and connected component analysis are used to locate the physical position and logical hierarchy of the title, body text, and list to obtain layout features; The decision correction algorithm dynamically adjusts the extraction results based on font and layout features, assigns higher confidence weights to high-frequency fonts or specific layout areas, and checks the spatial relationship between the title and the body text.

3. The nuclear power safety report data extraction method based on multimodal feature fusion as described in claim 1, characterized in that, The semantic vector is obtained by aligning the text feature vector with the layout feature vector and inputting it into a large pre-trained language model. Specifically, this includes: The text feature vector and the layout feature vector are aligned and fused. The resulting feature vector is then mapped to the same semantic space through a linear projection layer. The weights are then adjusted through a cross-modal attention mechanism to align the two in terms of dimension and distribution. The fused feature vector is then input into a large pre-trained language model to generate the context semantic vector of the text fragment.

4. The nuclear power safety report data extraction method based on multimodal feature fusion as described in claim 1, characterized in that, A three-dimensional feature matrix modeling method is adopted to fuse text feature vectors, layout feature vectors and semantic vectors. Specifically, the gated attention fusion network dynamically adjusts the weight contribution of the three types of modal features in different scenarios to form a "semantic-layout-content" composite expression unit of the document.

5. The nuclear power safety report data extraction method based on multimodal feature fusion as described in claim 1, characterized in that, Intelligent extraction of structured data based on the aforementioned composite expression unit includes multi-granularity target extraction and content alignment. The intelligent extraction comprises two levels: structure-level extraction and content-level extraction. Identify document structure based on heading level and semantic indicators, extract chapters, sub-chaps, tables, and paragraph units, and construct a document tree structure; Based on dynamic templates and semantic prompt word generators, the target content is extracted at the field level, including key fields such as event time, cause analysis, rectification measures, and verification conclusions.

6. A nuclear power safety report data extraction system based on multimodal feature fusion, characterized in that, include: The data acquisition module is used to acquire nuclear power safety report documents; The text parsing module is used to perform OCR recognition and paragraph segmentation on the report document to obtain the first text data; and to perform text semantic understanding on the first text data using a dynamic template matching and semantic-driven extraction mechanism to obtain a text feature vector. The layout analysis module is used to segment the report document into pages and identify fonts and sizes to generate first layout data; a cross-page table reconstruction algorithm is used to detect cells in the first layout data and concatenate the cross-page table to obtain a layout feature vector; The semantic analysis module is used to perform feature alignment between the text feature vector and the layout feature vector and input them into a large pre-trained language model to obtain semantic vectors; The intelligent extraction module is used to fuse the text feature vector, layout feature vector and semantic vector to form a composite expression unit, and intelligently extract structured data based on the composite expression unit; The dynamic template matching and semantic-driven extraction mechanism includes: The title hierarchy structure of the first text data is identified, a nuclear safety report title rule base is constructed, and a multi-level title relationship is identified using regular expressions combined with a semantic similarity model to construct a chapter tree structure. The constructed chapter tree structure is validated, chapter titles are evaluated, semantic similarity is assessed, and a semantic cue word extraction model is built. A two-layer filtering mechanism is introduced into the semantic prompt word extraction model to perform initial screening of keyword matching and contextual semantic judgment; The first text information is obtained by performing layout feature compensation on the document. The first text information after layout feature compensation is then aligned and standardized to output a structured text feature vector. The cross-page table reconstruction algorithm includes: Detect table lines, identify physical boundaries, and vectorize them; determine cell distribution and page spanning positions for the vectorized table. A Bi-LSTM row and column continuity model is constructed to determine the continuity relationship of cross-page tables. A cross-page reconstruction and padding mechanism is adopted, and a self-attention mechanism is used to merge and fuse multimodal features of cross-page cells to restore the cross-page table structure and obtain a structured layout feature vector.

7. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, complete the nuclear power safety report data extraction method based on multimodal feature fusion as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, complete the nuclear power safety report data extraction method based on multimodal feature fusion as described in any one of claims 1-5.

Images