Intelligent system information acquisition method based on cooperation of multi-modal large model and large language model

Abstract

The invention discloses a system information intelligent acquisition method based on cooperation of a multi-modal large model and a large language model, and relates to the field of fault tree analysis. In the prior art, the efficiency of obtaining the topological structure and fault knowledge of a system is low, the accuracy is insufficient, and the cross-modal cooperative processing capability is lacked, the invention provides the following scheme: obtaining and preprocessing a system flow chart, and generating high-quality input data through functional decoupling and physical boundary decomposition; carrying out symbol recognition and text extraction to obtain preliminary topological information; forming a standardized system information file through a regular expression; inputting the file into a large language model to obtain a fault information set containing a fault mode and traceability information; integrating the topology file and the fault information to form system information data; and after consistency and integrity verification, outputting a standardized file which can be directly used for fault tree analysis and reliability evaluation. The method is suitable for system topology construction, fault tree analysis and safety and reliability evaluation work of a complex industrial system.

Description

Technical Field

[0001] It involves the field of fault tree analysis, specifically the intelligent acquisition of system information based on the collaboration of multimodal large models and large language models. Background Technology

[0002] In the design, operation, and maintenance of complex industrial systems, acquiring system structure information and equipment fault knowledge is a prerequisite for fault tree analysis and safety assessment. Currently, several studies in this field have attempted to automate the acquisition and processing of system information.

[0003] On the one hand, research based on image recognition is quite common. For example, OCR technology is used to recognize text and symbols in engineering drawings or flowcharts, and computer vision methods are used to extract the connection relationships between devices. These methods improve information processing efficiency to some extent, but they are often limited by the complexity of drawing symbols, the ambiguity of text, and the differences in the formats of multiple source images, resulting in limited recognition accuracy and difficulty in ensuring the integrity of structural information.

[0004] On the other hand, research based on text mining and natural language processing is increasingly being applied to system information processing. For example, some studies have attempted to extract equipment names, parameters, and failure modes from design specifications, operation and maintenance manuals, or incident reports through keyword matching, entity recognition, and rule parsing. These methods can support knowledge acquisition to some extent, but their drawbacks include a lack of deep semantic understanding, difficulty in correctly handling the ambiguity of technical terms, and inability to establish topological relationships between devices.

[0005] In recent years, some works have explored cross-modal information processing methods, such as jointly training image and text features to achieve multimodal information extraction. However, these methods still suffer from insufficient generalization and poor adaptability to complex industrial systems. Furthermore, in engineering applications, they often require significant manual intervention for post-processing, making it difficult to achieve efficient and standardized information acquisition.

[0006] In summary, existing technologies suffer from low efficiency, insufficient accuracy, and a lack of cross-modal collaborative processing capabilities in acquiring system topology and fault knowledge. Summary of the Invention

[0007] To address the shortcomings of existing technologies, such as low efficiency, insufficient accuracy, and lack of cross-modal collaborative processing capabilities in acquiring system topology and fault knowledge, the technical solution provided by this invention is as follows: A system information intelligent acquisition method based on the collaboration of multimodal large model and large language model includes: The steps include: acquiring and preprocessing the system flowchart, decomposing the system flowchart into functional decoupling and physical boundaries, and generating high-quality input data through Base64 encoding and resolution adaptive technology; The steps involve using a multimodal large model to perform symbol recognition and text extraction on the input data to obtain preliminary topology information including device number, device name, parameters, and connection order; The steps include extracting the preliminary topology information using regular expressions and formatting it into JSON, removing redundant or conflicting information, and outputting a standardized system information file. The steps are as follows: inputting the standardized system information file into the large language model, performing vectorized retrieval and semantic matching on the system information knowledge base, and outputting a set of fault information containing fault modes and source information. The step of integrating the standardized system information file with the fault information set to form a system information document that includes equipment, connection and fault information; The steps involve verifying the uniqueness, completeness, and logical consistency of the system information data, and outputting standardized system information files that can be directly used for fault tree analysis and reliability assessment.

[0008] Furthermore, a preferred implementation method is provided, in which modular processing of the flowchart is achieved through dual decomposition of functional decoupling and physical boundary during the acquisition and preprocessing of the system flowchart, thereby reducing symbol ambiguity and improving recognition accuracy.

[0009] Furthermore, a preferred implementation is provided in which, in the steps of symbol recognition and text extraction using a multimodal large model, a role-and-process dual-track prompt word paradigm is used to constrain the output logic to ensure the consistency and standardization of device number, device name, and connection relationship.

[0010] Furthermore, a preferred implementation is provided in which, during the process of extracting preliminary topology information using regular expressions and formatting JSON, uniqueness verification, integrity verification, and logical consistency verification are performed to ensure the reliability of the standardized system information file.

[0011] Furthermore, a preferred implementation method is provided in which, during the process of vectorized retrieval and semantic matching of the system information knowledge base, a deep neural network embedding model is used to convert the text into a high-dimensional semantic vector, and semantic matching is completed through a similarity algorithm.

[0012] Furthermore, a preferred implementation method is provided, in the process of integrating standardized system information files and fault information sets, by establishing a mapping relationship between devices and fault modes, a unified representation of "device-connection-fault" is formed.

[0013] Based on the same inventive concept, this invention also provides a system information intelligent acquisition device based on the collaboration of a multimodal large model and a large language model, comprising: The system flowchart is acquired and preprocessed, and the system flowchart is decomposed into functional decoupling and physical boundary, and a module for generating high-quality input data is generated through Base64 encoding and resolution adaptive technology. The input data is subjected to symbol recognition and text extraction using a multimodal large model to obtain a module containing preliminary topology information including device number, device name, parameters and connection order; A module that performs regular expression extraction and JSON formatting on the preliminary topology information, removes redundant or conflicting information, and outputs a standardized system information file. The standardized system information file is input into a large language model, and vectorized retrieval and semantic matching are performed on the system information knowledge base to output a module containing a fault information set including fault modes and source information. The standardized system information files and the fault information set are integrated to form a module of system information data that includes equipment, connection and fault information. A module that performs uniqueness, completeness, and logical consistency checks on the system information data and outputs standardized system information files that can be directly used for fault tree analysis and reliability assessment.

[0014] Based on the same inventive concept, the present invention also provides a computer storage medium for storing a computer program, wherein when the computer program is read by a computer, the computer executes the method described thereon.

[0015] Based on the same inventive concept, the present invention also provides a computer, including a processor and a storage medium, wherein when the processor reads a computer program stored in the storage medium, the computer executes the method described thereon.

[0016] Based on the same inventive concept, the present invention also provides a computer program product, which, when executed, implements the method described.

[0017] Compared with the prior art, the advantages of the technical solution provided by the present invention are as follows: This solution uses a multimodal large model to identify symbols, connections, and text labels in system flowcharts, and cleans them using regular expressions and JSON formatting to output structured and standardized topology information files. Compared with traditional OCR technology, this method avoids recognition errors caused by complex symbols or unclear annotations on drawings, and improves the completeness and accuracy of device and connection relationship extraction.

[0018] This solution utilizes a large language model to perform semantic retrieval on the system's knowledge base, leverages a vector database and similarity calculation to achieve accurate matching of fault modes, and annotates traceable text fragments and source documents. Compared to existing text mining methods based on keyword retrieval or rule parsing, this approach possesses stronger semantic understanding capabilities and retrieval accuracy, resolving issues of ambiguous terminology and missing context, thereby ensuring that the extracted fault knowledge is more comprehensive and reliable.

[0019] This solution employs a dual decomposition standard of functional decoupling and physical boundary in the input processing stage, combined with Base64 encoding and resolution adaptive technology, to ensure high-quality and diverse compatibility of input images. Compared to conventional methods that require customized parsing rules for specific drawing formats, this approach offers better generalization and adaptability in cross-system and cross-version data processing, effectively improving the accuracy and stability of topology analysis for complex industrial systems.

[0020] This solution constrains the output logic of the multimodal large model through a dual-track system of role-based and process-based prompts, ensuring consistency and standardization in the generated device numbers, names, and connection relationships. Compared to existing solutions that rely on manual post-processing correction, this approach reduces human intervention and avoids the impact of subjective differences on the results, thereby significantly improving the standardization and automation level of the information extraction process.

[0021] This solution proposes a two-level cleaning mechanism based on regular expressions and JSON to verify the uniqueness, completeness, and logical consistency of the initially extracted topology information and fault knowledge, outputting a standardized system information file that integrates all three elements. Compared to existing methods that rely on single-step cleaning or manual checks, this mechanism can complete multi-dimensional quality control in the automated processing stage, ensuring that the final result has higher credibility in subsequent fault tree construction and reliability analysis.

[0022] This solution constructs a scalable intelligent information acquisition framework that can adapt to various complex industrial systems such as nuclear energy and chemical engineering. Unlike existing research that focuses on specific data types or single application scenarios, this framework supports cross-modal and cross-domain knowledge integration, providing high-precision and traceable data support for complex systems and solving the bottleneck of insufficient generalization ability of existing technologies in practical applications.

[0023] It is applicable to system topology construction, fault tree analysis, and safety and reliability assessment of complex industrial systems. Attached Figure Description

[0024] Figure 1 A system framework diagram illustrating a system information intelligent acquisition method based on the collaboration of a multimodal large model and a large language model; Figure 2 A system flowchart representing the CSS system; Figure 3 This represents the multimodal model extraction result of the CSS system; Figure 4 This indicates partial device malfunction information for the CSS system. Detailed Implementation

[0025] To make the advantages and benefits of the technical solution provided by the present invention clearer, the technical solution provided by the present invention will now be described in further detail with reference to the accompanying drawings, specifically: Implementation Method 1: This implementation method provides a system information intelligent acquisition method based on the collaboration of a multimodal large model and a large language model, including: The steps include: acquiring and preprocessing the system flowchart, decomposing the system flowchart into functional decoupling and physical boundaries, and generating high-quality input data through Base64 encoding and resolution adaptive technology; The steps involve using a multimodal large model to perform symbol recognition and text extraction on the input data to obtain preliminary topology information including device number, device name, parameters, and connection order; The steps include extracting the preliminary topology information using regular expressions and formatting it into JSON, removing redundant or conflicting information, and outputting a standardized system information file. The steps are as follows: inputting the standardized system information file into the large language model, performing vectorized retrieval and semantic matching on the system information knowledge base, and outputting a set of fault information containing fault modes and source information. The step of integrating the standardized system information file with the fault information set to form a system information document that includes equipment, connection and fault information; The steps involve verifying the uniqueness, completeness, and logical consistency of the system information data, and outputting standardized system information files that can be directly used for fault tree analysis and reliability assessment.

[0026] In the process of acquiring and preprocessing system flowcharts, modular processing of flowcharts is achieved through dual decomposition of functional decoupling and physical boundaries, which reduces symbol ambiguity and improves recognition accuracy.

[0027] In the steps of symbol recognition and text extraction using a multimodal large model, a dual-track system of role and process prompts is adopted to constrain the output logic, so as to ensure the consistency and standardization of device number, device name and connection relationship.

[0028] During the process of extracting preliminary topology information using regular expressions and formatting it into JSON, uniqueness checks, integrity checks, and logical consistency checks are performed to ensure the reliability of the standardized system information file.

[0029] In the process of vectorized retrieval and semantic matching of the system information knowledge base, a deep neural network embedding model is used to convert the text into high-dimensional semantic vectors, and semantic matching is completed through a similarity algorithm.

[0030] In the process of integrating standardized system information files and fault information sets, a unified representation of "device-connection-fault" is formed by establishing a mapping relationship between equipment and fault modes.

[0031] Implementation Method Two: This implementation method provides a further detailed description of the technical solution provided in Implementation Method One. Specifically: By leveraging the collaborative operation of multimodal large models and large language models, we can achieve unified parsing and fault knowledge extraction of flowcharts and textual data from complex industrial systems, thereby forming standardized system information files that support fault tree analysis and reliability assessment.

[0032] The method of the present invention includes the following steps: Step one involves acquiring and preprocessing the system flowchart, which is then input into a multimodal large-scale model for symbol and text recognition. Specifically, the flowchart undergoes both functional decoupling and physical boundary decomposition, dividing the complex flowchart into independent functional units with clear boundaries, thereby reducing recognition ambiguity. Next, Base64 encoding is used to encode the image, and resolution adaptive techniques are employed to optimize image clarity, ensuring compatibility with the input requirements of the multimodal model. This preprocessing generates high-quality input data, which serves as the input for subsequent symbol recognition and text extraction.

[0033] Step two involves using a multimodal large model to identify the preprocessed system flowchart, extracting device symbols, connection relationships, and relevant text labels. Through joint understanding of image and text information, the multimodal large model can accurately extract the corresponding labeled text while identifying device symbols and resolving the connection relationships between devices. The preliminary results include various information such as device numbers, device names, device parameters, and device connection sequences. These identification results serve as input for the next step of cleaning and standardization.

[0034] Step three involves cleaning and standardizing the initially extracted topology information. First, regular expressions are used to filter the identified text content, removing redundant, ambiguous, or duplicate information, retaining data with high uniqueness and logical consistency. Then, JSON formatting is used to structure the topology information, ensuring that the correspondence between device information, connection relationships, and text content is clear and traceable. The cleaned and standardized system information file serves as input for the next step of semantic retrieval.

[0035] Step four involves performing multi-dimensional semantic retrieval of the system information knowledge base using a large language model to obtain equipment failure modes. The system information knowledge base consists of professional documents such as system function analysis reports, historical failure records, and maintenance manuals. First, the text in the knowledge base is transformed into high-dimensional semantic vectors using a deep neural network embedding model and stored in a vector database. Then, standardized topological information is input into the large language model as query conditions. The large language model calculates vector similarity and matches text fragments of failure modes related to the equipment from the database. These failure modes include possible causes of equipment failure, their manifestations, and maintenance measures, and their corresponding original document sources are marked using the "reference tracing" function. The output of this process is a structured set of failure information.

[0036] Step five involves integrating the topology information file and the fault information set to form complete system information data. During the integration process, a dual-track prompt paradigm based on roles and processes is used to constrain the output logic of the large model, ensuring consistency in device numbers, names, and connection relationships, while also guaranteeing accurate matching between fault modes and corresponding devices. The final system information data includes detailed device attributes, connection relationships between devices, and possible fault modes, achieving a unified representation of "device-connection-fault." This integrated information data can be directly used as input data for fault tree construction, system reliability analysis, and security assessment.

[0037] Step six involves verifying the consistency and integrity of the integrated system information. A two-level cleaning mechanism using regular expressions and JSON formatting is employed to verify the uniqueness, logical consistency, and traceability of the output, ensuring that the final file meets the high-precision requirements of engineering applications. The verified system information file, as the final output, can be widely used in complex industrial systems such as nuclear energy and chemical engineering, providing support for intelligent operation and maintenance.

[0038] In summary, this invention forms an efficient, reliable, and scalable intelligent framework for system information acquisition through the collaborative processing of multimodal large models and large language models, thereby solving the problems of low efficiency, insufficient accuracy, and lack of cross-modal collaborative capabilities in existing technologies.

[0039] Combination Figure 1-4 This embodiment describes the technical solution provided above in further detail through specific examples. Specifically: A system information intelligent acquisition method based on the collaboration of multimodal large model and large language model, such as Figure 1 As shown, it includes the following steps: Step 1: Use a multimodal large model to perform symbol and text recognition on the system flowchart, extract preliminary results of text information, and clean them; The system information includes the device number, name, parameters, and connection sequence.

[0040] The cleaning process involves performing regular expression extraction and JSON formatting on the initially extracted text results to remove redundant or conflicting information and output a standardized system information file.

[0041] Step 2: Use a large language model to perform multi-dimensional semantic retrieval on the system information knowledge base, match equipment failure modes, and output a set of traceable failure information; The system information knowledge base integrates professional textual materials such as system equipment function analysis reports, historical failure records, and maintenance manuals.

[0042] The retrieval process involves locating and extracting text fragments of fault modes that semantically match the query vector from the vector database of the system information knowledge base.

[0043] The vector database is a vector database that uses a deep neural network embedding model to convert the text of the system information knowledge base into high-dimensional semantic vectors, which are then stored and optimized.

[0044] The semantic matching involves calculating the similarity between vectors using a similarity algorithm to measure the degree of semantic association.

[0045] The equipment fault information includes the equipment fault mode, as well as specific text fragments and source documents referenced by the "Reference and Source Tracing" function.

[0046] Step 3: Integrate the extracted information into text data.

[0047] The final text data includes detailed information about the devices within the system, the connection relationships between the devices, and the possible failure modes of the devices.

[0048] To further understand the present invention, the following detailed description of the system information intelligent acquisition method based on the collaboration of multimodal large model and large language model provided by the present invention is provided in conjunction with embodiments. The scope of protection of the present invention is not limited by the following embodiments.

[0049] Example 1 Taking the application of the containment sprinkler system (CSS) as an example, the intelligent acquisition method of system information based on the collaboration of multimodal large model and large language model includes the following steps: Step 1: Use a multimodal large model to analyze, for example... Figure 2 The system flowchart of the CSS system is used for symbol and text recognition, preliminary results of text information extraction are obtained, and then the text is cleaned. The results are as follows: Figure 3 As shown; Step 2: Utilize a large language model to perform multi-dimensional semantic retrieval on the CSS system's system information knowledge base, match equipment fault modes, and output a set of traceable fault information. Some results are shown below. Figure 4 As shown; Step 3: Integrate the extracted information into text data.

[0050] Compared with existing technologies, the intelligent system information acquisition method based on the collaboration of multimodal large models and large language models of the present invention has the following beneficial effects: (1) This invention proposes a multimodal collaborative parsing mechanism, which accurately identifies equipment symbols, connection relationships and text labels in the system flowchart through a multimodal large model, generates structured topology data, and at the same time uses a large language model to deeply mine the fault modes in the text knowledge base to achieve semantic alignment and complementarity of graphic and text data; (2) The preprocessing system flowchart of the dual decomposition standard of functional decoupling and physical boundary is adopted, and Base64 encoding and resolution adaptive technology are combined to ensure the high quality and compatibility of multimodal large model input and improve the analytical accuracy of complex system topology; (3) Design a role-process dual-track prompt word paradigm to constrain the output logic of the multimodal large model, generate standardized equipment numbers, names and connection relationships, and optimize the retrieval efficiency and accuracy of fault knowledge of the large language model through vector database and semantic matching technology. (4) A two-level cleaning mechanism based on regular expressions and JSON formatting is proposed to perform uniqueness, completeness and logical consistency checks on the initially extracted topology information and fault knowledge, and output a standardized system information file integrating "device-connection-fault"; (5) This invention provides a set of scalable intelligent information acquisition frameworks that are applicable to complex industrial systems in fields such as nuclear energy and chemical industry, and provide high-precision, traceable multimodal data support for subsequent fault tree construction, reliability analysis and safety assessment.

[0051] The above description of several specific embodiments further details the technical solution provided by the present invention in order to highlight the advantages and benefits of the technical solution provided by the present invention. However, the above-described specific embodiments are not intended to limit the present invention. Any reasonable modifications and improvements to the present invention, combinations of embodiments, and equivalent substitutions based on the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A system information intelligent acquisition method based on the collaboration of a multimodal large model and a large language model, characterized in that, include: The steps include: acquiring and preprocessing the system flowchart, decomposing the system flowchart into functional decoupling and physical boundaries, and generating high-quality input data through Base64 encoding and resolution adaptive technology; The steps involve using a multimodal large model to perform symbol recognition and text extraction on the input data to obtain preliminary topology information including device number, device name, parameters, and connection order; The steps include extracting the preliminary topology information using regular expressions and formatting it into JSON, removing redundant or conflicting information, and outputting a standardized system information file. The steps are as follows: inputting the standardized system information file into the large language model, performing vectorized retrieval and semantic matching on the system information knowledge base, and outputting a set of fault information containing fault modes and source information. The step of integrating the standardized system information file with the fault information set to form a system information document that includes equipment, connection and fault information; The steps involve verifying the uniqueness, completeness, and logical consistency of the system information data, and outputting standardized system information files that can be directly used for fault tree analysis and reliability assessment.

2. The intelligent system information acquisition method based on the collaboration of a multimodal large model and a large language model according to claim 1, characterized in that, In the process of acquiring and preprocessing system flowcharts, modular processing of flowcharts is achieved through dual decomposition of functional decoupling and physical boundaries, which reduces symbol ambiguity and improves recognition accuracy.

3. The intelligent system information acquisition method based on the collaboration of a multimodal large model and a large language model according to claim 1, characterized in that, In the steps of symbol recognition and text extraction using a multimodal large model, a role-and-process dual-track prompt word paradigm is adopted to constrain the output logic, so as to ensure the consistency and standardization of device number, device name and connection relationship.

4. The intelligent system information acquisition method based on the collaboration of a multimodal large model and a large language model according to claim 1, characterized in that, During the process of extracting preliminary topology information using regular expressions and formatting it into JSON, uniqueness checks, integrity checks, and logical consistency checks are performed to ensure the reliability of the standardized system information file.

5. The method for intelligent acquisition of system information based on the collaboration of a multimodal large model and a large language model according to claim 1, characterized in that, In the process of vectorized retrieval and semantic matching of the system information knowledge base, a deep neural network embedding model is used to convert the text into high-dimensional semantic vectors, and semantic matching is completed through a similarity algorithm.

6. The method for intelligent acquisition of system information based on the collaboration of a multimodal large model and a large language model according to claim 1, characterized in that, In the process of integrating standardized system information files and fault information sets, a unified representation of "device-connection-fault" is formed by establishing a mapping relationship between equipment and fault modes.

7. A system information intelligent acquisition device based on the collaboration of a multimodal large model and a large language model, characterized in that, include: The system flowchart is acquired and preprocessed, and the system flowchart is decomposed into functional decoupling and physical boundary, and a module for generating high-quality input data is generated through Base64 encoding and resolution adaptive technology. The input data is subjected to symbol recognition and text extraction using a multimodal large model to obtain a module containing preliminary topology information including device number, device name, parameters and connection order; A module that performs regular expression extraction and JSON formatting on the preliminary topology information, removes redundant or conflicting information, and outputs a standardized system information file. The standardized system information file is input into a large language model, and vectorized retrieval and semantic matching are performed on the system information knowledge base to output a module containing a fault information set including fault modes and source information. The standardized system information files and the fault information set are integrated to form a module containing system information data that integrates equipment, connections, and faults. A module that performs uniqueness, completeness, and logical consistency checks on the system information data and outputs standardized system information files that can be directly used for fault tree analysis and reliability assessment.

8. A computer storage medium for storing computer programs, characterized in that, When the computer program is read by the computer, the computer executes the method of claim 1.

9. A computer, comprising a processor and a storage medium, characterized in that, When the processor reads the computer program stored in the storage medium, the computer executes the method of claim 1.

10. A computer program product, as a computer program, is characterized by: When the computer program is executed, it implements the method of claim 1.

Images