Multi-modal analysis and diagnosis model based on combination of multi-source data fusion and large model

Abstract

The embodiment of the invention discloses a multi-modal analysis and diagnosis model based on combination of multi-source data fusion and a large model, and belongs to the technical field of large models, and the model comprises a scene model module which is used for calling different function modules according to different tasks and scenes; the industry large model module is used for establishing different industry large models according to different industry requirements; the task execution module is used for calling different task models and function applications according to different task requirements; the matching tool module is used for providing tools or engines according to task requirements; the data knowledge module is used for establishing various databases and storing different data in the corresponding databases in a classified mode, diversified and fragmented AI application requirements can be rapidly met, the development cost and time can be reduced, the development efficiency can be improved, information and value in the data can be fully mined, and the development efficiency can be improved. And the influence caused by insufficiency or inaccuracy of the annotated data can be avoided.

Description

Technical Field

[0001] The present application belongs to the technical field of large models, and specifically, relates to a multimodal analysis and diagnosis model based on multi-source data fusion combined with a large model. Background Art

[0002] AI models refer to models designed and trained for specific tasks. They usually only perform well on that task and have poor effects on other tasks. General AI models usually include the following steps: data preprocessing and feature extraction: convert the preprocessed text data into numerical vectors to facilitate model calculations; model construction: select appropriate machine learning or deep learning algorithms to build classifiers, such as naive Bayes, support vector machines, decision trees, random forests, logistic regression, multi-layer perceptrons, convolutional neural networks, recurrent neural networks, etc.; model training: use labeled training sets to train classifiers, and adjust model parameters by optimizing loss functions so that they can correctly predict text categories; model evaluation: use unlabeled test sets to evaluate the performance of the classifier, and measure the model effect by calculating indicators such as accuracy, recall, and F1 value; model application: apply the trained classifier to actual scenarios, classify the newly input text, and give corresponding output or feedback.

[0003] However, existing discriminative artificial intelligence requires a large amount of labeled data for training. Obtaining fault data in the field of fault diagnosis is a time-consuming, expensive, and inefficient task that often requires professional personnel and tools to complete. Moreover, the quality and quantity of labeled data directly affect the performance and generalization ability of the diagnostic model. If the labeled data is insufficient or inaccurate, the model will have problems of underfitting or overfitting. At the same time, they are designed and trained for specific equipment. They usually can only perform well on this equipment and have poor effects on other equipment. This means that if you want to deal with different equipment and working conditions, you need to redesign and train new models, which will increase development costs and time, and will also cause model redundancy and waste. Moreover, it is based on feature extraction and model building methods of time series data such as vibration, temperature, and process parameters, but ignores unstructured data such as equipment maintenance records and years of experience accumulated by front-line equipment maintenance engineers. This leads to the general AI diagnostic model application data mode being single and lacking in universality and interpretability. Summary of the invention

[0004] In order to solve the above problems and technical defects, the embodiment of the present application adopts the following technical solution, which is based on a multi-modal analysis and diagnosis model combining multi-source data fusion and a large model, including:

[0005] The scenario model module is used to call different functional modules according to different tasks and scenarios;

[0006] Industry big model module, used to establish different industry big models according to different industry requirements;

[0007] The task execution module is used to call different task models and functional applications according to different task requirements;

[0008] Supporting tool module, used to provide tools or engines according to task requirements;

[0009] The data knowledge module is used to establish multiple databases and store different data categories in corresponding databases.

[0010] Preferably, the functional modules in the scene model module include:

[0011] Diagnostic assistant module, which is used to conduct intelligent question-answering based on joint analysis of multi-source data, and automatically generate maintenance work orders, suggestions and fault reports;

[0012] The knowledge graph module is used to mine the interconnections among a large amount of fragmented data, output information in multiple directions, generate natural human-computer interaction, and reduce the demand for business upgrades and maintenance;

[0013] Intelligent control module, which is used for the generative intelligent control platform built on the big model and digital employee platform, and is connected with the information system, database, and unstructured knowledge base through calling and configuration;

[0014] Multimodal decision module, which is used to improve the performance and generalization ability of the model by extracting features, using the corpus information of the device and learning the associations between different modalities to enhance the semantic understanding and expression ability of the model;

[0015] The intelligent operation and maintenance module is used to communicate about equipment predictive maintenance in natural language, provide timely answers to customers’ questions about equipment status monitoring, fault types and service life, and create an unattended customer service model.

[0016] Furthermore, the intelligent control module uses a large model to retrieve information, evolving the static display into one that relies on databases and manual operations, integrating data sources, and optimizing the decision-making process based on operating conditions, including: digital interaction module, global query module, and associated question and answer module.

[0017] Furthermore, the digital interaction module allows the user to ask questions to the virtual person to obtain corresponding information and present the results digitally;

[0018] The global query module directly provides question-and-answer results by connecting to multiple data sources, without switching systems, and exploring data in real time;

[0019] The related question and answer module generates relevant answers from different angles and levels by using a large model to perform learning and reasoning based on the question content provided by the user.

[0020] The multimodal decision module extracts rich and comprehensive features from different media data information and uses the Transformer autoencoder training mode for data training.

[0021] Furthermore, the intelligent operation and maintenance module is based on equipment operation and maintenance experience, industry data, mechanism diagnosis basis, on-site and historically collected data and data provided by manufacturers, and uses a large model to generate an AI digital human, including: a data capture module, a data interaction module, a feature extraction module and a decision question and answer module.

[0022] Furthermore, the data capture module is used to capture and analyze the required target data from the massive underlying data, and to create corresponding answers through interactive AI to achieve asset equipment reliability management;

[0023] The data interaction module is used to use digital people as the interaction entrance, access the business system and AIC big model in the background, and realize open business questions and answers;

[0024] Feature extraction module, used to extract equipment failure features, maintenance suggestions, and dynamically create the causes of failures from the knowledge base;

[0025] The decision-making question and answer module is used to extract content generated by AI through the support of natural language processing text, and use generative AI to call the results to form interactive questions and answers, helping organizations optimize operations, improve decision-making, and promote digital transformation.

[0026] Preferably, the task models in the task execution module include: a time series large model, a natural language large model and a multimodal large model.

[0027] Furthermore, the functional applications in the task execution module include: vibration and voiceprint signal understanding and perception function, complex working condition element identification function, diagnosis expert knowledge inheritance function, diagnosis conclusion generation and creation function, complex table structure processing function, tool calling function and multimodal understanding function.

[0028] Preferably, the supporting tool module includes a knowledge engine, a prompt word tool and a professional capability function library.

[0029] Preferably, the data knowledge module includes: an equipment acquisition database, an equipment knowledge database, an equipment maintenance database, an industry domain knowledge base and an asset management base.

[0030] Compared with the prior art, the beneficial effects of the embodiments of the present application are:

[0031] (1) This application is first pre-trained on a large-scale general fault data set, and then fine-tuned according to different equipment and working conditions, without the need to redesign and train a new model. The AI ​​large model can quickly respond to diverse and fragmented AI application needs, and can also reduce development costs and time and improve development efficiency;

[0032] (2) This application performs unsupervised or self-supervised learning on unlabeled data sets, allowing the model to learn general fault diagnosis knowledge and capabilities. It uses massive amounts of unlabeled fault data for pre-training without relying on a small amount of labeled data, which can fully mine the information and value in the data and avoid the impact of insufficient or inaccurate labeled data.

[0033] (3) This application learns knowledge and capabilities such as general equipment operation logs of equipment operation during the pre-training stage, and adjusts and optimizes according to specific tasks and scenarios during the fine-tuning stage. The AI ​​big model can capture the global information and deep semantics in the text data, and can also understand the complex logic and knowledge in the text data, making the AI ​​big model highly versatile and flexible, and improving the application effect of intelligent operation and maintenance in various fields and scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In the attached picture:

[0035] Figure 1 This is a schematic diagram of the structure of an embodiment of the present application;

[0036] Figure 2 It is a schematic diagram of the framework of an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0038] Example 1

[0039] like Figure 1 and Figure 2 As shown in the figure, the multimodal analysis and diagnosis model based on multi-source data fusion combined with a large model includes:

[0040] The scenario model module is used to call different functional modules according to different tasks and scenarios;

[0041] The functional modules in the scene model module include:

[0042] Diagnostic assistant module, which is used to conduct intelligent question-answering based on joint analysis of multi-source data, and automatically generate maintenance work orders, suggestions and fault reports;

[0043] The knowledge graph module is used to mine the interconnections among a large amount of fragmented data, output information in multiple directions, generate natural human-computer interaction, and reduce the demand for business upgrades and maintenance;

[0044] Intelligent control module, which is used for the generative intelligent control platform built on the big model and digital employee platform, and is connected with the information system, database, and unstructured knowledge base through calling and configuration;

[0045] The intelligent control module uses a large model to retrieve information, evolving the static display into one that relies on databases and manual operations, integrating data sources, and optimizing the decision-making process based on operating conditions. It includes: a digital interaction module, a global query module, and an associated question-and-answer module.

[0046] The digital interaction module allows users to ask questions to a virtual person to obtain relevant information and present the results digitally;

[0047] The global query module directly provides question-and-answer results by connecting to multiple data sources, without switching systems, and exploring data in real time;

[0048] The associated question-answering module uses a large model to learn and reason based on the question content provided by the user, and generates relevant answers from different angles and levels.

[0049] Multimodal decision module, which is used to improve the performance and generalization ability of the model by extracting features, using the corpus information of the device and learning the associations between different modalities to enhance the semantic understanding and expression ability of the model;

[0050] The multimodal decision module extracts rich and comprehensive features from different media data information and uses the Transformer autoencoder training mode for data training.

[0051] The intelligent operation and maintenance module is used to communicate about equipment predictive maintenance in natural language, provide timely answers to customers’ questions about equipment status monitoring, fault types and service life, and create an unattended customer service model.

[0052] The intelligent operation and maintenance module is based on equipment operation and maintenance experience, industry data, mechanism diagnosis basis, on-site and historically collected data, and data provided by manufacturers. It uses a large model to generate AI digital humans, including: data capture module, data interaction module, feature extraction module, and decision question and answer module.

[0053] The data capture module is used to capture and analyze the required target data from massive underlying data, and to create corresponding answers through interactive AI to achieve asset equipment reliability management;

[0054] The data interaction module is used to use digital people as the interaction entrance, access the business system and AIC big model in the background, and realize open business questions and answers;

[0055] Feature extraction module, used to extract equipment failure features, maintenance suggestions, and dynamically create the causes of failures from the knowledge base;

[0056] The decision-making question and answer module is used to extract content generated by AI through the support of natural language processing text, and use generative AI to call the results to form interactive questions and answers, helping organizations optimize operations, improve decision-making, and promote digital transformation.

[0057] Industry big model module, used to establish different industry big models according to different industry requirements;

[0058] The task execution module is used to call different task models and functional applications according to different task requirements;

[0059] Task models include: large time series models, large natural language models, and large multimodal models;

[0060] Functional applications include: vibration and voiceprint signal understanding and perception function, complex working condition element identification function, diagnostic expert knowledge inheritance function, diagnostic conclusion generation and creation function, complex table structure processing function, tool calling function and multimodal understanding function.

[0061] Supporting tool module, used to provide tools or engines according to task requirements;

[0062] The supporting tools include knowledge engine, prompt word tool and professional ability function library.

[0063] The data knowledge module is used to establish multiple databases and store different data categories in corresponding databases.

[0064] The data knowledge module includes: equipment acquisition database, equipment knowledge database, equipment maintenance database, industry knowledge base and asset management database

[0065] The above-mentioned embodiments only express the preferred implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for ordinary technicians in this field, several modifications, improvements and substitutions can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A multimodal analysis and diagnosis model based on multi-source data fusion combined with a large model, characterized by: include: The scenario model module is used to call different functional modules according to different tasks and scenarios; Industry big model module, used to establish different industry big models according to different industry requirements; The task execution module is used to call different task models and functional applications according to different task requirements; Supporting tool module, used to provide tools or engines according to task requirements; The data knowledge module is used to establish multiple databases and store different data categories in corresponding databases.

2. The multimodal analysis and diagnosis model based on multi-source data fusion combined with a large model according to claim 1 is characterized in that: The functional modules in the scene model module include: Diagnostic assistant module, which is used to conduct intelligent question-answering based on joint analysis of multi-source data, and automatically generate maintenance work orders, suggestions and fault reports; The knowledge graph module is used to mine the interconnections among a large amount of fragmented data, output information in multiple directions, generate natural human-computer interaction, and reduce the demand for business upgrades and maintenance; Intelligent control module, which is used for the generative intelligent control platform built on the big model and digital employee platform, and is connected with the information system, database, and unstructured knowledge base through calling and configuration; Multimodal decision module, which is used to improve the performance and generalization ability of the model by extracting features, using the corpus information of the device and learning the associations between different modalities to enhance the semantic understanding and expression ability of the model; The intelligent operation and maintenance module is used to communicate about equipment predictive maintenance in natural language, provide timely answers to customers’ questions about equipment status monitoring, fault types and service life, and create an unattended customer service model.

3. The multimodal analysis and diagnosis model based on multi-source data fusion combined with a large model according to claim 2 is characterized in that: The intelligent control module uses a large model to retrieve information, evolving the static display into one that relies on databases and manual operations, integrating data sources, and optimizing the decision-making process based on operating conditions. It includes: a digital interaction module, a global query module, and an associated question-and-answer module.

4. The multimodal analysis and diagnosis model based on multi-source data fusion combined with a large model according to claim 3 is characterized in that: The digital interaction module allows the user to ask questions and answer questions with the virtual person to obtain corresponding information and present the results digitally; The global query module directly provides question-and-answer results by connecting to multiple data sources, without switching systems, and exploring data in real time; The related question and answer module generates relevant answers from different angles and levels by using a large model to perform learning and reasoning based on the question content provided by the user. The multimodal decision module extracts rich and comprehensive features from different media data information and uses the Transformer autoencoder training mode for data training.

5. The multimodal analysis and diagnosis model based on multi-source data fusion combined with a large model according to claim 2 is characterized in that: The intelligent operation and maintenance module is based on equipment operation and maintenance experience, industry data, mechanism diagnosis basis, on-site and historically collected data, and data provided by manufacturers. It uses a large model to generate AI digital humans, including: data capture module, data interaction module, feature extraction module, and decision question and answer module.

6. The multimodal analysis and diagnosis model based on multi-source data fusion combined with a large model according to claim 5 is characterized in that: The data capture module is used to capture and analyze the required target data from the massive underlying data, and to create corresponding answers through interactive AI to achieve asset equipment reliability management; The data interaction module is used to use digital people as the interaction entrance, access the business system and the AIC big model in the background, and realize open business questions and answers; The feature extraction module is used to extract equipment failure features, maintenance suggestions, and dynamically create the causes of failures from the knowledge base; The decision-making question and answer module is used to extract content generated by AI through the support of natural language processing text, and use generative AI to call the results to form interactive questions and answers, helping organizations optimize operations, improve decision-making, and promote digital transformation.

7. The multimodal analysis and diagnosis model based on multi-source data fusion combined with a large model according to claim 1 is characterized in that: The task models in the task execution module include: a time series large model, a natural language large model and a multimodal large model.

8. The multimodal analysis and diagnosis model based on multi-source data fusion combined with a large model according to claim 7 is characterized in that: The functional applications in the task execution module include: vibration and voiceprint signal understanding and perception function, complex working condition element identification function, diagnosis expert knowledge inheritance function, diagnosis conclusion generation and creation function, complex table structure processing function, tool calling function and multimodal understanding function.

9. The multimodal analysis and diagnosis model based on multi-source data fusion combined with a large model according to claim 1 is characterized in that: The supporting tool module includes a knowledge engine, a prompt word tool and a professional ability function library.

10. The multimodal analysis and diagnosis model based on multi-source data fusion combined with a large model according to claim 1 is characterized in that: The data knowledge module includes: equipment acquisition database, equipment knowledge database, equipment maintenance database, industry domain knowledge base and asset management database.

Images