Equipment information management method and system based on multi-modal large model

Abstract

The invention provides an equipment information management method and system based on a multi-modal large model in the technical field of equipment information management. The method comprises the following steps: S1, acquiring historical multi-modal data to construct a data set, and dividing the data set into a training set, a verification set and a test set; s2, creating a multi-modal large model based on a Vi T module, a Bert module, a numerical feature construction module, a feature fusion module and an output module, and setting hyper-parameters; s3, training and evaluating the multi-modal large model through the training set, the verification set and the test set; step S4, compressing the multi-modal large model which passes the evaluation; and S5, collecting real-time multi-modal data of installed equipment in the management area by each area server, and inputting the real-time multi-modal data into the multi-modal large model to obtain an equipment information analysis report. The method has the advantages that the efficiency, the quality, the traceability and the resource utilization rate of equipment information management are greatly improved, and then the reliability of equipment operation and maintenance is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of device information management, and in particular to a device information management method and system based on a multi-modal large model. Background Art

[0002] For some large companies in specific industries, some equipment may be added every month. These equipment belong to the company's assets, so the equipment information of these equipment needs to be managed. For example, the installed equipment is checked every month, the number of installed equipment is counted to see whether it is consistent with the planned number of installed equipment, whether the actual installation date of the equipment is consistent with the planned installation date of the equipment, the equipment operation status data is analyzed, and the equipment is predicted and warned of failures, so as to carry out equipment maintenance and replacement in advance to avoid losses in actual production due to equipment failures.

[0003] The management of equipment information traditionally relies on manual collection, recording and analysis of on-site equipment information, which has the following disadvantages: 1. It takes a lot of time and human resources; 2. It is difficult to combine multiple modal data to detect the equipment status, and it is difficult to fully reflect the real-time status of the equipment, and thus it is difficult to predict potential failures; 3. Manual recording is prone to errors, resulting in inaccurate recorded equipment information, and it is not easy to preserve and analyze for a long time; 4. Manual analysis may have subjective misjudgments or omissions, resulting in small problems turning into major failures, and it is difficult to conduct in-depth analysis, and it is impossible to effectively evaluate and predict the equipment operation status; 5. The timeliness is poor, and there is a delay between discovering the abnormality and taking action, which affects the timeliness of equipment maintenance; 6. It needs to rely on professional technicians and train relevant personnel, otherwise it is difficult to output effective analysis results, which makes the cost high, that is, low resource utilization.

[0004] Therefore, how to provide a device information management method and system based on a multimodal large model to improve the efficiency, quality, traceability, and resource utilization of device information management, and thereby improve the reliability of equipment operation and maintenance, has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a device information management method and system based on a multimodal large model, so as to improve the efficiency, quality, traceability and resource utilization of device information management, thereby improving the reliability of equipment operation and maintenance.

[0006] In a first aspect, the present invention provides a device information management method based on a multimodal large model, comprising the following steps:

[0007] Step S1, obtaining historical multimodal data of installed equipment, preprocessing each of the historical multimodal data and constructing a data set, and dividing the data set into a training set, a validation set, and a test set;

[0008] Step S2, creating a multimodal large model for equipment information statistical analysis and fault prediction based on the ViT module, the Bert module, the numerical feature construction module, the feature fusion module and the output module, and setting the hyperparameters of the multimodal large model;

[0009] Step S3: Based on the LoRA technology, the multimodal large model is trained and evaluated through the training set, the validation set and the test set;

[0010] Step S4, compressing the multimodal large model that has passed the evaluation by using knowledge distillation technology;

[0011] Step S5: deploying the compressed multimodal large model to each regional server, each regional server respectively collects real-time multimodal data of the installed equipment in the management area, and inputs the real-time multimodal data into the multimodal large model to obtain a device information analysis report;

[0012] Step S6: Each regional server aggregates the device information analysis report to the main server, and the main server displays and stores each device information analysis report, and continuously optimizes the multimodal large model based on each device information analysis report.

[0013] Furthermore, the step S1 is specifically as follows:

[0014] Using a time synchronization mechanism to obtain historical multimodal data of installed equipment, the historical multimodal data at least includes equipment operation logs, operation status images, operation status videos, operation temperatures, and operation voltages;

[0015] Performing data cleaning operations on abnormal values, missing values, duplicate data, low-frequency words, stop words, punctuation marks, and special characters in each of the historical multimodal data; the duplicate data is duplicate text, duplicate image, duplicate video, or duplicate numerical value;

[0016] The numerical historical multimodal data are normalized by using the MinMax normalization function, the categorical historical multimodal data are encoded by using the OneHot encoding, and the device text information including at least the production date, device model, device type, and rated voltage in the operating status image is identified by using the OCR technology, thereby completing the preprocessing of each of the historical multimodal data;

[0017] After the preprocessed historical multimodal data are annotated with fault assessment and fault handling suggestions, a data set is constructed, and the data set is divided into a training set, a validation set, and a test set based on a ratio of 7:2:1;

[0018] A vector database is created, and the device text information, fault assessment, and fault handling suggestions are stored in the vector database.

[0019] Further, in step S2, the Vi T module is used to extract image features in the running state image and the running state video;

[0020] The Bert module is used to extract text features in the device operation log;

[0021] The numerical feature construction module is used to construct a numerical feature set through operating temperature and operating voltage;

[0022] The feature fusion module is used to normalize the scale of the image features, text features and numerical feature sets, perform weighted summation according to preset importance weights, and then fuse them into a feature representation vector;

[0023] The output module is used to output a device information analysis report carrying device information statistical analysis results and fault prediction results according to the feature characterization vector;

[0024] The hyperparameters include at least a random dropout rate, a weight decay rate, a learning rate, and a loss function.

[0025] Furthermore, the step S3 is specifically as follows:

[0026] Based on the LoRA technology, the multimodal large model is trained through the training set, the hyperparameters are adjusted through the validation set to obtain the optimal hyperparameter combination, the multimodal large model is continued to be trained using the hyperparameter combination and the training set, and the multimodal large model is evaluated through the test set;

[0027] A relational database is created, and the data set and the model parameters of the multimodal large model after evaluation are stored in the relational database.

[0028] Furthermore, the step S5 is specifically as follows:

[0029] The compressed multimodal large model is deployed to each regional server, and each regional server collects real-time multimodal data of the installed equipment in the management area through a scanner, a camera, a temperature sensor, and a voltage sensor;

[0030] Matching a preset query template based on the device type carried by the real-time multimodal data, constructing a first query statement based on the query template and the real-time multimodal data, converting the first query statement into an embedded vector using an embedding model, constructing a vector memory index based on the embedded vector, and querying a fault assessment and a fault handling suggestion from a vector database based on the vector memory index through RAG technology;

[0031] A second query statement is constructed based on the real-time multimodal data, query fault assessment, fault handling suggestions and preset prompt templates, and the second query statement is input into the multimodal big model to obtain an equipment information analysis report carrying equipment information statistical analysis results and fault prediction results.

[0032] In a second aspect, the present invention provides a device information management system based on a multimodal large model, comprising the following modules:

[0033] A data set construction module, used to obtain historical multimodal data of installed equipment, preprocess each of the historical multimodal data and construct a data set, and divide the data set into a training set, a validation set, and a test set;

[0034] A multimodal large model creation module, used to create a multimodal large model for equipment information statistical analysis and fault prediction based on the Vi T module, the Bert module, the numerical feature construction module, the feature fusion module and the output module, and set the hyperparameters of the multimodal large model;

[0035] A multimodal large model training module, used for training and evaluating the multimodal large model based on the LoRA technology through the training set, the validation set and the test set;

[0036] A multimodal large model compression module, used to compress the multimodal large model that has passed the evaluation by using knowledge distillation technology;

[0037] The device information analysis module is used to deploy the compressed multimodal large model to each regional server, and each regional server collects real-time multimodal data of the installed devices in the management area, and inputs the real-time multimodal data into the multimodal large model to obtain a device information analysis report;

[0038] The device information management module is used for each regional server to aggregate the device information analysis report to the main server, and the main server displays and stores each device information analysis report, and continuously optimizes the multimodal large model based on each device information analysis report.

[0039] Furthermore, the data set construction module is specifically used for:

[0040] Using a time synchronization mechanism to obtain historical multimodal data of installed equipment, the historical multimodal data at least includes equipment operation logs, operation status images, operation status videos, operation temperatures, and operation voltages;

[0041] Performing data cleaning operations on abnormal values, missing values, duplicate data, low-frequency words, stop words, punctuation marks, and special characters in each of the historical multimodal data; the duplicate data is duplicate text, duplicate image, duplicate video, or duplicate numerical value;

[0042] The numerical historical multimodal data are normalized by using the MinMax normalization function, the categorical historical multimodal data are encoded by using the OneHot encoding, and the device text information including at least the production date, device model, device type, and rated voltage in the operating status image is identified by using the OCR technology, thereby completing the preprocessing of each of the historical multimodal data;

[0043] After the preprocessed historical multimodal data are annotated with fault assessment and fault handling suggestions, a data set is constructed, and the data set is divided into a training set, a validation set, and a test set based on a ratio of 7:2:1;

[0044] A vector database is created, and the device text information, fault assessment, and fault handling suggestions are stored in the vector database.

[0045] Further, in the multimodal large model creation module, the Vi T module is used to extract image features in the running state image and the running state video;

[0046] The Bert module is used to extract text features in the device operation log;

[0047] The numerical feature construction module is used to construct a numerical feature set through operating temperature and operating voltage;

[0048] The feature fusion module is used to normalize the scale of the image features, text features and numerical feature sets, perform weighted summation according to preset importance weights, and then fuse them into a feature representation vector;

[0049] The output module is used to output a device information analysis report carrying device information statistical analysis results and fault prediction results according to the feature characterization vector;

[0050] The hyperparameters include at least a random dropout rate, a weight decay rate, a learning rate, and a loss function.

[0051] Furthermore, the multimodal large model training module is specifically used for:

[0052] Based on the LoRA technology, the multimodal large model is trained through the training set, the hyperparameters are adjusted through the validation set to obtain the optimal hyperparameter combination, the multimodal large model is continued to be trained using the hyperparameter combination and the training set, and the multimodal large model is evaluated through the test set;

[0053] A relational database is created, and the data set and the model parameters of the multimodal large model after evaluation are stored in the relational database.

[0054] Furthermore, the device information analysis module is specifically used to:

[0055] The compressed multimodal large model is deployed to each regional server, and each regional server collects real-time multimodal data of the installed equipment in the management area through a scanner, a camera, a temperature sensor, and a voltage sensor;

[0056] Matching a preset query template based on the device type carried by the real-time multimodal data, constructing a first query statement based on the query template and the real-time multimodal data, converting the first query statement into an embedded vector using an embedding model, constructing a vector memory index based on the embedded vector, and querying a fault assessment and a fault handling suggestion from a vector database based on the vector memory index through RAG technology;

[0057] A second query statement is constructed based on the real-time multimodal data, query fault assessment, fault handling suggestions and preset prompt templates, and the second query statement is input into the multimodal big model to obtain an equipment information analysis report carrying equipment information statistical analysis results and fault prediction results.

[0058] The advantages of the present invention are:

[0059] 1. Preprocess and construct a data set by obtaining the historical multimodal data of the installed equipment, and divide the data set into a training set, a validation set, and a test set; then create a multimodal large model based on the Vi T module, the Bert module, the numerical feature construction module, the feature fusion module, and the output module, and set the hyperparameters of the multimodal large model; then, based on the LoRA technology, train and evaluate the multimodal large model through the training set, the validation set, and the test set, compress the multimodal large model that passes the evaluation through the knowledge distillation technology, and deploy the compressed multimodal large model to each regional server. Each regional server collects the real-time multimodal data of the installed equipment in the management area and inputs it into the multimodal large model to obtain the equipment information analysis report, which is then aggregated to the main server. The main server displays and stores the equipment information analysis reports based on each The equipment information analysis report continuously optimizes the multimodal large model; that is, the equipment information is managed through the multimodal data collected in real time by scanners, cameras, temperature sensors and voltage sensors, which can fully reflect the real-time status of the equipment without consuming a lot of time and human resources, and without relying on professional technicians, avoiding errors caused by manual recording and analysis. The saved labor costs can be used to upgrade equipment, and the equipment information analysis report is stored for later traceability. Combined with the analysis of the multimodal large model, faults can be accurately predicted, which ultimately greatly improves the efficiency, quality, traceability and resource utilization of equipment information management, thereby greatly improving the reliability of equipment operation and maintenance.

[0060] 2. Through scanners, cameras, temperature sensors and voltage sensors, multimodal data including at least equipment operation logs, operation status images, operation status videos, operation temperatures and operation voltages are collected, which can comprehensively capture the operation status of the equipment and provide a rich data foundation for subsequent analysis.

[0061] 3. By deploying the multimodal large model to each regional server, each regional server collects the real-time multimodal data of the installed equipment in the management area, that is, the real-time multimodal data in each management area is processed in a distributed manner, effectively improving the processing speed.

[0062] 4. By creating and training a large multimodal model for equipment information statistical analysis and fault prediction, the efficiency of equipment information management is greatly improved, while saving manpower and time costs.

[0063] 5. Collect multimodal data through the time synchronization mechanism to ensure the consistency of multimodal data collection timing. Preprocess the multimodal data to effectively eliminate the noise impact of the multimodal data, thereby greatly improving the data fusion capability of the multimodal data.

[0064] 6. Through RAG technology, fault assessment and fault handling suggestions are queried from the vector database based on the vector memory index. Based on real-time multimodal data, fault assessment, fault handling suggestions and preset prompt templates, a second query statement is constructed to input the multimodal big model. That is, the multimodal big model is relied upon for statistical analysis of equipment information and fault prediction, which increases the richness of equipment information and improves the accuracy of fault prediction.

[0065] 7. The image features in the operating status images and operating status videos are extracted through the Vi T module, the text features in the equipment operation log are extracted through the Bert module, and the numerical feature construction module is used to construct a numerical feature set based on the operating temperature and operating voltage. The image features, text features and numerical feature sets are scaled and normalized through the feature fusion module, and then weighted summed according to the preset importance weights to fuse them into a feature representation vector, making the extracted multimodal data richer and improving the representation capability of the multimodal large model feature extraction, thereby greatly improving the accuracy of fault prediction.

[0066] 8. Training large multimodal models through LoRA technology reduces the computational complexity and memory usage during the training process, saves training time and training resources, and effectively improves the accuracy and portability of multimodal large model predictions. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0068] Figure 1 It is a flow chart of a device information management method based on a multimodal large model of the present invention.

[0069] Figure 2 It is a structural schematic diagram of a device information management system based on a multi-modal large model of the present invention. DETAILED DESCRIPTION

[0070] The technical solution in the embodiment of the present application has the following overall idea: by managing the device information through multimodal data collected in real time by scanners, cameras, temperature sensors and voltage sensors, the real-time status of the device can be fully reflected without consuming a lot of time and human resources, without relying on professional technicians, and avoiding errors caused by manual recording and analysis. The saved labor costs can be used to upgrade the equipment, and the device information analysis report is stored for later traceability. Combined with the analysis of the multimodal large model, faults can be accurately predicted to improve the efficiency, quality, traceability and resource utilization of device information management, thereby improving the reliability of equipment operation and maintenance.

[0071] Please refer to Figure 1 to Figure 2As shown, a preferred embodiment of the device information management method based on a multimodal large model of the present invention includes the following steps:

[0072] Step S1, obtaining historical multimodal data of installed equipment, preprocessing each of the historical multimodal data and constructing a data set, and dividing the data set into a training set, a validation set, and a test set;

[0073] Step S2, creating a multimodal large model for equipment information statistical analysis and fault prediction based on the ViT module, the Bert module, the numerical feature construction module, the feature fusion module and the output module, and setting the hyperparameters of the multimodal large model;

[0074] Step S3: Based on the LoRA technology, the multimodal large model is trained and evaluated through the training set, the validation set and the test set;

[0075] Step S4: compressing the multimodal large model that has passed the evaluation by using knowledge distillation technology to improve the reasoning speed of the multimodal large model, that is, migrating the knowledge of the multimodal large model into a lightweight model;

[0076] Step S5: deploying the compressed multimodal large model to each regional server, each regional server respectively collects real-time multimodal data of the installed equipment in the management area, and inputs the real-time multimodal data into the multimodal large model to obtain a device information analysis report;

[0077] Step S6: Each regional server aggregates the device information analysis report to the main server, and the main server displays and stores each device information analysis report, and continuously optimizes the multimodal large model based on each device information analysis report.

[0078] During specific implementation, the main server creates a responsive Web front-end, displays the device information analysis report with intuitive charts and indicators, provides a convenient user interaction experience, and performs fault alarms based on the device information analysis report; the main server can perform further in-depth analysis of each device information analysis report.

[0079] The step S1 is specifically as follows:

[0080] A time synchronization mechanism is used to obtain historical multimodal data of installed equipment, wherein the historical multimodal data includes at least equipment operation log, operation status image, operation status video, operation temperature and operation voltage; the time synchronization mechanism is used to ensure that the timing of data collection of different modalities is consistent;

[0081] Perform data cleaning operations on abnormal values, missing values, repeated data, low-frequency words, stop words, punctuation marks and special characters in each of the historical multimodal data; the repeated data is repeated text, repeated images, repeated videos or repeated values; the abnormal values ​​may be caused by sensor failure, recording errors, transmission errors, scanning errors, etc.; the missing values ​​can be filled or deleted as needed;

[0082] The numerical historical multimodal data are normalized by using the MinMax normalization function to eliminate the influence of scale and dimension, the categorical variable historical multimodal data are encoded by using OneHot encoding, and the device text information including at least the production date, device model, device type, and rated voltage in the operating status image is identified by using OCR technology, thereby completing the preprocessing of each of the historical multimodal data;

[0083] After the preprocessed historical multimodal data are annotated with fault assessment and fault handling suggestions, a data set is constructed, and the data set is divided into a training set, a validation set, and a test set based on a ratio of 7:2:1;

[0084] A vector database is created, and the device text information, fault assessment, and fault handling suggestions are stored in the vector database.

[0085] In the step S2, the Vi T module is used to extract image features from the running state image and the running state video;

[0086] The Bert module is used to extract text features in the device operation log;

[0087] The numerical feature construction module is used to construct a numerical feature set through operating temperature and operating voltage;

[0088] The feature fusion module is used to normalize the scale of the image features, text features and numerical feature sets, perform weighted summation according to preset importance weights, and then fuse them into a feature representation vector;

[0089] The output module is used to output a device information analysis report carrying device information statistical analysis results and fault prediction results according to the feature characterization vector;

[0090] The hyperparameters include at least a random dropout rate, a weight decay rate, a learning rate, and a loss function.

[0091] The step S3 is specifically as follows:

[0092] Based on the LoRA technology, the multimodal large model is trained through the training set, the hyperparameters are adjusted through the validation set to obtain the optimal hyperparameter combination, the multimodal large model is continued to be trained using the hyperparameter combination and the training set, and the multimodal large model is evaluated through the test set;

[0093] The formula for LoRA is:

[0094]

[0095] Among them, H represents the output of the multimodal large model; X represents the input of the multimodal large model; W0 represents the weight parameter matrix of the multimodal large model; α / r represents the scaling factor, the larger the value, the greater the influence of the LoRA weight; A and B are both matrices, and the input of A is the same as the output of B, and the output of A is the same as the input of B;

[0096] A relational database is created to store the data set and the model parameters of the multimodal large model after evaluation in the relational database for subsequent continuous optimization of the multimodal large model.

[0097] The step S5 is specifically as follows:

[0098] The compressed multimodal large model is deployed to each regional server. Each regional server collects real-time multimodal data of installed devices in the management area through scanners, cameras, temperature sensors and voltage sensors based on a time synchronization mechanism, and pre-processes each real-time multimodal data;

[0099] Matching a preset query template based on the device type carried by the real-time multimodal data, constructing a first query statement based on the query template and the real-time multimodal data, converting the first query statement into an embedded vector using an embedding model, constructing a vector memory index based on the embedded vector, and querying a fault assessment and a fault handling suggestion from a vector database based on the vector memory index using RAG technology;

[0100] A second query statement is constructed based on the real-time multimodal data, query fault assessment, fault handling suggestions, and a preset prompt template, and the second query statement is input into the multimodal macro model to obtain an equipment information analysis report carrying equipment information statistical analysis results and fault prediction results. The equipment information statistical analysis results are whether the equipment installation quantity and equipment installation date are consistent.

[0101] A preferred embodiment of the device information management system based on a multimodal large model of the present invention includes the following modules:

[0102] A data set construction module, used to obtain historical multimodal data of installed equipment, preprocess each of the historical multimodal data and construct a data set, and divide the data set into a training set, a validation set, and a test set;

[0103] A multimodal large model creation module, used to create a multimodal large model for equipment information statistical analysis and fault prediction based on the Vi T module, the Bert module, the numerical feature construction module, the feature fusion module and the output module, and set the hyperparameters of the multimodal large model;

[0104] A multimodal large model training module, used for training and evaluating the multimodal large model based on the LoRA technology through the training set, the validation set and the test set;

[0105] A multimodal large model compression module, used to compress the multimodal large model that has passed the evaluation through knowledge distillation technology to improve the reasoning speed of the multimodal large model, that is, to migrate the knowledge of the multimodal large model into a lightweight model;

[0106] The device information analysis module is used to deploy the compressed multimodal large model to each regional server, and each regional server collects real-time multimodal data of the installed devices in the management area, and inputs the real-time multimodal data into the multimodal large model to obtain a device information analysis report;

[0107] The device information management module is used for each regional server to aggregate the device information analysis report to the main server, and the main server displays and stores each device information analysis report, and continuously optimizes the multimodal large model based on each device information analysis report.

[0108] During specific implementation, the main server creates a responsive Web front-end, displays the device information analysis report with intuitive charts and indicators, provides a convenient user interaction experience, and performs fault alarms based on the device information analysis report; the main server can perform further in-depth analysis of each device information analysis report.

[0109] The data set construction module is specifically used for:

[0110] A time synchronization mechanism is used to obtain historical multimodal data of installed equipment, wherein the historical multimodal data includes at least equipment operation log, operation status image, operation status video, operation temperature and operation voltage; the time synchronization mechanism is used to ensure that the timing of data collection of different modalities is consistent;

[0111] Perform data cleaning operations on abnormal values, missing values, repeated data, low-frequency words, stop words, punctuation marks and special characters in each of the historical multimodal data; the repeated data is repeated text, repeated images, repeated videos or repeated values; the abnormal values ​​may be caused by sensor failure, recording errors, transmission errors, scanning errors, etc.; the missing values ​​can be filled or deleted as needed;

[0112] The numerical historical multimodal data are normalized by using the MinMax normalization function to eliminate the influence of scale and dimension, the categorical variable historical multimodal data are encoded by using OneHot encoding, and the device text information including at least the production date, device model, device type, and rated voltage in the operating status image is identified by using OCR technology, thereby completing the preprocessing of each of the historical multimodal data;

[0113] After the preprocessed historical multimodal data are annotated with fault assessment and fault handling suggestions, a data set is constructed, and the data set is divided into a training set, a validation set, and a test set based on a ratio of 7:2:1;

[0114] A vector database is created, and the device text information, fault assessment, and fault handling suggestions are stored in the vector database.

[0115] In the multimodal large model creation module, the Vi T module is used to extract image features in the running state image and the running state video;

[0116] The Bert module is used to extract text features in the device operation log;

[0117] The numerical feature construction module is used to construct a numerical feature set through operating temperature and operating voltage;

[0118] The feature fusion module is used to normalize the scale of the image features, text features and numerical feature sets, perform weighted summation according to preset importance weights, and then fuse them into a feature representation vector;

[0119] The output module is used to output a device information analysis report carrying device information statistical analysis results and fault prediction results according to the feature characterization vector;

[0120] The hyperparameters include at least a random dropout rate, a weight decay rate, a learning rate, and a loss function.

[0121] The multimodal large model training module is specifically used for:

[0122] Based on the LoRA technology, the multimodal large model is trained through the training set, the hyperparameters are adjusted through the validation set to obtain the optimal hyperparameter combination, the multimodal large model is continued to be trained using the hyperparameter combination and the training set, and the multimodal large model is evaluated through the test set;

[0123] The formula for LoRA is:

[0124]

[0125] Among them, H represents the output of the multimodal large model; X represents the input of the multimodal large model; W0 represents the weight parameter matrix of the multimodal large model; α / r represents the scaling factor, the larger the value, the greater the influence of the LoRA weight; A and B are both matrices, and the input of A is the same as the output of B, and the output of A is the same as the input of B;

[0126] A relational database is created to store the data set and the model parameters of the multimodal large model after evaluation in the relational database for subsequent continuous optimization of the multimodal large model.

[0127] The device information analysis module is specifically used for:

[0128] The compressed multimodal large model is deployed to each regional server. Each regional server collects real-time multimodal data of installed devices in the management area through scanners, cameras, temperature sensors and voltage sensors based on a time synchronization mechanism, and pre-processes each real-time multimodal data;

[0129] Matching a preset query template based on the device type carried by the real-time multimodal data, constructing a first query statement based on the query template and the real-time multimodal data, converting the first query statement into an embedded vector using an embedding model, constructing a vector memory index based on the embedded vector, and querying a fault assessment and a fault handling suggestion from a vector database based on the vector memory index using RAG technology;

[0130] A second query statement is constructed based on the real-time multimodal data, query fault assessment, fault handling suggestions, and a preset prompt template, and the second query statement is input into the multimodal macro model to obtain an equipment information analysis report carrying equipment information statistical analysis results and fault prediction results. The equipment information statistical analysis results are whether the equipment installation quantity and equipment installation date are consistent.

[0131] In summary, the advantages of the present invention are:

[0132] 1. Preprocess and construct a data set by obtaining the historical multimodal data of the installed equipment, and divide the data set into a training set, a validation set, and a test set; then create a multimodal large model based on the Vi T module, the Bert module, the numerical feature construction module, the feature fusion module, and the output module, and set the hyperparameters of the multimodal large model; then, based on the LoRA technology, train and evaluate the multimodal large model through the training set, the validation set, and the test set, compress the multimodal large model that passes the evaluation through the knowledge distillation technology, and deploy the compressed multimodal large model to each regional server. Each regional server collects the real-time multimodal data of the installed equipment in the management area and inputs it into the multimodal large model to obtain the equipment information analysis report, which is then aggregated to the main server. The main server displays and stores the equipment information analysis reports based on each The equipment information analysis report continuously optimizes the multimodal large model; that is, the equipment information is managed through the multimodal data collected in real time by scanners, cameras, temperature sensors and voltage sensors, which can fully reflect the real-time status of the equipment without consuming a lot of time and human resources, and without relying on professional technicians, avoiding errors caused by manual recording and analysis. The saved labor costs can be used to upgrade equipment, and the equipment information analysis report is stored for later traceability. Combined with the analysis of the multimodal large model, faults can be accurately predicted, which ultimately greatly improves the efficiency, quality, traceability and resource utilization of equipment information management, thereby greatly improving the reliability of equipment operation and maintenance.

[0133] 2. Through scanners, cameras, temperature sensors and voltage sensors, multimodal data including at least equipment operation logs, operation status images, operation status videos, operation temperatures and operation voltages are collected, which can comprehensively capture the operation status of the equipment and provide a rich data foundation for subsequent analysis.

[0134] 3. By deploying the multimodal large model to each regional server, each regional server collects the real-time multimodal data of the installed equipment in the management area, that is, the real-time multimodal data in each management area is processed in a distributed manner, effectively improving the processing speed.

[0135] 4. By creating and training a large multimodal model for equipment information statistical analysis and fault prediction, the efficiency of equipment information management is greatly improved, while saving manpower and time costs.

[0136] 5. Collect multimodal data through the time synchronization mechanism to ensure the consistency of multimodal data collection timing. Preprocess the multimodal data to effectively eliminate the noise impact of the multimodal data, thereby greatly improving the data fusion capability of the multimodal data.

[0137] 6. Through RAG technology, fault assessment and fault handling suggestions are queried from the vector database based on the vector memory index. Based on real-time multimodal data, fault assessment, fault handling suggestions and preset prompt templates, a second query statement is constructed to input the multimodal big model. That is, the multimodal big model is relied upon for statistical analysis of equipment information and fault prediction, which increases the richness of equipment information and improves the accuracy of fault prediction.

[0138] 7. The image features in the operating status images and operating status videos are extracted through the ViT module, the text features in the equipment operation log are extracted through the Bert module, and the numerical feature construction module is used to construct a numerical feature set based on the operating temperature and operating voltage. The image features, text features and numerical feature sets are scaled and normalized through the feature fusion module, and then weighted summed according to the preset importance weights to fuse them into a feature representation vector, making the extracted multimodal data richer and improving the representation capability of the multimodal large model feature extraction, thereby greatly improving the accuracy of fault prediction.

[0139] 8. Training large multimodal models through LoRA technology reduces the computational complexity and memory usage during the training process, saves training time and training resources, and effectively improves the accuracy and portability of multimodal large model predictions.

[0140] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A device information management method based on a multimodal large model, characterized in that: The steps include: Step S1, obtaining historical multimodal data of installed equipment, preprocessing each of the historical multimodal data and constructing a data set, and dividing the data set into a training set, a validation set, and a test set; Step S2, creating a multimodal large model for equipment information statistical analysis and fault prediction based on the ViT module, the Bert module, the numerical feature construction module, the feature fusion module and the output module, and setting the hyperparameters of the multimodal large model; Step S3: Based on the LoRA technology, the multimodal large model is trained and evaluated through the training set, the validation set and the test set; Step S4, compressing the multimodal large model that has passed the evaluation by using knowledge distillation technology; Step S5: deploying the compressed multimodal large model to each regional server, each regional server respectively collects real-time multimodal data of the installed equipment in the management area, and inputs the real-time multimodal data into the multimodal large model to obtain a device information analysis report; Step S6: Each regional server aggregates the device information analysis report to the main server, and the main server displays and stores each device information analysis report, and continuously optimizes the multimodal large model based on each device information analysis report.

2. The device information management method based on a multimodal large model as claimed in claim 1, characterized in that: The step S1 is specifically as follows: Using a time synchronization mechanism to obtain historical multimodal data of installed equipment, the historical multimodal data at least includes equipment operation logs, operation status images, operation status videos, operation temperatures, and operation voltages; Performing data cleaning operations on abnormal values, missing values, duplicate data, low-frequency words, stop words, punctuation marks, and special characters in each of the historical multimodal data; the duplicate data is duplicate text, duplicate image, duplicate video, or duplicate numerical value; The numerical historical multimodal data are normalized by using the MinMax normalization function, the categorical historical multimodal data are encoded by using the OneHot encoding, and the device text information including at least the production date, device model, device type, and rated voltage in the operating status image is identified by using the OCR technology, thereby completing the preprocessing of each of the historical multimodal data; After the preprocessed historical multimodal data are annotated with fault assessment and fault handling suggestions, a data set is constructed, and the data set is divided into a training set, a validation set, and a test set based on a ratio of 7:2:1; A vector database is created, and the device text information, fault assessment, and fault handling suggestions are stored in the vector database.

3. The device information management method based on a multimodal large model as claimed in claim 1, characterized in that: In step S2, the ViT module is used to extract image features from the running state image and the running state video; The Bert module is used to extract text features in the equipment operation log; The numerical feature construction module is used to construct a numerical feature set through operating temperature and operating voltage; The feature fusion module is used to normalize the scale of the image features, text features and numerical feature sets, perform weighted summation according to preset importance weights, and then fuse them into a feature representation vector; The output module is used to output a device information analysis report carrying device information statistical analysis results and fault prediction results according to the feature characterization vector; The hyperparameters include at least a random dropout rate, a weight decay rate, a learning rate, and a loss function.

4. The device information management method based on a multimodal large model as claimed in claim 1, characterized in that: The step S3 is specifically as follows: Based on the LoRA technology, the multimodal large model is trained through the training set, the hyperparameters are adjusted through the validation set to obtain the optimal hyperparameter combination, the multimodal large model is continued to be trained using the hyperparameter combination and the training set, and the multimodal large model is evaluated through the test set; A relational database is created, and the data set and the model parameters of the multimodal large model after evaluation are stored in the relational database.

5. The device information management method based on a multimodal large model as claimed in claim 1, characterized in that: The step S5 is specifically as follows: The compressed multimodal large model is deployed to each regional server, and each regional server collects real-time multimodal data of the installed equipment in the management area through a scanner, a camera, a temperature sensor, and a voltage sensor; Matching a preset query template based on the device type carried by the real-time multimodal data, constructing a first query statement based on the query template and the real-time multimodal data, converting the first query statement into an embedded vector using an embedding model, constructing a vector memory index based on the embedded vector, and querying a fault assessment and a fault handling suggestion from a vector database based on the vector memory index through RAG technology; A second query statement is constructed based on the real-time multimodal data, query fault assessment, fault handling suggestions and preset prompt templates, and the second query statement is input into the multimodal big model to obtain an equipment information analysis report carrying equipment information statistical analysis results and fault prediction results.

6. A device information management system based on a multimodal large model, characterized by: Includes the following modules: A data set construction module, used to obtain historical multimodal data of installed equipment, preprocess each of the historical multimodal data and construct a data set, and divide the data set into a training set, a validation set, and a test set; A multimodal large model creation module, used to create a multimodal large model for equipment information statistical analysis and fault prediction based on the ViT module, the Bert module, the numerical feature construction module, the feature fusion module and the output module, and set the hyperparameters of the multimodal large model; A multimodal large model training module, used for training and evaluating the multimodal large model based on the LoRA technology through the training set, the validation set and the test set; A multimodal large model compression module, used to compress the multimodal large model that has passed the evaluation by using knowledge distillation technology; The device information analysis module is used to deploy the compressed multimodal large model to each regional server, and each regional server collects real-time multimodal data of the installed devices in the management area, and inputs the real-time multimodal data into the multimodal large model to obtain a device information analysis report; The device information management module is used for each regional server to aggregate the device information analysis report to the main server, and the main server displays and stores each device information analysis report, and continuously optimizes the multimodal large model based on each device information analysis report.

7. The device information management system based on multimodal large model as claimed in claim 6, characterized in that: The data set construction module is specifically used for: Using a time synchronization mechanism to obtain historical multimodal data of installed equipment, the historical multimodal data at least includes equipment operation logs, operation status images, operation status videos, operation temperatures, and operation voltages; Performing data cleaning operations on abnormal values, missing values, duplicate data, low-frequency words, stop words, punctuation marks, and special characters in each of the historical multimodal data; the duplicate data is duplicate text, duplicate image, duplicate video, or duplicate numerical value; The numerical historical multimodal data are normalized by using the MinMax normalization function, the categorical historical multimodal data are encoded by using the OneHot encoding, and the device text information including at least the production date, device model, device type, and rated voltage in the operating status image is identified by using the OCR technology, thereby completing the preprocessing of each of the historical multimodal data; After the preprocessed historical multimodal data are annotated with fault assessment and fault handling suggestions, a data set is constructed, and the data set is divided into a training set, a validation set, and a test set based on a ratio of 7:2:1; A vector database is created, and the device text information, fault assessment, and fault handling suggestions are stored in the vector database.

8. The device information management system based on multimodal large model as claimed in claim 6, characterized in that: In the multimodal large model creation module, the ViT module is used to extract image features in the running state image and the running state video; The Bert module is used to extract text features in the equipment operation log; The numerical feature construction module is used to construct a numerical feature set through operating temperature and operating voltage; The feature fusion module is used to normalize the scale of the image features, text features and numerical feature sets, perform weighted summation according to preset importance weights, and then fuse them into a feature representation vector; The output module is used to output a device information analysis report carrying device information statistical analysis results and fault prediction results according to the feature characterization vector; The hyperparameters include at least a random dropout rate, a weight decay rate, a learning rate, and a loss function.

9. The device information management system based on multimodal large model as claimed in claim 6, characterized in that: The multimodal large model training module is specifically used for: Based on the LoRA technology, the multimodal large model is trained through the training set, the hyperparameters are adjusted through the validation set to obtain the optimal hyperparameter combination, the multimodal large model is continued to be trained using the hyperparameter combination and the training set, and the multimodal large model is evaluated through the test set; A relational database is created, and the data set and the model parameters of the multimodal large model after evaluation are stored in the relational database.

10. The device information management system based on multimodal large model as claimed in claim 6, characterized in that: The device information analysis module is specifically used for: The compressed multimodal large model is deployed to each regional server, and each regional server collects real-time multimodal data of the installed equipment in the management area through a scanner, a camera, a temperature sensor, and a voltage sensor; Matching a preset query template based on the device type carried by the real-time multimodal data, constructing a first query statement based on the query template and the real-time multimodal data, converting the first query statement into an embedded vector using an embedding model, constructing a vector memory index based on the embedded vector, and querying a fault assessment and a fault handling suggestion from a vector database based on the vector memory index through RAG technology; A second query statement is constructed based on the real-time multimodal data, query fault assessment, fault handling suggestions and preset prompt templates, and the second query statement is input into the multimodal big model to obtain an equipment information analysis report carrying equipment information statistical analysis results and fault prediction results.

Images