Multi-modal large model optimization method and system based on defect detection and analysis

Abstract

The invention discloses a multi-modal large model optimization method and system based on defect detection and analysis. The method comprises the following steps: S1, collecting a multi-modal original sample with power business scene characteristics; s2, performing standardization preprocessing on the original data to obtain a fine-tuned standard data set; and S3, establishing relational mapping among the labels of the training data set to form a semantic relation graph of the labels, and extracting a topological structure relation of the semantic relation graph to generate the power text knowledge base. S4, various task function modules of the computer vision large model are trained, so that the computer vision large model has various tasks for processing the electric power scene. And S5, fusing the trained Clip semantic large model and the trained computer vision large model into a multi-modal large model by using a Transform structure. According to the method, the multi-modal large model subjected to fine tuning through the power text knowledge base not only maintains the specific analysis and reasoning performance under various power scenes, but also can ensure the generalization ability of the large model to various algorithm tasks.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a multi-modal large model optimization method and system based on defect detection and analysis. Background Art

[0002] In recent years, with the continuous growth of electricity demand in my country, the power system has become an indispensable infrastructure for production and life. There are abundant algorithm requirements in the power scenario, such as judgment of personnel operation norms, estimation of power system status, assessment of system potential risks, detection and analysis of power equipment status, etc. In terms of power equipment defect identification (such as blurred and damaged equipment meter dials, broken insulators, rusted metal parts, etc.), defective equipment may bring huge hidden dangers. Traditional manual detection often fails to detect and warn in time, and the maintenance cost is high. Artificial intelligence algorithms combined with multimodal large models can effectively reduce the investment of manpower and material resources, and achieve rapid response, accurate alarm and all-weather monitoring.

[0003] With the rise of multimodal large model technology, multimodal large models have shown great application potential due to their excellent cross-modal representation and generalization capabilities. This technology is a cutting-edge direction in the field of artificial intelligence. By combining multiple data types such as prompt words, text, images, audio and video, the model's understanding, reasoning and generation capabilities are improved. By integrating information from more modes, the model can fully understand complex queries, realize cross-modal knowledge fusion and application, and significantly improve performance in rich scenarios of power systems.

[0004] However, the training data scale and parameter volume of multimodal large models are huge, which requires high computing resources. At the same time, they also face challenges such as high inference latency and difficult deployment. For vertical applications, the parameter volume and representation capabilities of large models may be too redundant, resulting in a waste of hardware storage, bandwidth and computing resources. Therefore, accelerating the deployment of multimodal large models in the power vertical field has become an urgent problem to be solved.

[0005] Therefore, the prior art has defects and needs to be improved. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a multimodal large model optimization method and system based on defect detection and analysis, so as to solve the problems of large number of weight parameters, high computing resource requirements, long training and deployment reasoning time, and difficulty in implementation in the field of power equipment during multimodal large model training.

[0007] The technical solution of the present invention is as follows: a multimodal large model optimization method based on defect detection and analysis is provided, comprising the following steps.

[0008] S1: Collect multimodal original samples with the characteristics of power business scenarios, classify and annotate them into multimodal large model training datasets with multi-scenario and multi-category defect labels according to data attributes.

[0009] S2: Standardize and preprocess the raw data to obtain a fine-tuned standard dataset, and design the required pre-trained multimodal large model based on the properties of the standard dataset and computing resources.

[0010] S3: Build a large Clip semantic model to extract the feature vectors of the semantic text of the standard data set; establish a relationship mapping between the labels of the training data set to form a semantic relationship graph of the labels, and extract the topological structure relationship of the semantic relationship graph to generate an electric power text knowledge base.

[0011] S4: Train various task function modules of the computer vision big model so that the computer vision big model can handle various tasks in power scenarios.

[0012] S5: The trained Clip semantic model and computer vision model are fused into a multimodal model using the Transformer structure. Model pruning and quantization methods are introduced to reduce the number of model parameters and memory usage, and low-rank matrix decomposition technology is used to accelerate model training and make up for precision loss. The Clip semantic model can realize text semantic recognition and processing, and the computer vision model can realize image recognition. By using the Transformer structure to fuse into a multimodal model, the multimodal model can have both text semantics and image recognition functions. When processing pictures of power scenes, it can identify pictures with safety hazards and output specific safety hazard problems.

[0013] The multimodal large model optimization method based on defect detection and analysis specifically includes the following steps.

[0014] S1: Collect and generate multimodal data samples with power scenario characteristics, and combine multiple modal sample information into training set data pairs as training samples for fine-tuning the multimodal large model of the power scenario, and then classify and label the multimodal large model fine-tuning training samples based on various power scenarios and equipment attributes, so as to obtain a multimodal large model training data set with multi-scenario and multi-category defect labels.

[0015] S2: performing standardized preprocessing on the multimodal large model data set, wherein the preprocessing refers to performing z-score standardized processing on the data samples in the multimodal large model data set, and obtaining a standard training data set D of the multimodal large model according to a division ratio of 9:1. train and the labeled test set D test, and designs the required pre-trained multimodal large model according to the attributes of the standard data set and computing resources. The pre-trained multimodal large model is obtained by fusion of the Clip semantic large model and the computer vision large model.

[0016] S3: Build a large Clip semantic model, extract the feature vector of the semantic text of the standard dataset, and use the training set D train The text features in the pre-training and fine-tuning of the Clip semantic model are used to learn the correlation between images and texts. The model will receive image and text pairs as input, and optimize the model parameters through comparative learning to improve the matching degree between images and texts. A convolutional model with an input layer, 10 convolution layers, 3 pooling layers and 1 fully connected layer is constructed. The convolutional model is used to relationalize and numerate the semantic association rules of power scenarios to obtain the semantic relationship graph of power scenario labels, and the power text knowledge base is generated according to the topological structure relationship of the semantic relationship graph.

[0017] S4: The functions of the computer vision big model include object classification, target defect detection, image generation and repair functions. Based on the above functions, the corresponding image feature extraction module of the computer vision big model is trained. According to different visual tasks, the feature classification module, image segmentation module, target positioning module, image generation and repair module are trained, so that the computer vision big model is capable of processing various tasks in power scenarios.

[0018] S5: The trained Clip semantic big model and computer vision big model are fused into a multimodal big model using the Transformer structure, and low-rank matrix decomposition technology is used for matrix operations and parameter fusion during the fusion of the multimodal big model to accelerate model training and make up for the loss of precision; the model quantization and pruning methods are determined according to the obtained multimodal big model, and the quantization method is used to reduce the number of parameters of the multimodal big model, and the multimodal big model network is deeply pruned to retain the multimodal big model's ability to process and generalize power scenarios and reduce the pressure on computing resources.

[0019] In step S1, the method for constructing the training set data pair is to construct the text data set T and the power scene image data set G collected after user verification in the power scene into a multimodal data set D = (T, G) with a total of N sample categories and a total of C. The text modality provides semantic guidance for the picture modality, and the picture modality provides application scenarios for the text modality, using the knowledge between different modalities to complement the insufficient information of a single modality.

[0020] In step S1, the classification and labeling of power scene training samples are rated in proportion according to the power scenes and equipment categories from the perspective of common situations, actual applicability and differences with other categories. The graded image samples are divided into iconic object images, iconic scene images and non-iconic images, and category labeling, instance localization and instance segmentation tasks are performed respectively. A large number of power scenes and equipment categories are labeled using a hierarchical method, and all categories are divided into K superclasses and S subclasses. Only K labelers are needed to divide the superclasses, and attribute labeling and classification are performed on the subclasses in each superclass. The hierarchical labeling method splits the dataset labeling task, allowing labelers to label in scenarios they are more familiar with, reducing the dataset labeling time, and using the hierarchical relationship of categories to ensure the accuracy of the multimodal dataset labeling task.

[0021] In step S4, the process of training the large computer vision model is as follows.

[0022] S41: Use weight sampler to adjust the training set D Train The data distribution of makes the sampling probability proportional to the inverse of the number of samples of this class and the total number of samples, so as to keep the samples of each batch balanced, and the probability p of each category being extracted i for: Among them, n i represents the number of samples in the i-th category, C is the total number of categories, B represents the size of the batch of data, and the sampling probability of the i-th category is At the same time, the sampling probability is adjusted by setting the value of δ, and the entire data distribution changes as the value of δ is adjusted.

[0023] S42: Using Feature Extractor Perform feature extraction on each batch of samples. The expression is:

[0024] S43: Calculate the feature vector extracted by the feature extractor using global average pooling The feature vector is then mapped to the fully connected layer f c , the parameter learned by the fully connected layer is W fo .

[0025] S44: Finally, the features mapped by the fully connected layer are input into Softmax to predict the probability Prediction probability The representation is as follows: in, is the predicted output of the feature vector.

[0026] S45: Calculate the loss function Y of the predicted classification and the true value by optimizing the objective function Y(x) LossGet the misclassification matrix, optimize the misclassification matrix to back-propagate and update the training parameter matrix of the entire model; the misclassification matrix is ​​expressed as:

[0027]

[0028] Among them, j, k represent the category index, p ij is the probability that sample i is predicted to be class j, Represents the predicted category of sample i.

[0029] In step S5, specific steps include:

[0030] S51: The Transformer structure fusion process is to calculate the separately constructed Clip semantics model and computer vision model, that is, to calculate the data features of the unimodal classification model through multiple layers of nonlinear transformation to obtain the probability vector distribution, and then normalize the probability vector; where the unimodal classification model refers to a single Clip semantics model and a single computer vision model, and the Transform structure operation is as follows:

[0031] q i =Transform(V F ),

[0032]

[0033] Among them, q i is the feature vector of data type i of the unimodal classification model, V F Represents the probability vector obtained after nonlinear transformation of the unimodal classification model; j represents the category index, is the probability of predicting category i and is the total number of categories.

[0034] S52: Test set D Test The data is input into the Transformer fusion model, and the error of each single-modal classification model is calculated separately. k , according to the loss situation, the probability matrix of the data of each single-modal classification model is assigned a fusion weight, and the fusion weight of each modality is expressed as:

[0035]

[0036] Where m is the total number of unimodal classification models, i and k represent the indexes of unimodal classification models, errork represents the error rate of unimodal classification model k, and ω k is the weight learned by the unimodal classification model k.

[0037] S53: Use Lora optimization technology to fine-tune the multimodal large model, and approximate the original high-dimensional and complex weight matrix with a low-rank matrix; use low-rank matrix decomposition technology to accelerate model training when realizing matrix operations and parameter fusion when fusing multimodal large models.

[0038] The core of Lora optimization technology is that during the fine-tuning process of multimodal large models, the changes in the weight matrix are low-rank; this means that although the original weight matrix may be high-dimensional and complex, during fine-tuning, the changes in the weight matrix can be approximated by a low-rank matrix; the expression of LoRA is: h=W0x+ΔWx=W0x+BAx.

[0039] Among them, W0 represents the pre-trained weight matrix, ΔW represents the parameter update during fine-tuning; LoRA restricts the weight update so that ΔW=BAx, W0x represents the original forward propagation process, that is, the input data x obtains the corresponding output vector through the weight matrix W0; (W0+BA)x means adding the original weight matrix W0 and the correction term BA to obtain the fine-tuned weight matrix, which is then multiplied by the input x to obtain the final output h; in the language model, x is the word embedding vector corresponding to the input text, W0 is the weight matrix from the text embedding layer to the hidden layer; BA x Represents the correction term introduced by the fine-tuning process, A and B are two low-rank matrices, and their product BA represents the correction to the original weight matrix W. For example, if the dimension of W is 1000×1000, the dimensions of A and B are 50×1000 and 1000×50 respectively, then the dimension of BA is 50×50.

[0040] The present invention also provides a multi-modal large model optimization system based on defect detection and analysis, which is used to train and manage the aforementioned multi-modal large model optimization method based on defect detection and analysis; comprising:

[0041] The data acquisition and annotation module is used to load and select the original data set of the power scenario, determine the applicable scenario of the training model, annotate the selected original data set with the official standard of the power scenario, add semantic information to the annotated data set, and construct a multimodal large model training data set with multi-scenario and multi-category defect labels.

[0042] A model training module uses a standardized preprocessed data set to obtain a standard multimodal data set, obtains data semantic associations and numerical relationships through clustering and quantization, constructs a selected power scenario label relationship map, converts it into a power text knowledge base, analyzes data set characteristics and power scenario requirements, designs a suitable basic multimodal large model architecture and fine-tuning scheme, and then adds low-rank decomposition and pruning technology to obtain the corresponding multimodal large model.

[0043] A model reasoning application module is used to manage the multimodal large model scheduling of different power text knowledge bases, provide multimodal large model post-processing related interfaces to regulate the output results of the multimodal large model, and analyze the computing resource status in real time to select a suitable model scheduling solution, thereby managing the model life cycle and computing memory overhead.

[0044] The multimodal large model fine-tuned by the power text knowledge base not only maintains the specific analysis and reasoning performance in various power scenarios, but also ensures the generalization ability of the multimodal large model for various algorithm tasks.

[0045] By adopting the above scheme, the present invention provides a multi-modal large model optimization method and system based on defect detection and analysis, which has the following technical effects:

[0046] (1) The method of the present invention generates a standardized training data set of a multi-modal large model with multi-scenario and multi-category defect labels through S1 and S2; S3: obtain the semantic relationship map of the power scene category labels according to the semantic association of the labels, and optimize the specific text knowledge base according to the topological structure of the relationship map, which effectively solves the knowledge differences in different power scenes; S4 and S5 add model pruning and quantization methods to further reduce the pressure of large model parameters and computing resources, and use matrix low-rank decomposition technology to fine-tune the training process of the multi-modal large model to make up for the model accuracy lost by pruning and quantization. The model pruning and quantization algorithm can streamline the complex pre-training model, alleviate the requirements of computer storage, bandwidth and computing, and realize the acceleration and implementation of the multi-modal large model reasoning in the power scene.

[0047] (2) The system of the present invention uses standardized and engineered methods to implement a series of modules such as data set preparation, model training, reasoning operation, and resource allocation for the electric power multimodal big model, thereby obtaining a multimodal big model that is easy to train and deploy. The multimodal big model fine-tuned by the electric power text knowledge base not only maintains the specific analysis and reasoning performance under various electric power scenarios, but also ensures the generalization ability of the big model for various algorithm tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of the method of the present invention;

[0049] Figure 2 It is a functional module diagram of the system of the present invention;

[0050] Figure 3 This is a schematic diagram of LoRA technology;

[0051] Figure 4 A screenshot of an embodiment of the present invention when applied;

[0052] Figure 5The following is a screenshot of another embodiment of the present invention. DETAILED DESCRIPTION

[0053] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0054] See also Figure 1 The present invention provides a multi-modal large model optimization method based on defect detection and analysis, which includes the following steps.

[0055] S1: Collect multimodal original samples with the characteristics of power business scenarios, classify and annotate them into multimodal large model training datasets with multi-scenario and multi-category defect labels according to data attributes.

[0056] S2: Standardize and preprocess the raw data to obtain a fine-tuned standard dataset, and design the required pre-trained multimodal large model based on the properties of the standard dataset and computing resources.

[0057] S3: Build a large Clip semantic model to extract the feature vectors of the semantic text of the standard data set; establish a relationship mapping between the labels of the training data set to form a semantic relationship graph of the labels, and extract the topological structure relationship of the semantic relationship graph to generate an electric power text knowledge base.

[0058] S4: Train various task function modules of the computer vision big model so that the computer vision big model can handle various tasks in power scenarios.

[0059] S5: The trained Clip semantic model and computer vision model are fused into a multimodal model using the Transformer structure. Model pruning and quantization methods are introduced to reduce the number of model parameters and memory usage. Low-rank matrix decomposition technology is used to accelerate model training and compensate for precision loss.

[0060] The multimodal large model optimization method based on defect detection and analysis specifically includes the following steps.

[0061] S1: Collect and generate multimodal data samples with power scenario characteristics, and combine multiple modal sample information into training set data pairs as training samples for fine-tuning the multimodal large model of the power scenario, and then classify and label the multimodal large model fine-tuning training samples based on various power scenarios and equipment attributes, so as to obtain a multimodal large model training data set with multi-scenario and multi-category defect labels.

[0062] S2: performing standardized preprocessing on the multimodal large model data set, wherein the preprocessing refers to performing z-score standardized processing on the data samples in the multimodal large model data set, and obtaining a standard training data set D of the multimodal large model according to a division ratio of 9:1. train and the labeled test set D test, and designs the required pre-trained multimodal large model according to the attributes of the standard data set and computing resources. The pre-trained multimodal large model is obtained by fusion of the Clip semantic large model and the computer vision large model.

[0063] S3: Build a large Clip semantic model, extract the feature vector of the semantic text of the standard dataset, and use the training set D train The text features in the pre-training and fine-tuning of the Clip semantic model are used to learn the correlation between images and texts. The model will receive image and text pairs as input, and optimize the model parameters through comparative learning to improve the matching degree between images and texts. A convolutional model with an input layer, 10 convolution layers, 3 pooling layers and 1 fully connected layer is constructed. The convolutional model is used to relationalize and numerate the semantic association rules of power scenarios to obtain the semantic relationship graph of power scenario labels, and the power text knowledge base is generated according to the topological structure relationship of the semantic relationship graph.

[0064] S4: The functions of the computer vision big model include object classification, target defect detection, image generation and repair functions. Based on the above functions, the corresponding image feature extraction module of the computer vision big model is trained. According to different visual tasks, the feature classification module, image segmentation module, target positioning module, image generation and repair module are trained, so that the computer vision big model is capable of processing various tasks in power scenarios.

[0065] S5: The trained Clip semantic big model and computer vision big model are fused into a multimodal big model using the Transformer structure, and low-rank matrix decomposition technology is used for matrix operations and parameter fusion during the fusion of the multimodal big model to accelerate model training and make up for the loss of precision; the model quantization and pruning methods are determined according to the obtained multimodal big model, and the quantization method is used to reduce the number of parameters of the multimodal big model, and the multimodal big model network is deeply pruned to retain the multimodal big model's ability to process and generalize power scenarios and reduce the pressure on computing resources.

[0066] In step S1, the method for constructing the training set data pair is to construct the text data set T and the power scene image data set G collected after user verification in the power scene into a multimodal data set D = (T, G) with a total of N sample categories and a total of C. The text modality provides semantic guidance for the picture modality, and the picture modality provides application scenarios for the text modality, using the knowledge between different modalities to complement the insufficient information of a single modality.

[0067] In step S1, the classification and labeling of power scene training samples are rated in proportion according to the power scenes and equipment categories from the perspective of common situations, actual applicability and differences with other categories. The graded image samples are divided into iconic object images, iconic scene images and non-iconic images, and category labeling, instance localization and instance segmentation tasks are performed respectively. A large number of power scenes and equipment categories are labeled using a hierarchical method, and all categories are divided into K superclasses and S subclasses. Only K labelers are needed to divide the superclasses, and attribute labeling and classification are performed on the subclasses in each superclass. The hierarchical labeling method splits the dataset labeling task, allowing labelers to label in scenarios they are more familiar with, reducing the dataset labeling time, and using the hierarchical relationship of categories to ensure the accuracy of the multimodal dataset labeling task.

[0068] In step S4, the process of training the large computer vision model is as follows.

[0069] S41: Use weight sampler to adjust the training set D Train The data distribution of makes the sampling probability proportional to the inverse of the number of samples of this class and the total number of samples, so as to keep the samples of each batch balanced, and the probability p of each category being extracted i for: Among them, n i represents the number of samples in the i-th category, C is the total number of categories, B represents the size of the batch of data, and the sampling probability of the i-th category is At the same time, the sampling probability is adjusted by setting the value of δ, and the entire data distribution changes as the value of δ is adjusted.

[0070] S42: Using Feature Extractor Perform feature extraction on each batch of samples. The expression is:

[0071] S43: Calculate the feature vector extracted by the feature extractor using global average pooling The feature vector is then mapped to the fully connected layer fc, and the parameters learned by the fully connected layer are W fo .

[0072] S44: Finally, the features mapped by the fully connected layer are input into Softmax to predict the probability Prediction probability The representation is as follows: in, is the predicted output of the feature vector.

[0073] S45: Calculate the loss function Y of the predicted classification and the true value by optimizing the objective function Y(x) LossGet the misclassification matrix, optimize the misclassification matrix to back-propagate and update the training parameter matrix of the entire model; the misclassification matrix is ​​expressed as:

[0074]

[0075] Among them, j, k represent the category index, p ij is the probability that sample i is predicted to be class j, Represents the predicted category of sample i.

[0076] In step S5, specific steps include:

[0077] S51: The Transformer structure fusion process is to calculate the separately constructed Clip semantic model and computer vision model, that is, to calculate the data features of the unimodal classification model through multiple layers of nonlinear transformation to obtain the probability vector distribution, and then normalize the probability vector; where the unimodal classification model refers to a single Clip semantic model and a computer vision model, and the Transform structure operation is as follows:

[0078] q i =Transform(V F )

[0079]

[0080] Among them, q i is the feature vector of data type i of the unimodal classification model, V F Represents the probability vector obtained after nonlinear transformation of the unimodal classification model; j represents the category index, is the probability of predicting category i and is the total number of categories.

[0081] S52: Test set D Test The data is input into the Transformer fusion model, and the error of each single-modal classification model is calculated separately. k , according to the loss situation, the probability matrix of the data of each single-modal classification model is assigned a fusion weight, and the fusion weight of each modality is expressed as:

[0082]

[0083] Where m is the total number of unimodal classification models, i and k represent the indexes of unimodal classification models, errork represents the error rate of unimodal classification model k, and ω k is the weight learned by the unimodal classification model k.

[0084] S53: Use Lora optimization technology to fine-tune the multimodal large model, and approximate the original high-dimensional and complex weight matrix with a low-rank matrix; use low-rank matrix decomposition technology to accelerate model training when realizing matrix operations and parameter fusion when fusing multimodal large models.

[0085] See also Figure 2 The core of the Lora optimization technology is that the change of the weight matrix is ​​low-rank during the fine-tuning process of the multimodal large model; this means that although the original weight matrix may be high-dimensional and complex, during fine-tuning, the change of the weight matrix can be approximated by a low-rank matrix; the expression of LoRA is: h = W0x + ΔWx = W0x + DAx.

[0086] Among them, W0 represents the pre-trained weight matrix, ΔW represents the parameter update during fine-tuning; LoRA restricts the weight update so that ΔW=BAx, W0x represents the original forward propagation process, that is, the input data x obtains the corresponding output vector through the weight matrix W0; (W0+BA)x means adding the original weight matrix W0 and the correction term BA to obtain the fine-tuned weight matrix, which is then multiplied by the input x to obtain the final output h; in the language model, x is the word embedding vector corresponding to the input text, W0 is the weight matrix from the text embedding layer to the hidden layer; BA x Represents the correction term introduced by the fine-tuning process, A and B are two low-rank matrices, and their product BA represents the correction to the original weight matrix W. For example, if the dimension of w is 1000×1000, the dimensions of A and B are 50×1000 and 1000×50 respectively, then the dimension of BA is 50×50.

[0087] See also Figure 3 The present invention also provides a multi-modal large model optimization system based on defect detection and analysis, which is used to train and manage the aforementioned multi-modal large model optimization method based on defect detection and analysis; comprising:

[0088] The data acquisition and annotation module is used to load and select the original data set of the power scenario, determine the applicable scenario of the training model, annotate the selected original data set with the official standard of the power scenario, add semantic information to the annotated data set, and construct a multimodal large model training data set with multi-scenario and multi-category defect labels.

[0089] A model training module uses a standardized preprocessed data set to obtain a standard multimodal data set, obtains data semantic associations and numerical relationships through clustering and quantization, constructs a selected power scenario label relationship map, converts it into a power text knowledge base, analyzes data set characteristics and power scenario requirements, designs a suitable basic multimodal large model architecture and fine-tuning scheme, and then adds low-rank decomposition and pruning technology to obtain the corresponding multimodal large model.

[0090] A model reasoning application module is used to manage the multimodal large model scheduling of different power text knowledge bases, provide multimodal large model post-processing related interfaces to regulate the output results of the multimodal large model, and analyze the computing resource status in real time to select a suitable model scheduling solution, thereby managing the model life cycle and computing memory overhead.

[0091] The multimodal large model fine-tuned by the power text knowledge base not only maintains the specific analysis and reasoning performance in various power scenarios, but also ensures the generalization ability of the multimodal large model for various algorithm tasks.

[0092] See also Figure 4 and Figure 5 When using the trained multimodal large model to perform image analysis tasks, the operator needs to upload a task image to be analyzed and provide text task requirements that require reasoning and analysis. The multimodal large model can read the uploaded image and analyze the image based on the task requirements provided by the operator and combined with its own knowledge base, and output an analysis report for the task.

[0093] In summary, the present invention provides a multi-modal large model optimization method and system based on defect detection and analysis, which has the following technical effects:

[0094] (1) The method of the present invention generates a standardized training data set of a multi-modal large model with multi-scenario and multi-category defect labels through S1 and S2; S3: obtain the semantic relationship map of the power scene category labels according to the semantic association of the labels, and optimize the specific text knowledge base according to the topological structure of the relationship map, which effectively solves the knowledge differences in different power scenes; S4 and S5 add model pruning and quantization methods to further reduce the pressure of large model parameters and computing resources, and use matrix low-rank decomposition technology to fine-tune the training process of the multi-modal large model to make up for the model accuracy lost by pruning and quantization. The model pruning and quantization algorithm can streamline the complex pre-training model, alleviate the requirements of computer storage, bandwidth and computing, and realize the acceleration and implementation of the multi-modal large model reasoning in the power scene.

[0095] (2) The system of the present invention uses standardized and engineered methods to implement a series of modules such as data set preparation, model training, reasoning operation, and resource allocation for the electric power multimodal big model, thereby obtaining a multimodal big model that is easy to train and deploy. The multimodal big model fine-tuned by the electric power text knowledge base not only maintains the specific analysis and reasoning performance under various electric power scenarios, but also ensures the generalization ability of the big model for various algorithm tasks.

[0096] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-modal large model optimization method based on defect detection and analysis, characterized in that: The following steps are involved: S1: Collect multimodal original samples with power business scenario characteristics, classify and annotate them into multimodal large model training data sets with multi-scenario and multi-category defect labels according to data attributes; S2: Standardize and preprocess the raw data to obtain a fine-tuned standard dataset, and design the required pre-trained multimodal large model based on the properties of the standard dataset and computing resources; S3: Build a large Clip semantic model to extract feature vectors of semantic texts from standard datasets; Establish a relationship mapping between labels in the training data set to form a semantic relationship graph of the labels, and extract the topological structure relationship of the semantic relationship graph to generate an electric power text knowledge base; S4: Train various task function modules of the computer vision big model, so that the computer vision big model can handle various tasks of power scenarios; S5: The trained Clip semantic model and computer vision model are fused into a multimodal large model using the Transformer structure. Model pruning and quantization methods are introduced to reduce the number of model parameters and memory usage. Low-rank matrix decomposition technology is used to accelerate model training and make up for precision loss.

2. The multimodal large model optimization method based on defect detection and analysis according to claim 1 is characterized in that: The specific steps include: S1: Collect and generate multimodal data samples with power scene characteristics, and combine multiple modal sample information into training set data pairs as training samples for fine-tuning the multimodal large model of the power scene, and then classify and label the training samples for fine-tuning the multimodal large model based on various power scenes and equipment attributes, so as to obtain a multimodal large model training data set with multi-scene and multi-category defect labels; S2: performing standardized preprocessing on the multimodal large model data set, wherein the preprocessing refers to performing z-score standardized processing on the data samples in the multimodal large model data set, and obtaining a standard training data set D of the multimodal large model according to a division ratio of 9:

1. train and the labeled test set D test ; and design the required pre-trained multimodal large models based on the properties of standard datasets and computing resources; S3: Build a large Clip semantic model, extract the feature vector of the semantic text of the standard dataset, and use the training set D train In the pre-training and fine-tuning of the Clip semantic model, the association between images and texts is learned. The model receives image and text pairs as input, and optimizes the parameters of the Clip semantic model through comparative learning to improve the matching degree between images and texts. A convolutional model with an input layer, 10 convolution layers, 3 pooling layers, and 1 fully connected layer is constructed. The convolutional model is used to relationalize and quantify the semantic association rules of power scenarios to obtain the semantic relationship map of power scenario labels, and the power text knowledge base is generated according to the topological structure relationship of the semantic relationship map. S4: The functions of the computer vision big model include object classification, target defect detection, image generation and repair functions. Based on the above functions, the corresponding image feature extraction module of the computer vision big model is trained. According to different visual tasks, the feature classification module, image segmentation module, target positioning module, image generation and repair module are trained, so that the computer vision big model can handle various tasks in power scenarios. S5: The trained Clip semantic big model and computer vision big model are fused into a multimodal big model using the Transformer structure, and low-rank matrix decomposition technology is used for matrix operations and parameter fusion during the fusion of the multimodal big model to accelerate model training and make up for the loss of precision; the model quantization and pruning methods are determined according to the obtained multimodal big model, and the quantization method is used to reduce the number of parameters of the multimodal big model, and the multimodal big model network is deeply pruned to retain the multimodal big model's ability to process and generalize power scenarios and reduce the pressure on computing resources.

3. The multi-modal large model optimization method based on defect detection and analysis according to claim 2 is characterized in that: In step S1, the method for constructing the training set data pair is to construct the text data set T and the power scene image data set G collected after user verification in the power scene into a multimodal data set D = (T, G) with a total of N sample categories and a total of C. The text modality provides semantic guidance for the picture modality, and the picture modality provides application scenarios for the text modality, using the knowledge between different modalities to complement the insufficient information of a single modality.

4. The multi-modal large model optimization method based on defect detection and analysis according to claim 2 is characterized in that: In step S1, the classification and labeling of power scene training samples are rated in proportion according to the power scenes and equipment categories from the perspective of common situations, actual applicability and differences with other categories. The graded image samples are divided into iconic object images, iconic scene images and non-iconic images, and category labeling, instance localization and instance segmentation tasks are performed respectively. A large number of power scenes and equipment categories are labeled using a hierarchical method, and all categories are divided into K superclasses and S subclasses. Only K labelers are needed to divide the superclasses, and attribute labeling and classification are performed on the subclasses in each superclass. The hierarchical labeling method splits the dataset labeling task, allowing labelers to label in scenarios they are more familiar with, reducing the dataset labeling time, and using the hierarchical relationship of categories to ensure the accuracy of the multimodal dataset labeling task.

5. The multi-modal large model optimization method based on defect detection and analysis according to claim 2 is characterized in that: In step S4, the process of training the large computer vision model is as follows: S41: Use weight sampler to adjust the training set D Train The data distribution of makes the sampling probability proportional to the inverse of the number of samples of this class and the total number of samples, so as to keep the samples of each batch balanced, and the probability p of each category being extracted i for: Among them, n i represents the number of samples in the i-th category, C is the total number of categories, B represents the size of the batch of data, and the sampling probability of the i-th category is At the same time, by setting the value of δ to adjust the sampling probability, the entire data distribution changes as the value of δ is adjusted; S42: Using Feature Extractor Perform feature extraction on each batch of samples. The expression is: S43: Calculate the feature vector extracted by the feature extractor using global average pooling The feature vector is then mapped to the fully connected layer f c , the parameter learned by the fully connected layer is W f0 ; S44: Finally, the features mapped by the fully connected layer are input into Softmax to predict the probability Prediction probability The representation is as follows: in, is the predicted output of the feature vector. S45: Calculate the loss function Y of the predicted classification and the true value by optimizing the objective function Y(x) Loss Get the misclassification matrix, optimize the misclassification matrix to back-propagate and update the training parameter matrix of the entire model; the misclassification matrix is ​​expressed as: Among them, j, k represent the category index, p ij is the probability that sample i is predicted to be class j, Represents the predicted category of sample i.

6. The multi-modal large model optimization method based on defect detection and analysis according to claim 2 is characterized in that: In step S5, the specific steps include: S51: The Transformer structure fusion process is to calculate the separately constructed Clip semantics model and computer vision model, that is, to calculate the data features of the unimodal classification model through multiple layers of nonlinear transformation to obtain the probability vector distribution, and then normalize the probability vector; where the unimodal classification model refers to a single Clip semantics model and a single computer vision model, and the Transform structure operation is as follows: Among them, q i is the feature vector of data type i of the unimodal classification model, V F Represents the probability vector obtained after nonlinear transformation of the unimodal classification model; j represents the category index, is the probability of predicting category i and C is the total number of categories; S52: Test set D Test The data is input into the Transformer fusion model, and the error of each single-modal classification model is calculated separately. k , according to the loss situation, the probability matrix of the data of each single-modal classification model is assigned a fusion weight, and the fusion weight of each modality is expressed as: Among them, m is the total number of unimodal classification models, i and k represent the index of the unimodal classification model, and error k represents the error rate of the unimodal classification model k, ω k is the weight learned by the unimodal classification model k; S53: Use Lora optimization technology to fine-tune the multimodal large model, and approximate the original high-dimensional and complex weight matrix with a low-rank matrix; use low-rank matrix decomposition technology to accelerate model training when realizing matrix operations and parameter fusion when fusing multimodal large models.

7. The multi-modal large model optimization method based on defect detection and analysis according to claim 6 is characterized in that: In step S53, the core of the Lora optimization technology is that the change of the weight matrix is ​​low-rank during the fine-tuning process of the multimodal large model; this means that although the original weight matrix may be high-dimensional and complex, during fine-tuning, the change of the weight matrix can be approximated by a low-rank matrix; the expression of LoRA is: h = W0x + ΔWx = W0x + BAx, Among them, W0 represents the pre-trained weight matrix, ΔW represents the parameter update during fine-tuning; LoRA limits the weight update so that ΔW = BAx, W0x represents the original forward propagation process, that is, the input data x obtains the corresponding output vector through the weight matrix W0; (W0+BA)x means adding the original weight matrix W0 and the correction term BA to obtain the fine-tuned weight matrix, and then multiplying it with the input x to obtain the final output h; in the language model, x is the word embedding vector corresponding to the input text, W0 is the weight matrix from the text embedding layer to the hidden layer; BA x Represents the correction term introduced by the fine-tuning process, A and B are two low-rank matrices, and their product BA represents the correction to the original weight matrix W.

8. A multi-modal large model optimization system based on defect detection and analysis, characterized in that: A multimodal large model optimization method based on defect detection and analysis for training and managing any one of claims 1 to 7; comprising: A data acquisition and annotation module, which is used to load and select the original data set of the power scenario, determine the applicable scenario of the training model, annotate the selected original data set with the official standard of the power scenario, add semantic information to the annotated data set, and construct a multi-modal large model training data set with multi-scenario and multi-category defect labels; A model training module, wherein the model training module uses a standardized preprocessed data set to obtain a standard multimodal data set, obtains data semantic associations and numerical relationships by clustering and quantization, respectively, constructs a selected power scenario label relationship map, converts it into a power text knowledge base, and then analyzes the data set characteristics and power scenario requirements, designs a suitable basic multimodal large model architecture and fine-tuning scheme, and then adds low-rank decomposition and pruning technology to obtain a corresponding multimodal large model; A model reasoning application module is used to manage the multimodal large model scheduling of different power text knowledge bases, provide multimodal large model post-processing related interfaces to regulate the output results of the multimodal large model, and analyze the computing resource status in real time to select a suitable model scheduling solution, thereby managing the model life cycle and computing memory overhead.

9. The multi-modal large model optimization system based on defect detection and analysis according to claim 8, characterized in that: The multimodal large model fine-tuned by the power text knowledge base not only maintains the specific analysis and reasoning performance in various power scenarios, but also ensures the generalization ability of the multimodal large model for various algorithm tasks.

Images