Power equipment defect detection method and system based on multi-modal large model

Abstract

The invention discloses an electrical equipment defect detection method and system based on a multi-modal large model, and the method comprises the steps: analyzing a collected electrical equipment image, obtaining an equipment type and an equipment defect in the electrical equipment image, and marking the equipment type and the equipment defect; based on the equipment types and the equipment defects, respectively constructing corresponding tables of the equipment types and the equipment defects with attributes; constructing a self-attention equipment fault detection model; and inputting the marked electrical equipment image into the self-attention equipment fault detection model for training, and setting a basic task until the precision of the basic task reaches a specified threshold, thereby completing defect detection of the electrical equipment. According to the method, the problems of uncertain data preparation standard, indefinite training process, low professional adaptability and the like of the current large model in training and application in the power field can be solved, the accuracy of the large model in power equipment defect detection is effectively improved, and the operation and inspection quality efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of power equipment detection, and relates to a power equipment defect detection method and system based on a multi-modal large model. Background Art

[0002] As a core component of the power system, the stability and reliability of power equipment are crucial to ensuring power supply. However, with the increase in the service life of the equipment and the influence of environmental factors, power equipment is prone to various faults. If these faults are not discovered and handled in time, they will have a serious impact on the operation of the power system and even cause large-scale power outages. Therefore, power equipment detection is particularly important. At present, power equipment detection mainly collects images or operating data of power equipment, identifies and analyzes the information in the image or anomalies in the data to achieve quality detection or fault warning. However, when the label position on the surface of the power equipment is not completely consistent with the standard position, the image defect detection technology may misjudge it as a defect. At the same time, in the detection of abnormal operation data of power equipment, a large amount of manual data analysis and judgment is still required, resulting in low detection efficiency. Summary of the invention

[0003] The purpose of the present invention is to solve the problem that in the prior art, a large amount of manual data analysis and judgment is still required in the detection of abnormal operation data of power equipment, and to provide a power equipment defect detection method and system based on a multimodal large model.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] The present invention proposes a method for detecting defects in electric power equipment based on a multi-modal large model, comprising:

[0006] Analyze the collected power equipment images, obtain the equipment types and equipment defects in the power equipment images, and mark the equipment types and equipment defects in the power equipment images;

[0007] Based on the equipment type and equipment defect, determine the correspondence between the equipment defect and the equipment, and construct the correspondence table between the equipment type and equipment defect and the attribute respectively;

[0008] Build a self-attention device fault detection model with the ability to jointly learn text and images;

[0009] Based on the correspondence table of equipment types, equipment defects and attributes, the annotated power equipment images are input into the self-attention equipment fault detection model for training, and basic tasks are set until the accuracy of the basic tasks reaches the specified threshold to complete the power equipment defect detection.

[0010] Preferably, the collected power equipment image is analyzed to obtain the equipment type and equipment defects in the power equipment image, specifically:

[0011] Based on the collected power equipment images, an image dataset D is formed, and the images I = {I1, I2, ...} in the image dataset D are respectively established as device description sets S = {S1, S2, ..., S m} and defect description set Q = {Q1, Q2, ..., Q n}, where S m is the Chinese label of the device description, m is the total number of device types included in the dataset D; Q n is the Chinese label of the defect description, and n is the total number of defect types contained in the dataset D;

[0012] The equipment types and equipment defects in the power equipment images are marked, specifically:

[0013] Based on the device description set S = {S1, S2, ..., S m} and defect description set Q = {Q1, Q2, ..., Q n}, annotate the targets in each image and save the annotation results, which include the label name and annotation box coordinates (x, y, w, h) of the target of interest in the corresponding image, which respectively represent the horizontal and vertical coordinates of the upper left corner and lower right corner of the annotation box.

[0014] Preferably, based on the device type and the device defect, the correspondence between the device defect and the device is determined, and a correspondence table between the device type and the device defect and the attribute is constructed respectively, specifically:

[0015] Based on the device description set S and the defect description set Q, a device and defect correspondence table DYB is established. Each type of defect corresponds to a device, and each device can contain multiple defects. DYB = {S1 = Q11, Q12, ..., S2 = Q21, Q22, ..., ...;

[0016] Based on the device description set S, establish the correspondence table DYB between the device and its attributes S , characterizing the characteristics of the current device, among which DYB S ={T s1 = {Z 11 , Z 12 , …}, T s2 = {Z 21 , Z 22 , …}, …};

[0017] Based on the defect description set Q, establish the corresponding table DYB between the equipment and its attributes Q , characterizing the characteristics of the current device; among them, DYB Q={T q1 = {Y 11 , Y 12 , …}, T q2 = {Y 21 , Y 22 , …}, …}.

[0018] Preferably, the workflow of the self-attention device fault detection model with the ability of joint learning of text and image is as follows:

[0019] For the input text information T, the features are extracted through the text encoding network to obtain the text feature matrix F T ;

[0020] The input image information is extracted through the convolutional neural network CNN to obtain the image feature matrix F I ;

[0021] Based on the image feature F I And the text feature matrix F T , feature fusion is performed through an improved cross self-attention mechanism network to obtain text features fused with image information and image features fused with text information;

[0022] Based on the text features fused with image information and the image features fused with text information, the final image features are obtained through the attention mechanism of the text feature layer;

[0023] Decode the final image features based on the image attention module and the text attention module to obtain the output features;

[0024] Based on the output features, the position of the prediction box is obtained through the fully connected layer;

[0025] The self-attention device fault detection model is tuned based on the multi-task loss function. When the loss function is minimized, the optimal self-attention device fault detection model is obtained.

[0026] Preferably, the input text information T is subjected to feature extraction through a text encoding network to obtain a text feature matrix F T , specifically:

[0027] The text encoding network is composed of 12 Transformer encoders in series. i |i∈{1,2,3,4,5,6,7,8,9,10,11,12}}; each encoder e i ∈E is composed of several submodules with the same structure in series, each of which contains a multi-head self-attention module, a residual connection, a layer normalization, and a fully connected layer; each word in the input text sequence is first randomly initialized to obtain the corresponding initial word embedding ν0, which is then embedded into ν by superimposing the positionp and text embedding v s Then input the text encoding network to obtain the text feature F T ,

[0028] F T =TextEncoder(T)

[0029] The input image information is subjected to a convolutional neural network (CNN) to extract features and obtain an image feature matrix F I , specifically:

[0030] Input image I∈R 3×1280×1280 The image blocks are obtained by convolution with several convolution kernels, and each image block is embedded with v by superimposing the position p After that, it is input into the image encoding network E′ which is similar to the text encoding network structure to obtain the final image feature F I , where F I =CNN(I).

[0031] Preferably, the image feature F I And the text feature matrix F T , feature fusion is performed through the improved cross self-attention mechanism network to obtain text features fused with image information and image features fused with text information, specifically:

[0032] The improved cross self-attention mechanism network includes a multi-head self-attention module, a cross-image-text joint attention module and a cross-text-image joint attention module; the word embedding ν1 is calculated by the multi-head self-attention module to obtain the key vector K v Sum value vector V v The image feature F1 is calculated by the multi-head self-attention module to obtain the query vector Q F , the cross-image-text joint attention module is implemented by formula F v =softmax(Q F K v )V v , get the image features fused with text information The cross-text-image joint attention module is implemented by formula v F =softmax(Q v K F )V F Get text features that fuse image information

[0033] The text features based on the fused image information and the image features based on the fused text information are used to obtain the final image features through the text feature layer attention mechanism, specifically:

[0034] Based on M=Fv ×v F , get the similarity measurement matrix M between image features and text features, take the maximum value of each row vector in M ​​as the corresponding image feature and the correlation of the text feature most related to the image feature, sort the correlation from large to small and select the positions of the image features corresponding to the first several values ​​as the final selected image feature F final .

[0035] Preferably, the image attention module and the text attention module are used to decode the final image features to obtain output features, specifically:

[0036] Based on the final selected image feature F final Input to the image attention module, the image attention module uses the formula Update the query vector Q1, and the text attention module is calculated by formula Update to obtain query vector Q2; the output feature is query vector Q2;

[0037] The output feature is passed through the fully connected layer to obtain the position of the prediction box, specifically: the output feature Q2 is operated through the fully connected layer FFN1 to obtain the correlation matrix Wherein, d1 and d2 represent the number of prediction boxes output by the model and the number of input prompt words respectively, and the numerical value of each position in O represents the correlation between the prediction box and the prompt word corresponding to the position; the output feature Q2 is calculated by the fully connected layer FFN2 to obtain the position of the prediction box (x1, y1, x2, y2);

[0038] The self-attention device fault detection model is tuned based on the multi-task loss function, specifically: for the regression of the prediction box, the smooth L1 loss is used to update the model parameters; for the correlation between the prediction box and the input prompt word, the focal loss loss is used to update the model parameters.

[0039] Preferably, the basic task is set until the accuracy of the basic task reaches a specified threshold and the power equipment defect detection is completed, specifically:

[0040] Based on the annotated equipment and defect data, the self-attention equipment fault detection model is trained on a single type, where the input text of each picture is "find" plus the equipment name or "find" plus the defect name. If there are multiple types of defects in the picture, only one type of equipment or defect is displayed each time. If there are multiple types of equipment or defects in the picture, they are trained separately.

[0041] Multi-type training is performed on the self-attention device fault detection model, where the input text of each image is "find" plus all the devices and defect names in the image, and different targets are separated by ",";

[0042] For target devices and defects, use the target descriptive words in random combination for training; the input text is "find" plus the device or defect with the descriptive words;

[0043] For images containing multiple target devices and defects, the keywords "include" and "each" are used for training. This part trains the model to identify the specified device or defect capability on a certain associated device based on the device type affiliation.

[0044] The present invention proposes a power equipment defect detection system based on a multi-modal large model, comprising:

[0045] An acquisition module, wherein the acquisition module analyzes the collected power equipment image, acquires the equipment type and equipment defects in the power equipment image, and marks the equipment type and equipment defects in the power equipment image;

[0046] A first construction module, wherein the first construction module determines a correspondence between a device defect and a device based on a device type and a device defect, and constructs a correspondence table between a device type and a device defect and an attribute respectively;

[0047] A second building block, wherein the second building block builds a self-attention device fault detection model with text and image joint learning capability;

[0048] A training module, based on a correspondence table between equipment type and equipment defects and attributes, inputs the annotated power equipment images into a self-attention equipment fault detection model for training, and sets basic tasks until the accuracy of the basic tasks reaches a specified threshold to complete the power equipment defect detection.

[0049] A terminal device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of a method for detecting defects in electric power equipment based on a multimodal large model when executing the computer program.

[0050] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a method for detecting defects in electric power equipment based on a multi-modal large model.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] The present invention proposes a method for detecting defects in electric equipment based on a multimodal large model, which analyzes the collected electric equipment image, obtains the equipment type and equipment defect in the electric equipment image, and annotates the equipment type and equipment defect in the electric equipment image; based on the equipment type and equipment defect, respectively constructs a corresponding table of equipment type and equipment defect and attribute; constructs a self-attention equipment fault detection model; based on the corresponding table of equipment type and equipment defect and attribute, the annotated electric equipment image is input into the self-attention equipment fault detection model for training, and basic tasks are set until the accuracy of the basic tasks reaches a specified threshold, and the electric equipment defect detection is completed. The present invention can solve the problems of uncertain data preparation standards, unclear training processes, and weak professional adaptability in the current training and application of large models in the electric power field, effectively improve the accuracy of large models in electric equipment defect detection, and improve the quality and efficiency of operation and inspection.

[0053] The present invention proposes a power equipment defect detection system based on a multimodal large model, which completes the power equipment defect detection by dividing the system into an acquisition module, a first construction module, a second construction module and a training module. The modularization concept is adopted to make each module independent of each other, which is convenient for unified management of each module. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0055] Figure 1 The present invention is a flow chart of the method for detecting defects in electric power equipment based on a multi-modal large model.

[0056] Figure 2 Schematic diagram of the workflow of the self-attention device fault detection model.

[0057] Figure 3 Another schematic diagram of the flow chart of the method for detecting defects in electric power equipment based on a multi-modal large model of the present invention.

[0058] Figure 4 This is a network architecture diagram of the power equipment defect detection based on the multimodal large model of the present invention.

[0059] Figure 5 This is a diagram of the power equipment defect detection system based on a multi-modal large model of the present invention.

[0060] Figure 6 The figure is a schematic structural diagram of an electronic device of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0062] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0064] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0065] The present invention proposes a method for detecting defects in power equipment based on a multi-modal large model, such as Figure 1 As shown, including:

[0066] S1. Analyze the collected power equipment image, obtain the equipment type and equipment defects in the power equipment image, and mark the equipment type and equipment defects in the power equipment image;

[0067] Based on the collected power equipment images, an image dataset D is constructed, and the images I = {I1, I2, ...} in the image dataset D are used to establish a device description set S = {S1, S2, ..., S m} and defect description set Q = {Q1, Q2, ..., Q n}, where the device description S m is a Chinese label, corresponding to the label name of all equipment in image I, expressed as a single word, such as: insulator, pole transformer, tower, crossarm, m is the total number of equipment types included in the dataset D; defect description Q n is a Chinese label containing the label name of the equipment defect in the corresponding image I, such as: large area damage, yellow severe rust, longitudinal crack, and n is the total number of defect types contained in the data set D.

[0068] The equipment types and equipment defects in the power equipment image are annotated as follows: based on the equipment description set S = {S1, S2, ..., Sm} and defect description set Q = {Q1, Q2, ..., Q n}, annotate the targets in each image and save the annotation results, which include the label name and annotation box coordinates (x, y, w, h) of the target of interest in the corresponding image, which respectively represent the horizontal and vertical coordinates of the upper left corner and lower right corner of the annotation box.

[0069] S2. Based on the device type and device defect, determine the correspondence between the device defect and the device, and construct a correspondence table between the device type and device defect and the attribute respectively;

[0070] Based on the device description set S and the defect description set Q, a device and defect correspondence table DYB is established. Each type of defect corresponds to a device, and each device can contain multiple defects. DYB = {S1 = Q11, Q12, ..., S2 = Q21, Q22, ..., ...;

[0071] Based on the device description set S, establish the correspondence table DYB between the device and its attributes S , characterizing the characteristics of the current device, among which DYB S ={T s1 = {Z 11 , Z 12 , …}, T s2 = {Z 21 , Z 22 , …}, …};

[0072] Based on the defect description set Q, establish the corresponding table DYB between the equipment and its attributes Q , characterizing the characteristics of the current device; among them, DYB Q ={T q1 = {Y 11 , Y 12 , …}, T q2 = {Y 21 , Y 22 , …}, …}.

[0073] S3. Build a self-attention device fault detection model with text and image joint learning capabilities;

[0074] See also Figure 2 ,The workflow of the self-attention device fault detection model with the ability of joint learning of text and images is:

[0075] S3.1. For the input text information T, extract features through the text encoding network to obtain the text feature matrix F T ;

[0076] The text encoding network is composed of 12 Transformer encoders in series.i |i∈{1,2,3,4,5,6,7,8,9,10,11,12}}; each encoder e i ∈E is composed of several submodules with the same structure in series, each of which contains a multi-head self-attention module, a residual connection, a layer normalization, and a fully connected layer; each word in the input text sequence is first randomly initialized to obtain the corresponding initial word embedding ν0, which is then embedded in v by superimposing the position p and text embedding v s Then input the text encoding network to obtain the text feature F T ,

[0077] F T =TextEncoder(T)

[0078] S3.2, the input image information is extracted through the convolutional neural network CNN to obtain the image feature matrix F I ;

[0079] The input image information is subjected to a convolutional neural network (CNN) to extract features and obtain an image feature matrix F I , specifically:

[0080] Input image I∈R 3×1280×1280 The image blocks are obtained by convolution with several convolution kernels, and each image block is embedded with v by superimposing the position p After that, it is input into the image encoding network E′ which is similar to the text encoding network structure to obtain the final image feature F I ,in,

[0081] F I =CNN(I)

[0082] S3.3. Based on image feature F I And the text feature matrix F T , feature fusion is performed through an improved cross self-attention mechanism network to obtain text features fused with image information and image features fused with text information;

[0083] The improved cross self-attention mechanism network includes a multi-head self-attention module, a cross-image-text joint attention module and a cross-text-image joint attention module; the word embedding v1 is calculated by the multi-head self-attention module to obtain the key vector K v Sum value vector V v The image feature F1 is calculated by the multi-head self-attention module to obtain the query vector Q F , the cross-image-text joint attention module is implemented by formula F v =softmax(Q F K v )Vv , get the image features fused with text information The cross-text-image joint attention module is implemented by formula v F =softmax(Q v K F )V F Get text features that fuse image information

[0084] Formula F v =softmax(Q F K v )V v The detailed calculation process is as follows:

[0085] Q F =W q F I

[0086] Among them, the query vector Q F Through the image feature F I With the weight matrix W q Calculated.

[0087] K v =W k F T

[0088] Query vector K v Through the text feature matrix F T With the weight matrix W k Calculated.

[0089] V v =W v F T

[0090] Query vector V v From the text feature matrix F T With the weight matrix W v Calculated.

[0091] For the formula v F =softmax(Q v K F )V F The detailed calculation process is as follows:

[0092] Q v =W q F T

[0093] K F =W k F I

[0094] VF =W v F I

[0095] Among them, the query vector Q v Through the text feature F T With the weight matrix W q Calculated, K F Through the image feature F I With the weight matrix W k Calculated, V F Through the image feature F I With the weight matrix W v Calculated.

[0096] S3.4, based on the text features of the fused image information and the image features of the fused text information, the final image features are obtained through the text feature layer attention mechanism;

[0097] Based on the formula M=F v ×v F , get the similarity measurement matrix M between image features and text features, take the maximum value of each row vector in M ​​as the corresponding image feature and the correlation of the text feature most related to the image feature, sort the correlation from large to small and select the positions of the image features corresponding to the first several values ​​as the final selected image feature F final .

[0098] S3.5, decoding the final image features based on the image attention module and the text attention module to obtain output features;

[0099] Based on the final selected image feature F final Input to the image attention module, the image attention module uses the formula Update the query vector Q1, and the text attention module is calculated by formula The query vector Q2 is updated; the output feature is the query vector Q2.

[0100] For the formula The detailed calculation process is as follows:

[0101]

[0102] Among them, F final represents the selected image features, W q , W k , W v represents the corresponding weight matrix, F v Image features fused with text information.

[0103] For the formula The detailed calculation process is as follows:

[0104]

[0105] Among them, W q , W k , W v represents the corresponding weight matrix, Represents text features that fuse image information.

[0106] S3.6, based on the output features, the position of the prediction box is obtained through the fully connected layer;

[0107] The output feature is passed through the fully connected layer to obtain the position of the prediction box, specifically: the output feature Q2 is operated through the fully connected layer FFN1 to obtain the correlation matrix Wherein, d1 and d2 represent the number of prediction boxes output by the model and the number of input prompt words respectively. The value of each position in O represents the correlation between the prediction box and the prompt word corresponding to the position. The prediction prompt word with the largest value in the correlation matrix O is used as the category output; the output feature Q2 is calculated by the fully connected layer FFN2 to obtain the position of the prediction box (x1, y1, x2, y2);

[0108] S3.7. The self-attention device fault detection model is tuned based on the multi-task loss function. When the loss function is minimized, the optimal self-attention device fault detection model is obtained.

[0109] For the regression of the prediction box, smooth L1 loss is used to update the model parameters; for the correlation between the prediction box and the input prompt word, focal loss is used to update the model parameters.

[0110] S4. Based on the correspondence table between equipment type, equipment defects and attributes, the labeled power equipment image is input into the self-attention equipment fault detection model for training, and basic tasks are set until the accuracy of the basic tasks reaches the specified threshold to complete the power equipment defect detection.

[0111] Based on the annotated equipment and defect data, the self-attention equipment fault detection model is trained on a single type, where the input text of each picture is "find" plus the equipment name or "find" plus the defect name. If there are multiple types of defects in the picture, only one type of equipment or defect is displayed each time. If there are multiple types of equipment or defects in the picture, they are trained separately.

[0112] Multi-type training is performed on the self-attention device fault detection model, where the input text of each image is "find" plus all the devices and defect names in the image, and different targets are separated by ",";

[0113] For target devices and defects, use the target descriptive words in random combination for training; the input text is "find" plus the device or defect with the descriptive words;

[0114] For images containing multiple target devices and defects, the keywords "include" and "each" are used for training. This part trains the model to identify the specified device or defect capability on a certain associated device based on the device type affiliation.

[0115] like Figure 3 The method is described in detail as shown below:

[0116] S1. For the collected power equipment images, use a unified annotation method to annotate the content with detection frames and text information;

[0117] The labeling of equipment and defects in images is divided into the following steps: determining the equipment and defect types to be labeled, establishing a descriptive vocabulary for all equipment and defects, and labeling all equipment and defects in the images.

[0118] S11. Insulators and related defects (such as breakage, tilt, and dirt), binding wires and related defects (irregular binding, no binding), small-sized hardware and related defects (hardware rust, hardware falling off), and towers and related defects (transverse and longitudinal cracks) in the power distribution field are selected as the equipment and defect types that need to be labeled. Images are collected according to the selected equipment and defect types to form an image dataset D.

[0119] S12, for the images I = {I1, I2, ...} in the data set D, establish a device description set S = {S1, S2, ..., S m} and defect description set Q = {Q1, Q2, ..., Q n}, where the device description S m is a Chinese label corresponding to the label name of all equipment in image I (such as insulators, binding wires, pins, poles, etc., m is the total number of equipment types included in the dataset D), expressed as a single word; defect description Q n is a Chinese label containing the label name of the equipment defect in image I (such as large area damage, yellow severe rust, longitudinal crack, and n is the total number of defect types included in the data set D).

[0120] S13. For the device description set S and the defect description set Q, a device and defect correspondence table DYB is established. Each type of defect corresponds to a device. Each device can contain multiple defects. DYB = {S1 = {Q 11 , Q 12 , ...}, S2 = {Q 21 , Q 22 , …}, …}.

[0121] S14. For the device description set S, establish a corresponding table DYB between the device and its attributes S , characterizing the characteristics of the current equipment (such as the characteristics of porcelain bottle insulators include: ring, brown, porcelain, etc.). S ={T s1 = {Z 11 , Z 12 , …}, T s2 = {Z 21 , Z 22 , …}, …}.

[0122] S15. For the defect description set Q, establish a corresponding table DYB between the equipment and its attributes Q , characterizing the characteristics of the current equipment (such as the characteristics of porcelain bottle insulators include: ring, brown, porcelain, etc.). Q ={T q1 = {Y 11 , Y 12 , …}, T q2 = {Y 21 , Y 22 , …}, …}.

[0123] S16. Annotate the targets in each image according to the device description set S and the defect description set Q. The annotation results are saved as an xml file with the same name as the image. The file contains the label name and annotation box coordinates (x, y, w, h) of the target of interest in the corresponding image, which respectively represent the horizontal and vertical coordinates of the upper left corner and lower right corner of the annotation box.

[0124] S2. Build a self-attention device fault detection model with the ability to jointly learn text and images, and design a targeted loss function;

[0125] like Figure 4 As shown in the figure, a self-attention device fault detection model with the ability to jointly learn text and images is constructed, which mainly includes the following structure:

[0126] S21. Extract features from the input text information T through the text encoding network to obtain the text feature matrix F T Specifically, the text encoding network consists of 12 Transformer encoders connected in series. i |i∈{1,2,3,4,5,6,7,8,9,10,11,12}},each encoder e i ∈E is composed of several submodules with the same structure in series, each of which contains a multi-head self-attention module, residual connection, layer normalization, and a fully connected layer. Each word in the input text sequence is first randomly initialized to obtain the corresponding initial word embedding ν0, which is then embedded in v by superimposing the positionp and text embedding v s Then input the text encoding network to obtain the text feature F T .

[0127] F T =TextEncoder(T)

[0128] S22, extract features from the input image information through a convolutional neural network (CNN) to obtain the feature matrix F I Input image I∈R 3×1280×1280 First, it is convolved with 512 convolution kernels of size 80×80 and stride 80 to obtain image blocks, each with a dimension of 512; each image block is embedded by superimposing the position and then input into the image encoding network E′ which has a similar structure to the above text encoding network to obtain the final image feature F I .

[0129] F I =CNN(I)

[0130] S23. Update and optimize the prompt word vector through the environmental information fusion module.

[0131] S23, the feature information constructed by both parties is fused using an improved cross self-attention mechanism network. The improved cross self-attention mechanism network consists of a multi-head self-attention module, a cross-image-text joint attention module, and a cross-text-image joint attention module. The word embedding v1 is calculated by the multi-head self-attention module to obtain the key vector K v Sum value vector V v The image feature F1 is calculated by the multi-head self-attention module to obtain the query vector Q F , the cross-image-text joint attention module is implemented by formula F v =softmax(Q F K v )V v Calculate the image features that incorporate text information The cross-text-image joint attention module is implemented by formula v F =softmax(Q v K F )V F Calculate the text features that integrate image information

[0132] S24, through the text feature layer attention mechanism, the image features are adjusted between layers. First, through the formula M = F v ×v FThe similarity measurement matrix M between the image features and the text features is calculated, and the maximum value of each row vector in M ​​is taken as the correlation between the corresponding image feature and the text feature most related to the image feature. The image features corresponding to the first 512 values ​​are sorted from large to small and the positions are selected as the final selected image features F. final .

[0133] S25, feature expansion is performed by a deconvolution decoder. The decoder is composed of an image attention module and a text attention module. The image attention module is calculated by the formula Update the query vector Q1, and the text attention module is calculated by formula The query vector Q2 is updated and obtained. Q2 is the output of the decoder module.

[0134] S26, connect multiple task branches to make predictions. The output feature Q2 of the decoder is calculated by the fully connected layer FFN1 to obtain the correlation matrix Where d1 and d2 represent the number of prediction boxes output by the model and the number of input prompt words respectively. The numerical value of each position in O represents the correlation between the prediction box and the prompt word corresponding to the position. The prediction prompt word with the largest value in the matrix O is used as the category output. The output feature Q2 of the decoder is calculated through the fully connected layer FFN2 to obtain the position of the prediction box (x1, y1, x2, y2).

[0135] S27. Perform model optimization through multi-task loss function. For the regression of the prediction box, smooth L1 loss is used to update the model parameters; for the correlation between the prediction box and the input prompt word, focal loss is used to update the model parameters.

[0136] S3. For the data set generated by S1, use the large model structure constructed by S2 and the automatic prompt word generation method to perform multiple rounds of training until the accuracy of each basic task reaches the specified threshold.

[0137] The text prompt cross-modal model training method mainly includes the following steps:

[0138] S31. Using the equipment and defect data annotated in S16, the model is first trained on a single type, where the input text of each picture is "find" plus the equipment name or "find" plus the defect name. If there are multiple types of defects in the picture, only one type of equipment or defect is displayed each time. If there are multiple types of equipment or defects in the picture, they are trained separately.

[0139] S32. Perform multi-type training on the model, where the input text of each picture is "find" plus all the equipment and defect names in the picture, and different targets are separated by ",".

[0140] S33, for the target device and defect, use the randomly combined target description words for training. The input text is "find" plus the device or defect with the description word.

[0141] S34. For images containing multiple target devices and defects, use the keywords "include" and "each" for training. This part trains the model to identify the specified device or defect capability on a certain associated device based on the device type affiliation.

[0142] Example 2

[0143] The present invention proposes a power equipment defect detection system based on a multi-modal large model, such as Figure 5 As shown, including:

[0144] An acquisition module, wherein the acquisition module analyzes the collected power equipment image, acquires the equipment type and equipment defects in the power equipment image, and marks the equipment type and equipment defects in the power equipment image;

[0145] Based on the collected power equipment images, an image dataset D is constructed, and the images I = {I1, I2, ...} in the image dataset D are used to establish a device description set S = {S1, S2, ..., S m} and defect description set Q = {Q1, Q2, ..., Q n}, where the device description S m is a Chinese label, corresponding to the label name of all equipment in image I, expressed as a single word, such as: insulator, pole transformer, tower, crossarm, m is the total number of equipment types included in the dataset D; defect description Q n is a Chinese label containing the label name of the equipment defect in the corresponding image I, such as: large area damage, yellow severe rust, longitudinal crack, and n is the total number of defect types contained in the data set D.

[0146] The equipment types and equipment defects in the power equipment image are annotated as follows: based on the equipment description set S = {S1, S2, ..., S m} and defect description set Q = {Q1, Q2, ..., Q n}, annotate the targets in each image and save the annotation results, which include the label name and annotation box coordinates (x, y, w, h) of the target of interest in the corresponding image, which respectively represent the horizontal and vertical coordinates of the upper left corner and lower right corner of the annotation box.

[0147] A first construction module, wherein the first construction module determines a correspondence between a device defect and a device based on a device type and a device defect, and constructs a correspondence table between a device type and a device defect and an attribute respectively;

[0148] Based on the device description set S and the defect description set Q, a device and defect correspondence table DYB is established. Each type of defect corresponds to a device, and each device can contain multiple defects. DYB = {S1 = Q11, Q12, ..., S2 = Q21, Q22, ..., ...;

[0149] Based on the device description set S, establish the correspondence table DYB between the device and its attributes S , characterizing the characteristics of the current device, among which DYB S ={T s1 = {Z 11 , Z 12 , …}, T s2 = {Z 21 , Z 22 , …}, …};

[0150] Based on the defect description set Q, establish the correspondence table DYB between the equipment and its attributes Q , characterizing the characteristics of the current device; among them, DYB Q ={T q1 = {Y 11 , Y 12 , …}, T q2 = {Y 21 , Y 22 , …}, …}.

[0151] A second building block, wherein the second building block builds a self-attention device fault detection model with text and image joint learning capability;

[0152] The workflow of the self-attention device fault detection model with the ability of joint learning of text and images is as follows:

[0153] For the input text information T, the features are extracted through the text encoding network to obtain the text feature matrix F T ;

[0154] The text encoding network is composed of 12 Transformer encoders in series. i |i∈{1,2,3,4,5,6,7,8,9,10,11,12}}; each encoder e i ∈E is composed of several submodules with the same structure in series, each of which contains a multi-head self-attention module, a residual connection, a layer normalization, and a fully connected layer; each word in the input text sequence is first randomly initialized to obtain the corresponding initial word embedding ν0, which is then embedded in v by superimposing the position p and text embedding v s Then input the text encoding network to obtain the text feature F T ,

[0155] F T =TextEncoder(T)

[0156] The input image information is extracted through the convolutional neural network CNN to obtain the image feature matrix F I ;

[0157] The input image information is subjected to a convolutional neural network (CNN) to extract features and obtain an image feature matrix F I , specifically:

[0158] Input image I∈R 3×1280×1280 The image blocks are obtained by convolution with several convolution kernels, and each image block is embedded with v by superimposing the position p After that, it is input into the image encoding network E′ which is similar to the text encoding network structure to obtain the final image feature F I ,in,

[0159] F I =CNN(I)

[0160] Based on the image feature F I And the text feature matrix F T , feature fusion is performed through an improved cross self-attention mechanism network to obtain text features fused with image information and image features fused with text information;

[0161] The improved cross self-attention mechanism network includes a multi-head self-attention module, a cross-image-text joint attention module and a cross-text-image joint attention module; the word embedding v1 is calculated by the multi-head self-attention module to obtain the key vector K v Sum value vector V v The image feature F1 is calculated by the multi-head self-attention module to obtain the query vector Q F , the cross-image-text joint attention module is implemented by formula F v =softmax(Q F K v )V v , get the image features fused with text information The cross-text-image joint attention module is implemented by formula v F =softtmax(Q v K F )V F Get text features that fuse image information

[0162] Formula F v =softmax(Q F K v )V vThe detailed calculation process is as follows:

[0163] Q F =W q F I

[0164] Among them, the query vector Q F Through the image feature F I With the weight matrix W q Calculated.

[0165] K v =W k F T

[0166] Query vector K v Through the text feature matrix F T With the weight matrix W k Calculated.

[0167] V v =W v F T

[0168] Query vector V v From the text feature matrix F T With the weight matrix W v Calculated.

[0169] For the formula v F =softmax(Q v K F )V F The detailed calculation process is as follows:

[0170] Q v =W q F T

[0171] K F =W k F I

[0172] V F =W v F I

[0173] Among them, the query vector Q v Through the text feature F T With the weight matrix W q Calculated, K F Through the image feature F I With the weight matrix W k Calculated, V F Through the image feature F I With the weight matrix W vCalculated.

[0174] Based on the text features fused with image information and the image features fused with text information, the final image features are obtained through the attention mechanism of the text feature layer;

[0175] Based on the formula M=F v ×v F , get the similarity measurement matrix M between image features and text features, take the maximum value of each row vector in M ​​as the corresponding image feature and the correlation of the text feature most related to the image feature, sort the correlation from large to small and select the positions of the image features corresponding to the first several values ​​as the final selected image feature F final .

[0176] Decode the final image features based on the image attention module and the text attention module to obtain the output features;

[0177] Based on the final selected image feature F final Input to the image attention module, the image attention module uses the formula Update the query vector Q1, and the text attention module is calculated by formula The query vector Q2 is obtained by updating; the output feature is the query vector Q2.

[0178] For the formula The detailed calculation process is as follows:

[0179]

[0180]

[0181] Among them, F final represents the selected image features, W q , W k , W v represents the corresponding weight matrix, F v Image features fused with text information.

[0182] For the formula The detailed calculation process is as follows:

[0183]

[0184] Among them, W q , W k , W v represents the corresponding weight matrix, Represents text features that fuse image information.

[0185] Based on the output features, the position of the prediction box is obtained through the fully connected layer;

[0186] The output feature is passed through the fully connected layer to obtain the position of the prediction box, specifically: the output feature Q2 is operated through the fully connected layer FFN1 to obtain the correlation matrix Wherein, d1 and d2 represent the number of prediction boxes output by the model and the number of input prompt words respectively. The value of each position in O represents the correlation between the prediction box and the prompt word corresponding to the position. The prediction prompt word with the largest value in the correlation matrix O is used as the category output; the output feature Q2 is calculated by the fully connected layer FFN2 to obtain the position of the prediction box (x1, y1, x2, y2);

[0187] The self-attention device fault detection model is tuned based on the multi-task loss function. When the loss function is minimized, the optimal self-attention device fault detection model is obtained.

[0188] For the regression of the prediction box, smooth L1 loss is used to update the model parameters; for the correlation between the prediction box and the input prompt word, focal loss is used to update the model parameters.

[0189] A training module, based on a correspondence table between equipment type and equipment defects and attributes, inputs the annotated power equipment images into a self-attention equipment fault detection model for training, and sets basic tasks until the accuracy of the basic tasks reaches a specified threshold to complete the power equipment defect detection.

[0190] Based on the annotated equipment and defect data, the self-attention equipment fault detection model is trained on a single type, where the input text of each picture is "find" plus the equipment name or "find" plus the defect name. If there are multiple types of defects in the picture, only one type of equipment or defect is displayed each time. If there are multiple types of equipment or defects in the picture, they are trained separately.

[0191] Multi-type training is performed on the self-attention device fault detection model, where the input text of each image is "find" plus all the devices and defect names in the image, and different targets are separated by ",";

[0192] For target devices and defects, use the target descriptive words in random combination for training; the input text is "find" plus the device or defect with the descriptive words;

[0193] For images containing multiple target devices and defects, the keywords "include" and "each" are used for training. This part trains the model to identify the specified device or defect capability on a certain associated device based on the device type affiliation.

[0194] Example 3

[0195] See also Figure 6As shown, the present invention also provides an electronic device 100 for an electric power equipment defect detection method based on a multimodal large model; the electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on the at least one processor 102, and at least one communication bus 104.

[0196] The memory 101 can be used to store the computer program 103, and the processor 102 implements the steps of the method for detecting defects of power equipment based on a multimodal large model described in Example 1 by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101. The memory 101 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data (such as audio data) created according to the use of the electronic device 100, etc. In addition, the memory 101 can include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0197] The at least one processor 102 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 102 may be a microprocessor or any conventional processor, etc. The processor 102 is the control center of the electronic device 100, and uses various interfaces and lines to connect various parts of the entire electronic device 100.

[0198] The memory 101 in the electronic device 100 stores a plurality of instructions to implement a method for detecting defects in electric power equipment based on a multi-modal large model, and the processor 102 can execute the plurality of instructions to implement:

[0199] Analyze the collected power equipment images, obtain the equipment types and equipment defects in the power equipment images, and mark the equipment types and equipment defects in the power equipment images;

[0200] Based on the equipment type and equipment defect, determine the correspondence between the equipment defect and the equipment, and construct the correspondence table between the equipment type and equipment defect and the attribute respectively;

[0201] Build a self-attention device fault detection model with the ability to jointly learn text and images;

[0202] Based on the correspondence table of equipment types, equipment defects and attributes, the annotated power equipment images are input into the self-attention equipment fault detection model for training, and basic tasks are set until the accuracy of the basic tasks reaches the specified threshold to complete the power equipment defect detection.

[0203] Example 4

[0204] If the module / unit integrated in the electronic device 100 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory and read-only memory (ROM, Read-Only Memory).

[0205] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0206] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0207] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0208] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0209] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for detecting defects in power equipment based on a multimodal large model, characterized in that: include: Analyze the collected power equipment images, obtain the equipment types and equipment defects in the power equipment images, and mark the equipment types and equipment defects in the power equipment images; Based on the equipment type and equipment defect, determine the correspondence between the equipment defect and the equipment, and construct the correspondence table between the equipment type and equipment defect and the attribute respectively; Build a self-attention device fault detection model with the ability to jointly learn text and images; Based on the correspondence table of equipment types, equipment defects and attributes, the annotated power equipment images are input into the self-attention equipment fault detection model for training, and basic tasks are set until the accuracy of the basic tasks reaches the specified threshold to complete the power equipment defect detection.

2. The method for detecting defects in electric power equipment based on a multimodal large model according to claim 1, characterized in that: The collected power equipment image is analyzed to obtain the equipment type and equipment defects in the power equipment image, specifically: Based on the collected power equipment images, an image dataset D is constructed, and the images I = {I1, I2, ...} in the image dataset D are respectively established as device description sets S = {S1, S2, ..., S m } and defect description set Q = {Q1,Q2,…,Q n }, where S m is the Chinese label of the device description, m is the total number of device types included in the dataset D; Q n is the Chinese label of the defect description, and n is the total number of defect types contained in the dataset D; Annotate the equipment types and equipment defects in the power equipment images, specifically: Based on the device description set S = {S1, S2, ..., S m } and defect description set Q = {Q1,Q2,…,Q n }, annotate the targets in each image and save the annotation results, which include the label name and annotation box coordinates (x, y, w, h) of the target of interest in the corresponding image, which respectively represent the horizontal and vertical coordinates of the upper left corner and lower right corner of the annotation box.

3. The method for detecting defects in electric power equipment based on a multimodal large model according to claim 1, characterized in that: Based on the device type and device defect, the corresponding relationship between the device defect and the device is determined, and a corresponding table between the device type and the device defect and the attribute is constructed respectively, specifically: Based on the device description set S and the defect description set Q, a device and defect correspondence table DYB is established. Each type of defect corresponds to a device, and each device can contain multiple defects belonging to it. DYB = {S1 = {Q 11 ,Q 12 ,…},S2={Q 21 ,Q 22 ,…},…}; Based on the device description set S, establish the correspondence table DYB between the device and its attributes S , characterizing the characteristics of the current device, among which DYB S ={T s1 = {Z 11 ,Z 12 ,…},T s2 = {Z 21 ,Z 22 ,…},…}; Based on the defect description set Q, establish the correspondence table DYB between the equipment and its attributes Q , characterizing the characteristics of the current device; among them, DYB Q ={T q1 = {Y 11 ,Y 12 ,…},T q2 = {Y 21 ,Y 22 ,…},…}.

4. The method for detecting defects in electric power equipment based on a multimodal large model according to claim 1, characterized in that: The workflow of the self-attention device fault detection model with the ability of joint learning of text and images is as follows: For the input text information T, the features are extracted through the text encoding network to obtain the text feature matrix F T ; The input image information is extracted through the convolutional neural network CNN to obtain the image feature matrix F I ; Based on the image feature F I And the text feature matrix F T , feature fusion is performed through an improved cross self-attention mechanism network to obtain text features fused with image information and image features fused with text information; Based on the text features fused with image information and the image features fused with text information, the final image features are obtained through the attention mechanism of the text feature layer; Decode the final image features based on the image attention module and the text attention module to obtain the output features; Based on the output features, the position of the prediction box is obtained through the fully connected layer; The self-attention device fault detection model is tuned based on the multi-task loss function. When the loss function is minimized, the optimal self-attention device fault detection model is obtained.

5. The method for detecting defects in electric power equipment based on a multi-modal large model according to claim 4 is characterized in that: The input text information T is extracted through the text encoding network to obtain the text feature matrix F T , specifically: The text encoding network is composed of 12 Transformer encoders in series. i |i∈{1,2,3,4,5,6,7,8,9,10,11,12}}; each encoder e i ∈E is composed of several submodules with the same structure in series, each of which contains a multi-head self-attention module, a residual connection, a layer normalization, and a fully connected layer; each word in the input text sequence is first randomly initialized to obtain the corresponding initial word embedding v0, and v0 is embedded in v by superimposing the position p and text embedding v s Then input the text encoding network to obtain the text feature F T , F T =TextEncoder(T) The input image information is subjected to a convolutional neural network (CNN) to extract features and obtain an image feature matrix F I , specifically: Input image I∈R 3×1280×1280 The image blocks are obtained by convolution with several convolution kernels, and each image block is embedded with v by superimposing the position p After that, it is input into the image encoding network E' which is similar to the text encoding network structure to obtain the final image feature F I , where F I =CNN(I).

6. The method for detecting defects in electric power equipment based on a multi-modal large model according to claim 4, characterized in that: The image feature F I And the text feature matrix F T , feature fusion is performed through the improved cross self-attention mechanism network to obtain text features fused with image information and image features fused with text information, specifically: The improved cross self-attention mechanism network includes a multi-head self-attention module, a cross-image-text joint attention module and a cross-text-image joint attention module; the word embedding v1 is calculated by the multi-head self-attention module to obtain the key vector K v Sum value vector V v The image feature F1 is calculated by the multi-head self-attention module to obtain the query vector Q F , the cross-image-text joint attention module is implemented by formula F v =softmax(Q F K v )V v , get the image features fused with text information The cross-text-image joint attention module is implemented by formula v F =softmax(Q v K F )V F Get text features that fuse image information The text features based on the fused image information and the image features based on the fused text information are used to obtain the final image features through the text feature layer attention mechanism, specifically: Based on M=F v ×v F , get the similarity measurement matrix M between image features and text features, take the maximum value of each row vector in M ​​as the corresponding image feature and the correlation of the text feature most related to the image feature, sort the correlation from large to small and select the positions of the image features corresponding to the first several values ​​as the final selected image feature F final .

7. The method for detecting defects in electric power equipment based on a multi-modal large model according to claim 4 is characterized in that: The final image features are decoded based on the image attention module and the text attention module to obtain output features, which are specifically: Based on the final selected image feature F final Input to the image attention module, the image attention module uses the formula Update the query vector Q1, and the text attention module is calculated by formula Update to obtain query vector Q2; the output feature is query vector Q2; The output feature is passed through the fully connected layer to obtain the position of the prediction box, specifically: the output feature Q2 is operated through the fully connected layer FFN1 to obtain the correlation matrix Wherein, d1 and d2 represent the number of prediction boxes output by the model and the number of input prompt words respectively, and the numerical value of each position in O represents the correlation between the prediction box and the prompt word corresponding to the position; the output feature Q2 is calculated by the fully connected layer FFN2 to obtain the position of the prediction box (x1, y1, x2, y2); The self-attention device fault detection model is tuned based on the multi-task loss function, specifically: for the regression of the prediction box, the smooth L1 loss is used to update the model parameters; for the correlation between the prediction box and the input prompt word, the focalloss loss is used to update the model parameters.

8. The method for detecting defects in electric power equipment based on a multimodal large model according to claim 1, characterized in that: The basic tasks are set until the accuracy of the basic tasks reaches a specified threshold and the power equipment defect detection is completed, specifically: Based on the annotated equipment and defect data, the self-attention equipment fault detection model is trained on a single type, where the input text of each image is "find" plus the equipment name or "find" plus the defect name. If there are multiple types of defects in the image, only one type of equipment or defect is displayed each time. If there are multiple types of equipment or defects in the image, they are trained separately. Multi-type training is performed on the self-attention device fault detection model, where the input text of each image is "find" plus all the devices and defect names in the image, and different targets are separated by "". For target devices and defects, use the target descriptive words in random combination for training; the input text is "find" plus the device or defect with the descriptive words; For images containing multiple target devices and defects, the keywords "include" and "each" are used for training. This part trains the model to identify the specified device or defect capability on a certain associated device based on the device type affiliation.

9. A power equipment defect detection system based on a multi-modal large model, characterized in that: include: An acquisition module, wherein the acquisition module analyzes the collected power equipment image, acquires the equipment type and equipment defects in the power equipment image, and marks the equipment type and equipment defects in the power equipment image; A first construction module, wherein the first construction module determines a correspondence between a device defect and a device based on a device type and a device defect, and constructs a correspondence table between a device type and a device defect and an attribute respectively; A second building block, wherein the second building block builds a self-attention device fault detection model with text and image joint learning capability; A training module, based on a correspondence table between equipment type and equipment defects and attributes, inputs the annotated power equipment images into a self-attention equipment fault detection model for training, and sets basic tasks until the accuracy of the basic tasks reaches a specified threshold to complete the power equipment defect detection.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for detecting defects in electric power equipment based on a multi-modal large model as claimed in any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for detecting defects in electric power equipment based on a multi-modal large model as claimed in any one of claims 1 to 8 are implemented.

Images