A domain knowledge-driven multi-modal power inspection method, device and system

Abstract

The present application discloses a domain knowledge-driven multimodal power inspection method, device and system. The present application belongs to the field of power transmission and transformation line detection technology. The method includes: using a lightweight target detection network deployed at the edge to perform preliminary detection and classification of the image data set of the target to be detected, filtering out defect-free samples, and transmitting defect samples containing location information and category information to the cloud; classifying defect samples, and marking the defect sample data as the data format required for subsequent defect detection; judging whether it is a small target defect; if so, performing small target defect detection based on a large language model and a multimodal large model, otherwise using a pre-trained YOLOv8 model for defect detection. The present application adopts a combination of large and small models, and uses a multimodal large model to achieve small target defect detection, which can significantly improve detection efficiency and accuracy.

Description

Technical Field

[0001] The present application belongs to the technical field of power transmission and transformation line detection, and specifically relates to a multi-modal power inspection method, device and system driven by domain knowledge. Background Art

[0002] There are hundreds of defects in power transmission and transformation lines, and the number and form of each defect vary greatly. Traditional manual inspection methods have problems such as low efficiency, low detection accuracy, and high cost; therefore, the use of drone technology for inspection and combined with deep learning algorithms for intelligent fault detection of power transmission lines has become an indispensable and important technical means.

[0003] However, power inspections are carried out using drones, and the amount of data is very large. If all the collected data is sent to the cloud for detection using the model, there will be great bandwidth pressure. If the model is deployed at the edge, a lot of computing power will be consumed, causing unnecessary losses.

[0004] In addition, existing deep learning models mainly distinguish and locate object categories through surface features such as color, shape, and texture, without considering the relative position information between objects. At the same time, these models are difficult to accurately distinguish for small target devices with high appearance similarity, resulting in missed detection or false detection of small target defects on power transmission and transformation lines. However, in specific parts of power transmission and transformation equipment, there is usually a fixed spatial layout between these small target components. This positional relationship and co-occurrence form are also important references for the detection of power transmission and transformation component defects. How to make full use of the above position and form information to improve the accuracy of small target defect detection in power transmission and transformation lines is an urgent problem to be solved. Summary of the invention

[0005] Due to the wide variety of power transmission and transformation line defects and complex installation scenarios, the same components are installed differently in different locations. Therefore, the types of power transmission and transformation line defects are more diverse and complex. The use of existing intelligent detection algorithms is prone to problems such as false detection or missed detection. To address this, the present application proposes a domain knowledge-driven multimodal power inspection method, device, and system. The present application uses a combination of large and small models to deploy a lightweight model (i.e., a small model) at the edge to perform preliminary detection of defects, and transmits images of small targets that are difficult to detect to the cloud for detection using a large model. At the same time, the final detection results are generated in a multimodal form, which can significantly improve detection efficiency and accuracy.

[0006] This application is implemented through the following technical solutions:

[0007] A domain knowledge-driven multi-modal power inspection method, the multi-modal power inspection method comprising:

[0008] Receive defect samples containing location information and category information transmitted by the edge, where the defect samples are obtained by performing preliminary detection and classification on an image data set of the target to be detected using a lightweight target detection network deployed on the edge and filtering out defect-free samples;

[0009] Classifying the defect samples according to a preset target size, and marking the defect sample data into a data format required for subsequent defect detection;

[0010] According to the classification result of the previous step, determine whether it is a small target defect. If so, perform small target defect detection based on the large language model and the multimodal large model. Otherwise, use the pre-trained YOLOv8 model for defect detection.

[0011] In some embodiments, the lightweight object detection network deployed on the edge performs preliminary detection and classification on the image data set of the target to be detected and filters out defect-free samples. The specific process includes:

[0012] Input the image into the ResNet model to obtain the classification results and feature maps;

[0013] Processing the feature map to preliminarily extract location information and category information of the defects;

[0014] The judgment statement is used to judge and eliminate the images that do not contain the preset target category and are determined to be defect-free.

[0015] In some implementations, marking the defect sample data into a data format required for subsequent defect detection specifically includes:

[0016] The defect samples are labeled as whole parts and defective parts respectively, and the labeled image data is preprocessed, including: adjusting the image size, normalizing the pixel values ​​of the image, converting the image into the format required by the YOLOv8 algorithm, and labeling the small target defect type images in the form of image-text pairs, in preparation for fine-tuning of the multimodal large model.

[0017] In some implementations, the small target defect detection based on the large language model and the multimodal large model specifically includes:

[0018] Based on the power operation procedures, equipment defect classification specifications and equipment defect elimination guidelines, the Llama3 language model is used to generate the defect generation environment;

[0019] The pre-trained CLIP model is used to perform category recognition on the image of the defect sample, and the output recognition result is compared with the defect generation environment generated by the Llama3 language model. If they match, it is determined that the image has the corresponding defect.

[0020] In some embodiments, the training process of the CLIP model specifically includes:

[0021] The description texts of various small targets on the transmission line are fed into the text encoder of the CLIP model to obtain the corresponding text features;

[0022] The historical image data with category labels is fed into the image encoder of the CLIP model to obtain image features;

[0023] The CLIP model is learned using the contrastive loss function, with the goal of minimizing the distance between relevant text features and image features, and maximizing the distance between irrelevant text features and image features, so that the image classification prediction result is the category corresponding to the text feature with the greatest similarity.

[0024] In some embodiments, the contrast loss function is expressed as:

[0025]

[0026] in, A , P , N represent target samples, positive samples and negative samples respectively. d ( A,P )and d ( A,N ) represent the distances between the target sample and the positive sample and the negative sample respectively. margin is the threshold value.

[0027] In some embodiments, the training process of the CLIP model further comprises:

[0028] The CLIP model is optimized using the Adam optimizer. This application fine-tunes the model by adding power field data and inserting an Adapter in each Transformer layer, so that the model understands the spatial layout between power transmission components and the co-occurrence form under normal conditions, helping the model to better adapt to defect detection in power scenarios.

[0029] In some implementations, the defect detection using the pre-trained YOLOv8 model specifically includes:

[0030] According to the position information, target positioning and classification are realized, and the target classified as a defective part is positioned and defect classified to obtain a defect position frame and a category name;

[0031] The defect location box and category name are passed to the subsequent classification network to determine whether it belongs to a certain category.

[0032] In a second aspect, the present application also proposes a domain knowledge-driven multi-modal power inspection device, the multi-modal power inspection device comprising:

[0033] A data receiving unit, the data receiving unit is used to receive defect samples containing location information and category information transmitted by the edge, the defect samples are obtained by performing preliminary detection and classification on the image data set of the target to be detected by using a lightweight target detection network deployed on the edge and filtering out defect-free samples;

[0034] A classification and labeling unit, wherein the classification and labeling unit classifies the defect samples according to a preset target size and labels the defect sample data into a data format required for subsequent defect detection;

[0035] And, a judgment unit, which judges whether it is a small target defect according to the classification result of the classification and labeling unit. If so, small target defect detection is performed according to the large language model and the multimodal large model; otherwise, a pre-trained YOLOv8 model is used for defect detection.

[0036] In a third aspect, the present application also proposes a domain knowledge-driven multi-modal power inspection system, the multi-modal power inspection system comprising:

[0037] The edge device is configured to perform the following operations:

[0038] Use the pre-deployed lightweight object detection network to perform preliminary detection and classification on the image set of the target to be detected, filter out defect-free samples, and transmit defective samples containing location information and category information to the cloud server;

[0039] and a cloud server, wherein the cloud server is configured to perform the following operations:

[0040] Classify defect samples according to preset target sizes and annotate defect sample data into the data format required for subsequent defect detection;

[0041] Determine whether it is a small target defect based on the classification result. If so, perform small target defect detection based on the large language model and the multimodal large model. Otherwise, use the pre-trained YOLOv8 model for defect detection.

[0042] 1. The present application proposes a domain knowledge-driven multimodal power inspection method, device and system, which first performs preliminary detection and classification of targets through a lightweight target detection network deployed at the edge, and eliminates the detected defect-free image data, thereby greatly reducing the amount of image data, thereby reducing the transmission bandwidth pressure, ensuring the real-time and security of cloud server processing, and improving detection efficiency.

[0043] 2. The domain knowledge-driven multimodal power inspection method, device and system proposed in the present application also divide the defect sample data into large targets and small targets through an algorithm deployed in the cloud, and adopt different defect detection algorithms for them, so that the training method of each defect is more suitable for the sample characteristics, avoiding the phenomenon that small target defects are integrated with the background and are difficult to detect. A large language model is used to enhance the defect retrieval capability, and the specific environment in which the defects occur is learned through the large language model. The defect range is further narrowed by combining the category parameters passed in at the edge, thereby improving the detection accuracy and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are used to provide a further understanding of the embodiments of the present application, constitute a part of the present application, and do not constitute a limitation on the embodiments of the present application. In the drawings:

[0045] Figure 1 This is a flow chart of the detection method of an embodiment of the present application;

[0046] Figure 2 A flowchart of performing preliminary detection on a lightweight detection network used in an embodiment of the present application;

[0047] Figure 3 This is a structural diagram of Llama3 used in the embodiments of the present application;

[0048] Figure 4 This is a schematic diagram of the CLIP model training principle used in the embodiment of the present application;

[0049] Figure 5 The recognition process of the CLIP model is adopted in the embodiment of the present application;

[0050] Figure 6 This is a principle block diagram of the inspection device of an embodiment of the present application;

[0051] Figure 7 This is a functional block diagram of the inspection system according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with examples and drawings. The illustrative implementation scheme of the present application and its description are only used to explain the present application and are not intended to limit the present application.

[0053] Embodiment: Since there are hundreds of types of defects in power transmission and transformation lines, and the number and form of each type of defect vary greatly, and the characteristics are different, the current intelligent defect detection technology is difficult to focus on the detection of various defects in the power transmission and transformation lines at the same time, and it is easy to misdetect or miss small target defects. Based on this, this embodiment proposes a domain knowledge-driven multimodal power inspection method. The detection method proposed in this embodiment first deploys a lightweight target detection network at the edge to perform preliminary detection and classification of the target, and then transmits the classification results and target location to the cloud, and then subdivides it into large targets and small targets to select different networks for defect detection, and uses a multimodal large model to realize small target defect detection, which can significantly improve detection efficiency and accuracy.

[0054] like Figure 1 As shown, the inspection method proposed in this embodiment specifically includes the following steps:

[0055] Step 100, receiving a defect sample containing location information and category information transmitted by an edge, wherein the defect sample is obtained by performing preliminary detection and classification on an image data set of a target to be detected using a lightweight target detection network deployed on the edge and filtering out defect-free samples;

[0056] Step 200, classifying defect samples according to preset target sizes, and marking defect sample data into a data format required for subsequent defect detection;

[0057] Step 300, judging whether it is a small target defect according to the classification result of step 200; if so, performing small target defect detection according to the large language model and the multimodal large model, otherwise using the pre-trained YOLOv8 model for defect detection.

[0058] Further, such as Figure 2 As shown, the defect sample generation process specifically includes the following sub-steps:

[0059] Step 101, loading the pre-trained ResNet model on the edge (edge ​​device);

[0060] Step 102, input the image into the ResNet model to obtain the classification result and feature map;

[0061] Step 103, processing the feature map to preliminarily extract defect location information (such as bounding box coordinates) and category information (such as defect category);

[0062] Step 104, using judgment statements to make judgments, remove images that do not have the preset target category and can be determined to be defect-free by edge detection, and transmit the images with location information and category information as defect samples to the cloud for further processing and analysis by the cloud.

[0063] It should be noted that the ResNet model architecture and training method are conventional technical means in this field and will not be elaborated here.

[0064] This embodiment loads a pre-trained lightweight model on the edge device for preliminary detection of images. On the one hand, the use of a lightweight model has little impact on the operation of the device. On the other hand, through preliminary processing on the edge, the identified defect-free images can be eliminated and no longer uploaded to the cloud for further detection. This can greatly reduce data samples and alleviate the pressure on data transmission bandwidth. At the same time, it ensures the real-time and security of cloud computing and improves detection efficiency.

[0065] Furthermore, the labeling process specifically includes: labeling the defective samples as whole parts and defective parts respectively, and preprocessing the labeled data set, which includes adjusting the image size, normalizing the pixel values ​​of the image using Min-Max to get between 0 and 1, and converting the image to the number required by YOLOv8: that is, the input image size is 640*640, and the labeling format is [target category, x_center, y_center, w, h], x_center is the x-coordinate of the center point of the target box, the ratio relative to the width of the picture; y_center is the y-coordinate of the center point of the target box, the ratio relative to the height of the picture; w is the width of the target box, the ratio relative to the width of the picture; h is the height of the target box, the ratio relative to the height of the picture. Labeling the small sample data into a format that can be used by the multimodal large model, labeling the picture as follows: the picture contains bolts and nuts (that is, the form of image-text pairs). Further, according to the large language model, defect sample images that may be difficult to detect are detected, which specifically includes:

[0066] Step 201, based on standard information such as power operation procedures, equipment defect grading specifications, and equipment defect elimination guidelines, the Llama3 language model is used to generate a defect generation environment. Specifically, the Llama3 language model can be input with the transmission line domain knowledge source, such as power operation procedures, equipment defect grading specifications, and equipment defect elimination guidelines, so that the model output is such as: there are bolts and there are active parts that may cause pins to rust or be missing. Among them, the Llama3 language model architecture is as follows Figure 3As shown in the figure, the input embedding layer uses the root mean square layer normalization operation Llama to improve the running efficiency. The rotation position encoding method is used. The position encoding information is added to Q and V before the self-attention operation of each layer. RoPE (rotation position encoding) uses the periodic characteristics of sine and cosine functions to encode information at different positions through sine and cosine functions of different frequencies. This enables the model to better understand the sequential relationship in the input sequence, not just through simple position index increments. The absolute position is encoded using a rotation matrix, and the relative position information is directly added to the self-attention operation. In the self-attention mechanism, the idea of ​​KVCache and Group Query Attention (GQA) group query attention is adopted. K stands for Key, V stands for Value, and Cache stands for cache. This technology stores the Key and Value vectors corresponding to historical tokens and reuses them directly in subsequent calculations instead of recalculating them every time, thereby increasing the access speed and reducing the database load. GQA divides queries (Q) into multiple groups, and the queries in each group share a common attention weight. This grouping method reduces the number of times the attention score is calculated, thereby reducing the computational complexity. Among them, the root mean square layer normalization removes the decentralization operation and only retains the scaling operation, which can improve the operation efficiency. The root mean square layer normalization formula is shown in formula (1), where a For input, d is the feature dimension, It is i The input is of dimensions. RMS is the root mean square calculation. For a i The normalized result is then scaled using learnable parameters and offset parameters To zoom and pan, For the i Dimensional output.

[0067] (1)

[0068] SiLU (SwiGLU) is a combination of GLU and Swish activation function. GLU introduces two different linear layers, one of which first passes through the sigmoid function, and its result is multiplied element-by-element with the output of the other linear layer as the final output. The formula is shown in the following formula (2):

[0069] (2)

[0070] In the formula, W , V , b , care the parameters of the two linear layers, σ ( xW + b ) as a gate, control xV +c output, σ is the sigmoid function, θ is a learnable variable. x For input.

[0071] Step 202, use the pre-trained CLIP model to perform category recognition on the defect sample image, and compare the output recognition result with the defect generation environment output by the Llama3 language model. If they match, it is determined that the defect sample image generates the corresponding defect. For example, if a bolt is recognized in the image and there is a moving part, it is determined that the image is a defect sample image that is difficult to detect.

[0072] Further, if Figure 4 As shown in the figure, the CLIP model is mainly composed of an image encoder, a text encoder, etc. The training process of the CLIP model specifically includes:

[0073] Step 301, the description texts of various small targets on the transmission line, such as description texts of hardware such as bolts, nuts, and pins, are sent to the text encoder to obtain corresponding text features. If the number of categories is n, then n text features will be obtained.

[0074] Step 302: Send the historical image data with category labels to an image encoder (Vit may be selected) to obtain image features.

[0075] Step 303, using the contrast loss function, the CLIP model is learned, the goal is to minimize the distance between relevant text features and image features, while maximizing the distance between irrelevant text features and image features, so that the image classification prediction result is the category corresponding to the text feature with the greatest similarity. Then, the trained CLIP model is used to identify the image to be predicted (i.e., the defect sample image uploaded by the edge end) and output its corresponding category, such as Figure 5 As shown, it is concluded through recognition that the image contains a shock-absorbing hammer.

[0076] The contrast loss function is shown in formula (3). Its goal is to make the distance between relevant text and image as small as possible, the distance between irrelevant samples and image as large as possible, and the difference between the two is greater than the threshold margin .

[0077] (3)

[0078] in, A , P , NThey represent the target (anchor) sample, the positive (positive) sample and the negative (negative) sample respectively. d ( A,P )and d ( A,N ) represent the distances between the target sample and the positive sample and the negative sample, respectively.

[0079] Optionally, this embodiment can also use the Adam optimizer to optimize the CLIP model, by inserting an Adapter into each transformer encoder layer of the image encoder model of the CLIP model, freezing all parameters in the image encoder model except the adapter layer, and using the Vit model with the adapter to train the task-specific dataset. The Adam optimization formula is shown in formula (4).

[0080] (4)

[0081] in, t is the number of steps in the current iteration. is a parameter. is the gradient of the current step. and are the exponential moving averages of the first-order moment estimate (mean) and the second-order moment estimate (uncentered variance), respectively. and They are the decay rates of the first-order moment estimate and the second-order moment estimate, respectively. They are usually taken as numbers close to 1 and are used to control the estimated weights of the first-order moment and the second-order moment of the gradient. and It is a bias-corrected estimate of the first and second moments. is the learning rate. It is a small constant added for numerical stability, usually with a very small value, such as .

[0082] Furthermore, the training process of the YOLOv8 model specifically includes:

[0083] The YOLOv8 model is trained using a historical image dataset with defect category labels, so that the trained YOLOv8 model can be used to quickly and accurately locate and classify targets: according to the location information, target location and classification are achieved, and the target classified as a defective part is located and defect classification is performed to obtain the defect location box and category name; the defect location box and category name are passed to the subsequent classification network to determine whether it is a certain category. Among them, the YOLOv8 model architecture can use a conventional network structure, which will not be described in detail here. The classification loss function used in the YOLOv8 model training process is expressed as:

[0084] (5)

[0085] in, N is the sample size, y i It is i The true label of each sample (0 or 1), p i is the label probability predicted by the model (in the range [0, 1]).

[0086] The regression loss function is used to locate the position of the loss. In this embodiment, CIOU+DFL loss can be used as the regression loss. DFL loss optimizes the label y Nearby two positions ( y i and y i+1 ) probability, making the network distribution focus near the label value. C IOU middle ρ represent b and b gt The Euclidean distance between b Parameters representing the coordinates of the prediction center, b gt The parameters representing the center of the ground-truth object bounding box, c 1 represents the minimum diagonal length of the circumscribed rectangle of the two rectangles. α Is a balance coefficient used to balance the relationship between the center point distance penalty term and the aspect ratio consistency penalty term. v represents the aspect ratio. w gt and h gt Represent the width and height of the prediction box respectively, w and h Represent the width and height of the real frame respectively, I OU is the ratio of the intersection to the union between two boxes. The regression loss function is expressed as:

[0087] (6)

[0088] Among them, in DFL, y i and y i+1 For label y The network focuses on the two predicted values ​​near the label. y i and yi+1 The corresponding probability is S i and S i+1 .

[0089] In another optional implementation, based on the diversity of categories, the WIOU loss function can be used instead of the CIOU loss function to better adapt to target boxes of different sizes and improve the overall effect.

[0090] Based on the same technical concept as above, this embodiment also proposes a multi-modal power inspection device driven by domain knowledge, such as Figure 6 As shown, the inspection device proposed in this embodiment includes:

[0091] A data receiving unit, which is used to receive defect samples containing location information and category information transmitted by the edge, wherein the defect samples are obtained by performing preliminary detection and classification on the image data set of the target to be detected using a lightweight target detection network deployed on the edge and filtering out defect-free samples;

[0092] A classification and labeling unit, which classifies the defect samples according to a preset target size and labels the defect sample data into a data format required for subsequent defect detection;

[0093] And, a judgment unit, which judges whether it is a small target defect according to the classification result of the classification and labeling unit. If so, small target defect detection is performed according to the large language model and the multimodal large model; otherwise, a pre-trained YOLOv8 model is used for defect detection.

[0094] This embodiment also proposes a multi-modal power inspection system driven by domain knowledge, such as Figure 7 As shown, the inspection system proposed in this embodiment includes:

[0095] The edge device is configured to perform the following operations:

[0096] The pre-deployed lightweight target detection network is used to perform preliminary detection and classification on the image set of the target to be detected, filter out defect-free samples, and transmit defective samples containing location information and category information to the cloud server; the specific process is as described in step 100 above and will not be repeated here.

[0097] and a cloud server configured to perform the following operations:

[0098] Classify defect samples according to preset target sizes and annotate defect sample data into the data format required for subsequent defect detection;

[0099] Determine whether it is a small target defect based on the classification result. If so, perform small target defect detection based on the large language model and the multimodal large model. Otherwise, use the pre-trained YOLOv8 model for defect detection.

[0100] Furthermore, the detection system of this embodiment also includes an image acquisition device, which is used to acquire an image of the target to be detected.

[0101] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0102] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0103] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0105] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation method of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A multi-modal power inspection method driven by domain knowledge, characterized in that: The multi-mode power inspection method comprises: Receive defect samples containing location information and category information transmitted by the edge, where the defect samples are obtained by performing preliminary detection and classification on an image data set of the target to be detected using a lightweight target detection network deployed on the edge and filtering out defect-free samples; Classifying the defect samples according to a preset target size, and marking the defect sample data into a data format required for subsequent defect detection; Determine whether it is a small target defect based on the classification result of the previous step. If it is, perform small target defect detection based on the large language model and the multimodal large model. Otherwise, use the pre-trained YOLOv8 model for defect detection. The small target defect detection based on the large language model and the multimodal large model specifically includes: Based on the power operation procedures, equipment defect classification specifications and equipment defect elimination guidelines, the Llama3 language model is used to generate the defect generation environment; The pre-trained CLIP model is used to perform category recognition on the image of the defect sample, and the output recognition result is compared with the defect generation environment generated by the Llama3 language model. If they match, it is determined that the image has the corresponding defect; The training process of the CLIP model specifically includes: The description texts of various small targets on the transmission line are fed into the text encoder of the CLIP model to obtain the corresponding text features; The historical image data with category labels is fed into the image encoder of the CLIP model to obtain image features; The CLIP model is learned using the contrastive loss function, with the goal of minimizing the distance between relevant text features and image features, and maximizing the distance between irrelevant text features and image features, so that the image classification prediction result is the category corresponding to the text feature with the greatest similarity.

2. The multi-modal power inspection method driven by domain knowledge according to claim 1 is characterized in that: The lightweight target detection network deployed on the edge performs preliminary detection and classification on the image data set of the target to be detected and filters out defect-free samples. The specific process includes: Input the image into the ResNet model to obtain the classification results and feature maps; Processing the feature map to preliminarily extract location information and category information of the defects; The judgment statement is used to judge and eliminate the images that do not contain the preset target category and are determined to be defect-free.

3. The multi-modal power inspection method driven by domain knowledge according to claim 1 is characterized in that: The marking of the defect sample data into the data format required for subsequent defect detection specifically includes: The defect samples are respectively labeled as whole parts and defective parts, and the labeled image data is preprocessed, including: adjusting the image size, normalizing the pixel values ​​of the image, and converting the image into YOLO v 8 The format required by the algorithm annotates small target defect type images as image-text pairs to prepare for fine-tuning of the multimodal large model.

4. The multi-modal power inspection method driven by domain knowledge according to claim 1 is characterized in that: The contrast loss function is expressed as: L(A,P,N)=max(0,margin+d(A,P)-d(A,N)) Among them, A, P, and N represent the target sample, positive sample, and negative sample respectively, d(A, P) and d(A, N) represent the distance between the target sample and the positive sample and negative sample respectively, and margin is the threshold.

5. The multi-modal power inspection method driven by domain knowledge according to claim 1 is characterized in that: The training process of the CLIP model also includes: Using Ada m The optimizer optimizes the CLIP model.

6. The multi-modal power inspection method driven by domain knowledge according to claim 1 is characterized in that: The above-mentioned defect detection using the pre-trained YOLOv8 model includes the following specific processes: According to the position information, target positioning and classification are realized, and the target classified as a defective part is positioned and defect classified to obtain a defect position frame and a category name; The defect location box and category name are passed to the subsequent classification network to determine whether it belongs to a certain category.

7. A multi-modal power inspection device driven by domain knowledge, characterized in that: The multi-mode power inspection device comprises: A data receiving unit, the data receiving unit is used to receive defect samples containing location information and category information transmitted by the edge, the defect samples are obtained by performing preliminary detection and classification on the image data set of the target to be detected by using a lightweight target detection network deployed on the edge and filtering out defect-free samples; A classification and labeling unit, wherein the classification and labeling unit classifies the defect samples according to a preset target size and labels the defect sample data into a data format required for subsequent defect detection; and, a judgment unit, wherein the judgment unit judges whether it is a small target defect according to the classification result of the classification and labeling unit, and if so, performs small target defect detection according to the large language model and the multimodal large model, otherwise, uses the pre-trained YOLOv8 model for defect detection; The small target defect detection based on the large language model and the multimodal large model specifically includes: Based on the power operation procedures, equipment defect classification specifications and equipment defect elimination guidelines, the Llama3 language model is used to generate the defect generation environment; The pre-trained CLIP model is used to perform category recognition on the image of the defect sample, and the output recognition result is compared with the defect generation environment generated by the Llama3 language model. If they match, it is determined that the image has the corresponding defect; The training process of the CLIP model specifically includes: The description texts of various small targets on the transmission line are fed into the text encoder of the CLIP model to obtain the corresponding text features; The historical image data with category labels is fed into the image encoder of the CLIP model to obtain image features; The CLIP model is learned using the contrastive loss function, with the goal of minimizing the distance between relevant text features and image features, and maximizing the distance between irrelevant text features and image features, so that the image classification prediction result is the category corresponding to the text feature with the greatest similarity.

8. A multi-modal power inspection system driven by domain knowledge, characterized in that: The multi-modal power inspection system comprises: The edge device is configured to perform the following operations: Use the pre-deployed lightweight object detection network to perform preliminary detection and classification on the image set of the target to be detected, filter out defect-free samples, and transmit defective samples containing location information and category information to the cloud server; and a cloud server, wherein the cloud server is configured to perform the following operations: Classify defect samples according to preset target sizes and annotate defect sample data into the data format required for subsequent defect detection; Determine whether it is a small target defect based on the classification result. If it is, perform small target defect detection based on the large language model and the multimodal large model. Otherwise, use the pre-trained YOLOv8 model for defect detection. The small target defect detection based on the large language model and the multimodal large model specifically includes: Based on the power operation procedures, equipment defect classification specifications and equipment defect elimination guidelines, the Llama3 language model is used to generate the defect generation environment; The pre-trained CLIP model is used to perform category recognition on the image of the defect sample, and the output recognition result is compared with the defect generation environment generated by the Llama3 language model. If they match, it is determined that the image has the corresponding defect; The training process of the CLIP model specifically includes: The description texts of various small targets on the transmission line are fed into the text encoder of the CLIP model to obtain the corresponding text features; The historical image data with category labels is fed into the image encoder of the CLIP model to obtain image features; The CLIP model is learned using the contrastive loss function, with the goal of minimizing the distance between relevant text features and image features, and maximizing the distance between irrelevant text features and image features, so that the image classification prediction result is the category corresponding to the text feature with the greatest similarity.

Images