Small sample new energy equipment defect identification method and system based on multi-modal statistical prompt learning, storage medium and equipment

Through the multi-modal statistical prompt learning method, combined with statistical query and soft prompts, the problems of medium and high cost and computing resource limitations of new energy equipment monitoring are solved, efficient training and deployment on edge devices are realized, and new energy equipment defect identification is improved. accuracy and generalization ability.

CN119066513AActive Publication Date: 2024-12-03DALIAN UNIV OF TECH
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411107300.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-13
Publication Date
2024-12-03
Estimated Expiration
2044-08-13

AI Technical Summary

Technical Problem

The existing technology relies on high-cost and high-performance servers for model training in new energy equipment monitoring, resulting in high labor costs, time-consuming and unsuitable for edge device deployment. The traditional small sample learning method has limited performance in new tasks with large data differences. , it is difficult to achieve efficiently on edge devices.

Method used

Using multimodal statistical prompt learning method, the statistical query module is introduced to utilize the marked features of text and image encoder, combined with soft prompts, to reduce the computing resource requirements, and only the soft prompts and classifier parameters are updated when the backbone network parameters are frozen. , realize efficient learning and deployment of few samples.

Benefits of technology

It realizes efficient training and deployment of algorithms on edge devices, reduces data dependence and computing costs, and improves the recognition accuracy and generalization ability of defect recognition of new energy equipment. It is suitable for diverse scenarios and performs well in the task of defect recognition of new energy equipment with few samples. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066513A_ABST
    Figure CN119066513A_ABST
Patent Text Reader

Abstract

The invention discloses a small sample new energy equipment defect identification method and system based on multi-modal statistical prompt learning, a storage medium and equipment, and belongs to the field of new energy equipment defect identification. The method comprises six steps of data acquisition, feature extraction, statistical query, soft prompt combination, classifier construction and model training. Discriminative features are obtained from features extracted from a trunk by taking statistical features as queries and are combined with soft prompts to reduce the demand for computing resources, and meanwhile, text input is introduced to enhance the understanding of defect categories, so that the recognition precision and generalization ability are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of defect recognition of new energy equipment, and in particular to a method, system, storage medium and equipment for defect recognition of small sample new energy equipment using multimodal statistical prompt learning. Background Art

[0002] New energy sources such as wind power and photovoltaics are gradually replacing traditional energy sources with their clean and renewable advantages. However, these devices are usually deployed in remote areas and are difficult to monitor in real time. If faults cannot be discovered in time, it may affect economic benefits and the stability of the power system.

[0003] Existing technologies rely on edge devices equipped with artificial intelligence algorithms for monitoring, but these algorithms usually require large-scale labeled data training, which increases labor costs, and rely on high-performance servers for model training, which is time-consuming and costly. Therefore, there is an urgent need for an algorithm that can be efficiently trained and deployed on edge devices, reducing data dependence and computing costs, while adapting to diverse scenarios.

[0004] Few-shot learning can extract features from limited data and demonstrate strong generalization capabilities. However, traditional methods have limited performance when dealing with new tasks that are very different from the training data. Multimodal models, such as CLIP, have demonstrated excellent zero-shot capabilities by pre-training on large-scale data, and have achieved efficient generalization in few-shot tasks through tuning strategies. However, these tuning methods, such as soft-hint tuning, usually require introducing hints at the input stage and updating hint parameters, which is difficult to implement efficiently on edge devices due to their limited computing and memory resources.

[0005] To solve this problem, the present invention proposes a multimodal statistical prompt learning method for small sample new energy equipment defect identification. Different from the existing methods, the present invention introduces a statistical query module to extract discriminative features using all the tags of the text and image encoders, and then combines them with soft prompts at the end of the backbone to improve the classification performance. In addition, while keeping the backbone network parameters frozen, the present invention only needs to update the soft prompts and classifiers, ensuring parameter efficiency, and the present invention only needs to reversely update the gradient to the end of the backbone, significantly reducing the demand for computing resources, making it suitable for few-sample learning and deployment on edge devices. Summary of the invention

[0006] In view of the defects in the prior art, the present invention proposes a small sample new energy equipment defect recognition method based on multimodal statistical prompt learning. The statistical features are used as queries to obtain discriminative features from the features extracted by the backbone, and they are combined with soft prompts to reduce the demand for computing resources. At the same time, text input is introduced to enhance the understanding of defect categories, thereby improving recognition accuracy and generalization ability.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A small sample defect recognition method for new energy equipment based on multimodal statistical prompt learning, the steps are as follows:

[0009] S1, data acquisition: obtain defect images of new energy equipment and text descriptions of defect categories, taking images and text as input.

[0010] S2, feature extraction: Use the pre-trained multimodal model CLIP as the text encoder and image encoder to extract text and visual features respectively. The text encoder is denoted as E T , the image encoder is E I , the text input is denoted by X T , the image (vision) input is recorded as X I , the feature extraction process is as follows:

[0011]

[0012] in, and are the outputs of the text and image encoders, respectively, and their channel size is denoted by C. and denote M text tokens and N visual tokens, respectively, x eos The token indicating the end of the sentence, x cls represents the class token in the image. WE and PE represent word embedding for text processing and position embedding for image processing, respectively.

[0013] S3, statistical query: Perform statistical query operations on the text and visual features extracted by the multimodal model CLIP. For the features of the visual part, only visual tokens are used when calculating statistical queries. , and record it as First, calculate X along the channel dimension v Statistical moments of:

[0014]

[0015] in, is the a-th element in a dimension, is the q-th moment, q is an integer greater than or equal to 3, is the first-order moment (i.e., the mean), and then the q-order moment is normalized:

[0016]

[0017] in, is the second-order moment (i.e., standard deviation). Statistical features can reflect the overall structure and behavior of the data, so using them as queries can effectively retrieve distinguishing features from the original feature map. However, a single statistical moment query method may not be sufficient to capture the complexity and diversity of the data. To this end, the present invention combines multiple statistical moments as a query method to more comprehensively characterize the statistical properties of the data. The query process is as follows:

[0018]

[0019] in, represents the combination of Q statistical moments, is the softmax function, which is used to normalize the features. T is the transpose of the matrix S. Finally, F is used to represent the discriminative features obtained from the output of the image encoder. For the output of the text encoder, all M tokens are used, and the rest of the process is the same as the image encoder output.

[0020] S4, combined with soft prompts: The discriminative features obtained by statistical query are combined with soft prompts. For the convenience of explanation, the statistical query process can be expressed as follows:

[0021]

[0022] Among them, F T and F I Respectively represent the text and visual features after the statistical query operation. Then the soft prompts are combined with these discriminative features in the form of splicing, and integrated into the overall representation of text and vision through mean pooling:

[0023]

[0024] in, and denote i textual soft prompts and j visual soft prompts respectively, which are learnable vectors with feature dimension C. Represents mean pooling.

[0025] S5, build a classifier: use a linear classifier to predict the probability of defects in new energy equipment by combining the overall representation of text and vision:

[0026] y T =W(x eos +O T ),y I =W(x cls +O I )

[0027] where W represents the learnable parameters of the classifier and W is shared between textual and visual features.

[0028] S6, training model: During the training process, the CLIP backbone remains frozen and only the soft prompts and classifier parameters are updated. The learned model can directly perform defect recognition on new energy equipment images.

[0029] The second aspect of the present invention provides a small sample new energy equipment defect recognition system using multimodal statistical prompt learning, comprising:

[0030] Data acquisition module: obtains input image data and corresponding text description;

[0031] Feature extraction module: uses pre-trained CLIP as the backbone to extract visual and text features;

[0032] Statistical query module: Calculates statistical features of visual and textual features as queries and obtains discriminative features from the features;

[0033] Combined soft hint module: Combine soft hints with discriminative features and obtain a holistic representation of text and vision through mean pooling;

[0034] Classifier: Use a shared line classifier to identify defect categories based on text and visual features;

[0035] Model training module: Update learnable parameters through back-propagation.

[0036] The third aspect of the present invention provides a medium having a program stored thereon, which, when executed by a processor, implements the steps of the small sample new energy equipment defect identification method based on multimodal statistical prompt learning described in the first aspect of the present invention.

[0037] The fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps of the small sample new energy equipment defect identification method based on multimodal statistical prompt learning described in the first aspect of the present invention are implemented.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] The proposed method for defect recognition of new energy equipment with small samples based on multimodal statistical prompt learning achieves high efficiency of parameters and memory by obtaining the distinguishing features of the encoder output and combining soft prompts. It also introduces text as auxiliary information to understand the concept of defects, and performs well in defect recognition of new energy equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0041] Figure 1 It is a flow chart of a small sample new energy equipment defect identification method based on multimodal statistical prompt learning of the present invention;

[0042] Figure 2 It is a framework diagram of a small sample new energy equipment defect identification method based on multimodal statistical prompt learning of the present invention;

[0043] Figure 3 It is the experimental equipment used in the present invention, wherein (a) is the edge computing platform and (b) is the AAEON BOXER-8640AI device. DETAILED DESCRIPTION

[0044] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0045] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0046] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0047] Embodiment 1

[0048] like Figure 1 As shown, the first embodiment of the present invention provides a flowchart of a method for identifying defects of new energy equipment with a small sample size using multimodal statistical prompt learning, including the following steps:

[0049] S1. Data acquisition module: obtains input image data and corresponding text description;

[0050] S2. Feature extraction module: uses pre-trained CLIP as the backbone to extract visual and text features;

[0051] S3. Statistical query module: Calculate statistical features for visual and textual features as queries and obtain discriminative features from the features;

[0052] S4. Combined soft hint module: Combine soft hints with discriminative features and obtain the overall representation of text and vision through mean pooling;

[0053] S5. Classifier: Use a shared line classifier to identify defect categories based on text and visual features;

[0054] S6. Model training module: Update learnable parameters through back-propagation.

[0055] Among them, S1 is specifically:

[0056] The present invention utilizes two datasets, wind turbine blade defects (WTBD) and solar cell defects (SCD). WTBD is derived from videos captured during drone inspections and includes four categories: dirt, erosion, oil leakage, and paint peeling, with 686, 606, 699, and 893 images in each category, respectively. SCD comes from near-infrared images of solar cells, including six categories: black core, crack, finger, horizontal dislocation, short circuit, and thick line, with 981, 471, 963, 266, 492, and 583 images in each category, respectively. WTBD and SCD datasets are allocated 30% as test sets and 70% as training sets. For the validation set, the same number of samples are selected from each class in the test set, 128 images per class for WTBD and 64 images per class for SCD. For visual input, 16 samples are randomly sampled from the training set for small sample training; for text input, multiple manual prompt templates are used in combination with class names to construct the templates, as shown in Table 1.

[0057] S2 is specifically:

[0058] The pre-trained multimodal model CLIP is used as the text encoder and image encoder to extract text and visual features respectively. The ViT-B / 16 version of CLIP is used in the specific implementation. The text encoder is denoted as E T , the image encoder is E I , the text input is denoted by X T , the image (vision) input is recorded as X I , the feature extraction process is as follows:

[0059]

[0060] in, and are the outputs of the text and image encoders, respectively, and their channel size is denoted by C. and denote M text tokens and N visual tokens respectively, A token indicating the end of a sentence, Then represents the class token in the image. WE and PE represent word embedding in text processing and position embedding in image processing respectively. In the specific implementation, M=77, N=196, C=512.

[0061] S3 is specifically:

[0062] Perform statistical query operations on the text and visual features extracted from the backbone. For the features of the visual part, the present invention only uses visual tokens when calculating statistical queries. , and record it as First, calculate X along the channel dimension v Since the first-order moment and the second-order moment can be obtained by calculating the mean and variance, the following formula only represents the calculation process above the third-order moment:

[0063]

[0064] in, is the a-th element in a dimension, is the q-th moment, q is an integer greater than or equal to 3, is the first-order moment (i.e., the mean), and then the q-order moment is normalized:

[0065]

[0066] in, is the second-order moment (i.e., standard deviation). The present invention combines multiple statistical moments as a query method, and the query process is as follows:

[0067]

[0068] in, represents a combination of Q statistical moments, and in the implementation, Q = 2. For the WTBD dataset, a combination of the third-order moment and the fourth-order moment is used as the query, that is, For the SCD dataset, a combination of the third-order moment and the fifth-order moment is used as the query, i.e. is the softmax function, which is used to normalize the features. T is the transpose of the matrix S. Finally, F is used to represent the discriminative features obtained from the output of the visual encoder. Due to the characteristics of matrix multiplication, For the output of the text encoder, all M tokens are used, and the rest of the process is the same as the image encoder output processing.

[0069] S4 is specifically:

[0070] Combined with soft prompts: The discriminative features obtained by statistical query are combined with soft prompts. For the convenience of explanation, the statistical query process can be expressed as follows:

[0071]

[0072] Among them, F T and F I Respectively represent the text and visual features after the statistical query operation. Then the soft prompts are combined with these discriminative features in the form of splicing, and integrated into the overall representation of text and vision through mean pooling:

[0073]

[0074] in, and They represent i textual soft prompts and j visual soft prompts respectively, which are learnable vectors with feature dimension C = 512. Represents mean pooling. In the implementation, i=j=4. Overall representation They are all vectors of C=512.

[0075] S5 is as follows:

[0076] Build a classifier: Use a linear classifier to combine the overall text and visual representation to predict the probability of defects in new energy equipment:

[0077] y T =W(x eos +O T ),y I =W(x cls +O I )

[0078] in, represents the weight of the classifier, C = 512 represents the input dimension of the classifier, which is consistent with the label dimension, and K represents the number of categories of new energy equipment defects. In the WTBD dataset, K = 4, and in the SCD dataset, K = 6. In addition, since the classifiers are shared, the W values ​​in the text and visual classifiers are the same.

[0079] S6 is specifically:

[0080] Since the text and image encoders are frozen during training, the present invention pre-extracts features to optimize efficiency. Training was performed for 12,800 iterations using the learning rate with cosine function adjustment and the AdamW optimizer, including 50 iterations of warm-up. The batch size was set to 2, and the learning rate and weight decay were determined by range search, selecting {0.001, 0.0001} and {0.0, 0.0.01, 0.0001}, respectively. The experiments were conducted on an edge computing platform equipped with an AAEON BOXER-8640AI device, and the code was implemented using the PyTorch framework.

[0081] The model of the present invention performs excellently in the task of defect identification of new energy equipment with a small number of samples. On the WTBD dataset, the result of training with 16 samples is 92.35%. On the SCD dataset, the result of training with 16 samples is 81.03%. These results show the great potential and excellent performance of the present invention in the task of defect identification of new energy equipment with a small number of samples. On both datasets, the average running time of the method of the present invention on the edge device is only about 20 minutes, and the power consumption is less than 20Wh. Therefore, the method of the present invention is fully capable of being trained on the edge device. During reasoning, the method of the present invention processes data at a speed of approximately 50 frames per second (FPS), demonstrating its real-time capability on the edge device. Taking drones as an example, the video frame rate of drones is typically 30 frames per second, which shows that the method of the present invention can achieve real-time reasoning on ordinary edge devices.

[0082] Embodiment 2

[0083] Embodiment 2 of the present invention provides a small sample new energy equipment defect recognition system using multimodal statistical prompt learning, including:

[0084] Data acquisition module: obtains input image data and corresponding text description;

[0085] Feature extraction module: uses pre-trained CLIP as the backbone to extract visual and text features;

[0086] Statistical query module: Calculates statistical features of visual and textual features as queries and obtains discriminative features from the features;

[0087] Combined soft hint module: Combine soft hints with discriminative features and obtain a holistic representation of text and vision through mean pooling;

[0088] Classifier: Use a shared line classifier to identify defect categories based on text and visual features;

[0089] Model training module: Update learnable parameters through back-propagation.

[0090] The more detailed steps are the same as those in the first embodiment and will not be repeated here.

[0091] Embodiment 3

[0092] Embodiment 3 of the present invention provides a medium on which a program is stored. When the program is executed by a processor, the steps of the small sample new energy equipment defect identification method based on multimodal statistical prompt learning as described in Embodiment 1 of the present invention are implemented.

[0093] The more detailed steps are the same as those in the first embodiment and will not be repeated here.

[0094] Embodiment 4

[0095] Embodiment 4 of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, defect identification of small sample new energy equipment based on multimodal statistical prompt learning as described in Embodiment 1 of the present invention is implemented.

[0096] The more detailed steps are the same as those in the first embodiment and will not be repeated here.

[0097] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0098] Table 1 Prompt template

[0099]

[0100]

Claims

1. A small sample defect recognition method for new energy equipment based on multimodal statistical prompt learning, characterized in that: Here are the steps: S1, data acquisition: obtain defect images of new energy equipment and text descriptions of defect categories, taking images and text as input; S2, feature extraction: Use the pre-trained multimodal model CLIP as the text encoder and image encoder to extract text and visual features respectively. The text encoder is denoted as E T , the image encoder is E I , the text input is denoted as X T , the image input is recorded as X I , the feature extraction process is as follows: in, and are the outputs of the text and image encoders, respectively, and their channel size is denoted by C. and denotes M text tokens and N visual tokens, respectively, X eos The token indicating the end of the sentence, X cls represents the class token in the image; WE and PE represent word embedding in text processing and position embedding in image processing, respectively; S3, statistical query: Perform statistical query operations on the text and visual features extracted by the multimodal model CLIP. For the features of the visual part, only visual tokens are used when calculating statistical queries. , and record it as First, calculate X along the channel dimension v Statistical moments of: in, is the a-th element in a dimension, is the q-th moment, q is an integer greater than or equal to 3, is the first-order moment, and then the q-order moment is normalized: in, is the second-order moment; statistical features can reflect the overall structure and behavior of data, so using them as queries can effectively retrieve distinguishing features from the original feature map; however, the query method of a single statistical moment may not be sufficient to capture the complexity and diversity of data; for this reason, the present invention combines multiple statistical moments as a query method to more comprehensively characterize the statistical properties of data, and the query process is as follows: in, represents the combination of Q statistical moments, is the softmax function, which is used to normalize the features. T is the transpose of the matrix S; finally, F is used to represent the discriminative features obtained from the output of the image encoder; for the output of the text encoder, all M tokens are used, and the rest of the process is the same as the image encoder output processing; S4, combined with soft prompts: The discriminative features obtained by statistical query are combined with soft prompts. For the convenience of explanation, the statistical query process can be expressed as follows: Among them, F T and F I Respectively represent the text and visual features after the statistical query operation; then the soft prompts are combined with these discriminative features in the form of splicing, and integrated into the overall representation of text and vision through mean pooling: in, and denote i textual soft prompts and j visual soft prompts respectively, which are learnable vectors with feature dimension C. represents mean pooling; S5, build a classifier: use a linear classifier to predict the probability of defects in new energy equipment by combining the overall representation of text and vision: y T =W(x eos +O T ),y I =W(x cls +O I ) Where W represents the learnable parameters of the classifier, and W is shared between text and visual features; S6, training model: During the training process, the CLIP backbone remains frozen and only the soft prompts and classifier parameters are updated. The learned model can directly perform defect recognition on new energy equipment images.

2. A small sample size new energy equipment defect recognition system based on multimodal statistical prompt learning, characterized in that: include: Data acquisition module: obtains input image data and corresponding text description; Feature extraction module: uses pre-trained CLIP as the backbone to extract visual and text features; Statistical query module: Calculates statistical features of visual and textual features as queries and obtains discriminative features from the features; Combined soft hint module: Combine soft hints with discriminative features and obtain a holistic representation of text and vision through mean pooling; Classifier: Use a shared line classifier to identify defect categories based on text and visual features; Model training module: Update learnable parameters through back-propagation.

3. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method for identifying defects of new energy equipment with a small sample size using multimodal statistical prompt learning as described in claim 1 are implemented.

4. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method for identifying defects of new energy equipment with a small sample size using multimodal statistical prompt learning as described in claim 1 are implemented.

Citation Information

Patent Citations

  • Commodity identification method and system based on multi-modal data

    CN115601582A

  • Multi-modal false news detection method based on multi-level semantic enhancement

    CN117315695A

  • Wafer defect identification method and device based on multi-modal learning, equipment and medium

    CN117541973A

  • Small sample visual classification method and device based on retrieval enhancement mechanism and visual cue learning

    CN117953282A

  • Multi-modal prompt learning for representation transfer on image recognition tasks

    US20240220722A1