An intelligent electric meter visual fault detection method and system based on multi-task routing and memory retrieval

Abstract

The application provides a smart meter visual fault detection method and system based on multi-task routing and memory retrieval, which comprises: after normalizing and preprocessing a smart meter image, the image is input into a two-stage backbone network to extract features. In the base class training stage, the feature map generates a to-be-detected region feature through a region proposal network, and the region feature is sent into a multi-task dynamic routing module to adaptively decouple regression and classification task features; the classification features are input into a visual prototype memory regularization module to construct a memory bank, and the two types of features are jointly input into a task head to output a prediction and calculate a loss. In the new class fine-tuning stage, the task features are extracted through the multi-task dynamic routing; the classification features pass through a visual prototype semantic refinement module to calculate an attention weight, and weighted classification features are obtained, which are input into the task head together with the regression features to output a prediction, calculate a loss, and iteratively optimize the model. Finally, the to-be-detected meter image is input into the trained model to realize fault category detection.

Description

Technical Field

[0001] This application belongs to the field of electricity metering technology, and more specifically, relates to a visual fault detection method and system for smart meters based on multi-task routing and memory retrieval. Background Technology

[0002] With the accelerated construction of new power systems and the continuous expansion of the power Internet of Things (IoT), smart meters, as core devices at the edge of the power grid's sensing network, directly impact the stability and data integrity of the electricity consumption information collection system. Currently, smart meters have achieved diversified functions such as electricity consumption information collection, load monitoring, and energy efficiency management, providing crucial data support for smart cities, smart homes, and demand-side response. However, the contradiction between the rapid growth in the scale of terminal devices and the capacity for operation and maintenance support is becoming increasingly prominent, and traditional manual inspection methods are no longer sufficient to meet the maintenance needs of massive numbers of terminal devices.

[0003] With the large-scale deployment of terminal equipment, visible defects such as missing LCD screens, cracked casings, burned wiring terminals, and abnormal lead seals are frequently observed during operation. Existing maintenance methods, relying on manual on-site inspections and manual recording of defects, not only suffer from high missed detection rates and long troubleshooting cycles, but also suffer from inconsistent defect judgment standards due to varying professional competence among maintenance personnel. Faced with the exponentially increasing number of terminals, traditional maintenance models have revealed systemic risks such as delayed response and low handling efficiency. There is an urgent need to deeply integrate intelligent diagnostic technology with automated maintenance systems to build a new maintenance paradigm with self-awareness of status and self-diagnosis of faults, providing a solid foundation for the safe and reliable operation of new power systems.

[0004] Current mainstream methods primarily rely on feature engineering and template matching techniques to detect external defects in electricity meters under controlled environments. The accuracy of these methods is highly dependent on expert knowledge, requiring complex parameter optimization for different meter models. In real-world deployment environments, factors such as lighting conditions, shooting angle, and background interference can easily lead to false positives and false negatives in traditional algorithms, resulting in weak adaptability and difficulty in meeting the detection needs of complex on-site environments. In contrast, deep learning models, represented by convolutional neural networks, can autonomously learn multi-level feature representations from massive amounts of image data, effectively overcoming the problem of morphological diversity in complex scenes and providing a more powerful technical path for electricity meter defect detection.

[0005] Deep learning-based visual fault detection methods for electricity meters primarily limit identification to pixel-level pattern matching, failing to incorporate the deep semantic information inherent in large language models. This approach can only determine if an image "looks like" a certain fault, but cannot connect it to abstract knowledge such as the causes and risks of the fault, resulting in a lack of interpretability and depth in the diagnostic results. By introducing semantic embedding vectors from large models, visual features can be aligned with the semantic space of similar textual descriptions. This cross-modal semantic alignment not only improves the classification confidence of difficult samples but also builds a semantic bridge from visual appearance to fault mechanisms, ultimately achieving intelligent fault diagnosis with causal inference capabilities. Therefore, effectively integrating multi-dimensional semantic information into visual features is a key challenge for achieving intelligent diagnosis. Introducing semantic embedding from language models can construct a visual-semantic joint representation space, significantly improving the model's understanding and reasoning capabilities. However, inappropriate selection of semantic information can easily lead to feature space conflicts or introduce external noise, and some abstract semantic concepts may not be effectively aligned with specific fault features. Summary of the Invention

[0006] In view of this, the present invention provides a method and system for visual fault detection of smart meters based on multi-task routing and memory retrieval, so as to improve the accuracy of visual fault detection of smart meters.

[0007] According to the first aspect of this application, a visual fault detection method for smart meters based on multi-task routing and memory retrieval is provided, comprising the following steps:

[0008] Normalization preprocessing is performed on the smart meter image data to obtain the normalized image corresponding to the smart meter image data;

[0009] The normalized graph corresponding to the smart meter data is input into the detection model, and the smart meter image data is progressively trained to obtain a trained smart meter visual fault detection model.

[0010] The image data of the meter under test is input into the trained smart meter visual fault detection model to complete the fault category detection.

[0011] The step of performing normalization preprocessing on the smart meter image data to obtain the normalized image corresponding to the smart meter image data includes:

[0012] Each smart meter image in the data is labeled with a fault category to form a training set;

[0013] Adjust the size of all samples in the training set to... , The pixel dimensions of the image;

[0014] Calculate the mean and standard deviation of pixel values ​​for each channel for all samples in the training set, so that the pixel values ​​satisfy a distribution with a mean of 0 and a standard deviation of 1, to obtain the normalized image corresponding to the smart meter image data.

[0015] The steps for inputting the normalized image corresponding to the smart meter data into the detection model, progressively training the smart meter image data, and obtaining the trained smart meter visual fault detection model include:

[0016] In the first training phase, the base class image in the normalized graph corresponding to the smart meter data is sampled, and the base class image is used as input to the smart meter visual fault detection model constructed with the DeFRCN framework.

[0017] The smart meter visual fault detection model based on multi-task routing and memory retrieval is driven to adaptively learn the features of the base class image, and the feature map of the base class image is extracted by the backbone network;

[0018] The feature map is input into a proposal region network to obtain the coordinates of each proposal box. Based on the coordinates of each proposal box, the feature map is aligned to obtain the features of the region to be tested.

[0019] The proposed region features are input into the multi-task dynamic routing, which adaptively decouples the task information of the proposed region features, and further extracts the regression task features and classification task features of the proposed region features;

[0020] The classification task features are input into the visual prototype memory regularization module to obtain a memory module composed of visual prototypes for each category.

[0021] The regression task features are input into the regression task header to output predicted coordinate information, and the regression loss is calculated based on the true label and the predicted coordinate information. The classification task features are input into the classification task header to output predicted category information, and the classification loss is calculated based on the true label and the predicted category information.

[0022] Gradient backpropagation is performed based on the loss values ​​of regression loss and classification loss to iteratively optimize the parameters of the smart meter visual fault detection model, thereby obtaining the trained first-stage smart meter visual fault detection model.

[0023] In the second training phase, the base class and new class balance images in the normalized graph corresponding to the smart meter data are sampled, and the balance image data is input into the smart meter visual fault detection model constructed with the DeFRCN framework.

[0024] The smart meter visual fault detection model based on multi-task routing and memory retrieval adaptively learns each part of the multi-task information of the suggested features, and the backbone network extracts the feature maps of the base class and new class balanced images.

[0025] The feature map is input into a proposal region network to obtain the coordinates of each proposal box. Based on the coordinates of each proposal box, the feature map is aligned to obtain the features of the region to be tested.

[0026] The proposed region features are input into the multi-task dynamic routing, which adaptively decouples the task information of the proposed region features, and further extracts the regression task features and classification task features of the proposed region features;

[0027] The classification task features are input into the semantic refinement module of the visual prototype, and the scaling dot product attention between the memory items of the memory module and the classification task features is calculated to obtain the attention-weighted classification task features.

[0028] The regression task features are input into the regression task header to output the predicted coordinate information, and the regression loss is calculated based on the true label and the predicted coordinate information.

[0029] The classification task features are input into the classification task head to output the predicted category information, and the classification loss is calculated based on the true label and the predicted category information.

[0030] Gradient backpropagation is performed based on the loss values ​​of regression loss and classification loss to iteratively optimize the parameters of the smart meter visual fault detection model, resulting in the final trained smart meter visual fault detection model.

[0031] The step of driving the smart meter visual fault detection model to adaptively learn the features of the base class image, and extracting the feature map of the base class image by the backbone network includes:

[0032] The base class image in the normalized graph corresponding to the smart meter data is input into the deep residual network ResNet101.

[0033] Features of the base class image are extracted from the convolutional layers of the deep residual network to obtain the final feature map of the base class image.

[0034] The steps of inputting the proposed region features into the multi-task dynamic routing, adaptively decoupling the task information of the proposed region features, and further extracting the regression task features and classification task features of the proposed region features include:

[0035] The proposed region features are input into a multi-task dynamic routing algorithm, and three five-layer deep residual networks are used to output the intermediate regression features. Intermediate features of classification and shared intermediate features ;

[0036] The classification intermediate features and shared intermediate features are input into a gated fusion network, and then pass through a fully connected layer. and activation function Output the corresponding branch scores, multiply the branch scores by the intermediate features to obtain their respective task features, and finally calculate the shared task features. and classification task features The summation and output of the final classification task features are then performed. The specific formula is as follows:

[0037]

[0038]

[0039]

[0040] The regression intermediate features The input is fed into a gated fusion network and passes through a fully connected layer. and activation function Output the corresponding branch scores, multiply the branch scores by the intermediate features to obtain their respective task features, and finally calculate the shared task features. and regression task features The summation and output of the final classification task features are then performed. The specific formula is as follows:

[0041]

[0042]

[0043] The steps of inputting the classification task features into a memory regularization module for visual prototypes to obtain a memory module composed of visual prototypes for each category include:

[0044] First, the category labels of the classification task features are filled into the prompt word template "A photo of a{category}" to obtain the prompt words. Then, the filled prompt words are processed by word segmentation to generate word units. ;

[0045] No. each word element The text encoder input to the pre-trained CLIP model Generate semantic vectors The formula is as follows:

[0046]

[0047] The first Text embedding Input to linear layer Perform mapping to obtain the mapped first... Text embedding Calculate the mapped first The text embedding and the first Characteristics of each classification task cosine similarity , The formula for calculating the magnitude of a vector is as follows:

[0048]

[0049]

[0050] Next, we calculate the alignment loss between cosine similarity and the true class label of the classification task features. The cross-entropy loss is optimized during model training to align visual features and text embeddings. The specific formula for the alignment loss is as follows:

[0051]

[0052] in and Representing the features of the classification task respectively and text embedding features The number of The probability that the true class label and the predicted class are the same as the true class label in the classification task. Represents the probability of the predicted category;

[0053] In each round of training, the classification task features are stored as visual prototypes in a memory matrix of size [C,D], where C is the number of categories and D is the dimension of the category vector. Finally, when the loss approaches convergence, the memory matrix obtained is the memory module composed of the visual prototypes of each category.

[0054] Input the regression task features and classification task features into the classification task header and regression task header to obtain the predicted coordinates and predicted class labels;

[0055] use As the true label for regression loss calculation and predicted coordinates The loss value between them is calculated using the following formula:

[0056]

[0057] in Represents the difference between the true label and the predicted coordinates;

[0058] Using the multivariate cross-entropy loss function The loss value used to calculate the classification loss is the loss between the true label and the predicted label of the category, and the formula is as follows:

[0059]

[0060] in It's a real label. It is the output class probability.

[0061] According to a second aspect of this application, a smart meter visual fault detection system based on multi-task routing and memory retrieval is provided, including a data processing module, a model training module, and a recognition module;

[0062] The data processing module is used to preprocess smart meter image data to obtain a normalized image corresponding to the smart meter image data;

[0063] The model training module is used to progressively train the smart meter visual fault detection model based on multi-task routing and memory retrieval using the normalized meter image as input to the detection model, so as to obtain the trained smart meter visual fault detection model.

[0064] The recognition module is used to input the image data of the meter under test into the trained smart meter visual fault detection model to complete the detection of fault categories.

[0065] According to a third aspect of this application, an electronic device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that the processor executes the computer program to implement the steps of any one of the methods described in the first aspect.

[0066] According to a fourth aspect of this application, a storage medium is provided that stores a computer program thereon, characterized in that the computer program, when executed by a processor, implements the steps of the method described in any one of the first aspects above.

[0067] The present invention has the following beneficial effects:

[0068] In the technical solution implemented in this invention, based on the visual characteristics of visible faults in electricity meters, visual-language cross-modal learning is used to associate fault manifestations with professional diagnostic knowledge, and a scalable memory bank is constructed to store typical fault patterns. During actual deployment, memory-enhanced inference utilizes prototype features to constrain the current detection results, improving the model's generalization ability. Furthermore, a feature fusion mechanism based on scaling dot product attention adaptively adjusts the contribution of multi-source information, enabling the system to accurately identify novel fault patterns while maintaining stable detection capabilities for common faults, thereby significantly improving the accuracy and robustness of fault detection. Attached Figure Description

[0069] Figure 1This is a schematic diagram of the framework of the smart meter visual fault detection method based on multi-task routing and memory retrieval provided by the present invention;

[0070] Figure 2 This is the training process for the smart meter visual fault detection method based on multi-task routing and memory retrieval.

[0071] Figure 3 This is a schematic diagram illustrating the implementation process and subsequent processing of the multi-task dynamic routing module.

[0072] Figure 4 This is a schematic diagram illustrating the implementation process and subsequent processing of the memory regularization module of the visual prototype.

[0073] Figure 5 This is a schematic diagram illustrating the implementation process and subsequent processing of the semantic refinement module of the visual prototype.

[0074] Figure 6 This is a schematic diagram of the smart meter visual fault detection system based on multi-task routing and memory retrieval provided by the present invention;

[0075] Figure 7 This is a schematic diagram of the device structure of the electronic device provided by the present invention. Detailed Implementation

[0076] To better understand the technical solution of the present invention, the present invention will be described in detail below with reference to the accompanying drawings.

[0077] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0078] This application provides a method for visual fault detection of smart meters based on multi-task routing and memory retrieval. To meet the requirements of visual fault detection in smart meters, the invention performs normalization preprocessing on smart meter image data to obtain a normalized image corresponding to the smart meter image data; inputs the normalized image corresponding to the smart meter data into a detection model, and progressively trains the smart meter image data to obtain a trained smart meter visual fault detection model; finally, inputs the image data of the meter to be tested into the trained smart meter visual fault detection model to complete the fault category detection.

[0079] Figure 1 This is a schematic diagram of the framework of the smart meter visual fault detection method based on multi-task routing and memory retrieval proposed in this invention. The method includes the following steps:

[0080] Step 101: Perform normalization preprocessing on the smart meter image data to obtain the normalized image corresponding to the smart meter image data, specifically as follows:

[0081] Each smart meter image in the data is labeled with a fault category to form a training set;

[0082] Adjust the size of all samples in the training set to... , The pixel dimensions of the image;

[0083] Calculate the mean and standard deviation of pixel values ​​for each channel for all samples in the training set, so that the pixel values ​​satisfy a distribution with a mean of 0 and a standard deviation of 1, to obtain the normalized image corresponding to the smart meter image data.

[0084] Step 102: The normalized image corresponding to the smart meter data is input into the detection model. Progressive training is performed on the smart meter image data to obtain a trained smart meter visual fault detection model. Specifically:

[0085] In the first training phase, the base class image in the normalized graph corresponding to the smart meter data obtained in step 101 is sampled, and the base class image is used as input to the smart meter visual fault detection model constructed with the DeFRCN framework.

[0086] The smart meter visual fault detection model based on multi-task routing and memory retrieval is driven to adaptively learn the features of the base class image, and the feature map of the base class image is extracted by the backbone network;

[0087] The base class image in the normalized graph corresponding to the smart meter data is input into the deep residual network ResNet101. The convolutional layer of the deep residual network extracts the features of the base class image to obtain the final feature map of the base class image.

[0088] The feature map is input into a proposal region network to obtain the coordinates of each proposal box. Based on the coordinates of each proposal box, the feature map is aligned to obtain the proposal region features.

[0089] The proposed region features are input into the multi-task dynamic routing, adaptively decoupling the task information of the proposed region features. Further, regression task features and classification task features of the proposed region features are extracted. These proposed region features are then input into the multi-task dynamic routing, and three five-layer deep residual networks are used to output regression task features. Classification task characteristics and shared task features ;

[0090] The classification intermediate features and shared intermediate features are input into a gated fusion network, and then pass through a fully connected layer. and activation function Output the corresponding branch scores, multiply the branch scores by the intermediate features to obtain their respective task features, and finally calculate the shared task features. and classification task features The summation and output of the final classification task features are then performed. The specific formula is as follows:

[0091]

[0092]

[0093]

[0094] The regression task features and shared task features are processed according to the above procedure to obtain the final regression task features. .

[0095] The classification task features are input into the visual prototype memory regularization module. First, the category labels of the classification task features are filled into the prompt word template "A photo of a {category}" to obtain prompt words. The filled prompt words are then processed by word segmentation to generate word units. First, fill the category labels into the prompt word template to obtain prompt words, such as "A photo of a cat";

[0096] Next, a special token is added to the prompt template "A photo of a cat" to obtain the preprocessed text. The formula is as follows:

[0097]

[0098] in and These represent the global token and the segmented token, respectively. The preprocessed text is segmented into tokens, resulting in a number of tokens composed of characters. The model's vocabulary is queried, and each token is converted into a corresponding ID sequence. Finally, attention masks are used to generate the final lexical units.

[0099] The first each word element The text encoder input to the pre-trained CLIP model Generate semantic vectors The formula is as follows:

[0100]

[0101] The first Text embedding Input to linear layer Perform mapping to obtain the mapped first... Text embedding Calculate the mapped first The text embedding and the first Characteristics of each classification task cosine similarity , The formula for calculating the magnitude of a vector is as follows:

[0102]

[0103]

[0104] Next, we calculate the alignment loss between cosine similarity and the true class label of the classification task features. The cross-entropy loss is optimized during model training to align visual features and text embeddings. The specific formula for the alignment loss is as follows:

[0105]

[0106] in and Representing the features of the classification task respectively and text embedding features The number of The probability that the true class label and the predicted class are the same as the true class label in the classification task. Represents the probability of the predicted category;

[0107] The regression task features are input into the regression task header, which outputs predicted coordinate information. The regression loss is calculated based on the true labels and predicted coordinate information. Similarly, the classification task features are input into the classification task header, which outputs predicted class information. The classification loss is calculated based on the true labels and predicted class information. As the true label for regression loss calculation and predicted coordinates The loss value between them is calculated using the following formula:

[0108]

[0109] in Represents the difference between the true label and the predicted coordinates;

[0110] Using the multivariate cross-entropy loss function The loss value used to calculate the classification loss is the loss between the true label and the predicted label of the category, and the formula is as follows:

[0111]

[0112] in It's a real label. It is the output class probability.

[0113] Gradient backpropagation is performed based on the loss values ​​of regression loss and classification loss to iteratively optimize the parameters of the smart meter visual fault detection model, thereby obtaining the trained first-stage smart meter visual fault detection model.

[0114] In the second training phase, the base class and new class balance images in the normalized graph corresponding to the smart meter data are sampled, and the balance image data is input into the smart meter visual fault detection model constructed with the DeFRCN framework.

[0115] The smart meter visual fault detection model based on multi-task routing and memory retrieval adaptively learns each part of the multi-task information of the suggested features, and the backbone network extracts the feature maps of the base class and new class balanced images.

[0116] The feature map is input into a proposal region network to obtain the coordinates of each proposal box. Based on the coordinates of each proposal box, the feature map is aligned to obtain the proposal region features.

[0117] The proposed region features are input into the multi-task dynamic routing, which adaptively decouples the task information of the proposed region features, and further extracts the regression task features and classification task features of the proposed region features;

[0118] The classification task features are input into the semantic refinement module of the visual prototype. The scaled dot product attention between the memory items in the memory module and the classification task features is calculated. First, a memory prototype is generated for each category by averaging the memory items in each memory queue of the memory module. Then, the classification task features are calculated. With this memory prototype The scaled dot product attention is applied between the attention points to obtain the attention-weighted features for the classification task, as shown in the following formula:

[0119]

[0120] in This is represented as belonging to the first... Categories For activation function, Indicates the weighted number of... One characteristic, Indicates the number of memory prototypes. The dimension representing the prototype of memory.

[0121] Finally, we obtain the attention-weighted task features;

[0122] The regression task features are input into the regression task header to output the predicted coordinate information, and the regression loss is calculated based on the true label and the predicted coordinate information.

[0123] The classification task features are input into the classification task head to output the predicted category information, and the classification loss is calculated based on the true label and the predicted category information.

[0124] Gradient backpropagation is performed based on the loss values ​​of regression loss and classification loss to iteratively optimize the parameters of the smart meter visual fault detection model, resulting in the final trained smart meter visual fault detection model.

[0125] Figure 2 The training process of the smart meter visual fault detection method based on multi-task routing and memory retrieval is as follows: the meter fault dataset is divided into training set and validation set, and the training set and validation set are divided into base class and new class. Data normalization is performed on all image data to ensure that the pixel value range of the image is [0,1].

[0126] During the base class training phase, the text encoder is frozen, and the base class image data is input into the detection model, whereby the backbone network obtains the feature maps of the images.

[0127] The feature map is input into the proposal box generation network to generate preliminary proposal boxes for the image, and the proposal region features are generated by ROI alignment operation according to the proposal boxes.

[0128] It is suggested that regional features be input into regression routing, shared routing, and classification routing to obtain corresponding task features. The gated fusion network adaptively allocates the task features and outputs the adaptively fused features.

[0129] The features obtained by fusing shared routes and regression routes are input into the regression task header, and the final suggestion box coordinate information is output.

[0130] The designed text prompts are input into the frozen text encoder, and the embedded text is output.

[0131] The classification task features output by the classification route are aligned with the text embedding, and the aligned visual features are stored in the memory module.

[0132] The features fused from shared routing and regression routing are input into the regression task head, which outputs the final suggestion box coordinate information. The features fused from shared routing and classification routing are input into the classification task head, which outputs the final suggestion box category probability.

[0133] Calculate the L1 loss between the proposal box coordinates and the ground truth coordinates, and the cross-entropy loss between the proposal box class probability and the ground truth class label, and optimize the two losses as the base class is trained;

[0134] During the new class fine-tuning stage, the text encoder is frozen, and the base class and new class balancing data are input into the detection model, and the backbone network obtains the feature map of the image.

[0135] The feature map is input into the proposal box generation network to generate preliminary proposal boxes for the image, and the proposal region features are generated by ROI alignment operation according to the proposal boxes.

[0136] It is suggested that regional features be input into regression routing, shared routing, and classification routing to obtain corresponding task features. The gated fusion network adaptively allocates the task features and outputs the adaptively fused features.

[0137] The features obtained by fusing shared routes and regression routes are input into the regression task header, and the final suggestion box coordinate information is output.

[0138] The classification task features output by the classification route are aligned with the text embedding, and the scaling attention between the aligned visual features and the memory items in the memory module is calculated to obtain the attention-weighted feature map.

[0139] The features obtained by fusing the shared route and the regression route are input into the regression task head, which outputs the final suggestion box coordinate information. The attention-weighted feature map is input into the classification task head, which outputs the probability of the corresponding model's predicted category.

[0140] Calculate the L1 loss between the proposal box coordinates and the ground truth coordinates, and the cross-entropy loss between the class probability and the ground truth class label, and optimize the two losses as the training process fine-tunes for new classes.

[0141] Figure 3 This is a schematic diagram illustrating the implementation process and subsequent processing of multi-task dynamic routing. First, the suggested region features are input into the multi-task dynamic routing system, and then three five-layer deep residual networks are used to output regression task features. Classification task characteristics and shared task features ;

[0142] The classification intermediate features and shared intermediate features are input into a gated fusion network, and then pass through a fully connected layer. and activation function Output the corresponding branch scores, multiply the branch scores by the intermediate features to obtain their respective task features, and finally calculate the shared task features. and classification task features The summation and output of the final classification task features are then performed. The specific formula is as follows:

[0143]

[0144]

[0145]

[0146] The regression intermediate features The input is fed into a gated fusion network and passes through a fully connected layer. and activation function Output the corresponding branch scores, multiply the branch scores by the intermediate features to obtain their respective task features, and finally calculate the shared task features. and regression task features The summation and output of the final classification task features are then performed. The specific formula is as follows:

[0147]

[0148] .

[0149] Figure 4 This is a schematic diagram illustrating the implementation process and subsequent processing of the visual prototype memory regularization module. First, the category labels of the classification task features are filled into the prompt word template "A photo of a {category}" to obtain prompt words. The filled prompt words are then processed by word segmentation to generate word units. First, fill the category labels into the prompt word template to obtain prompt words, such as "A photo of a cat";

[0150] Next, a special token is added to the prompt template "A photo of a cat" to obtain the preprocessed text. The formula is as follows:

[0151]

[0152] in and These represent the added global token and segment token, respectively. The preprocessed text is segmented into several tokens composed of characters. The model's vocabulary is queried, and each token is converted into a corresponding ID sequence. Finally, the attention mask is used to generate the final ID sequence.

[0153] The preprocessed text is segmented to obtain several tokens composed of characters. The vocabulary of the model is queried, each token is converted into a corresponding ID sequence, and lexical units are generated by attention mask.

[0154] No. each word element The text encoder input to the pre-trained CLIP model Generate semantic vectors The formula is as follows:

[0155]

[0156] The first Text embedding Input to linear layer Perform mapping to obtain the mapped first... Text embedding Calculate the mapped first The text embedding and the first Characteristics of each classification task cosine similarity , The formula for calculating the magnitude of a vector is as follows:

[0157]

[0158]

[0159] Next, we calculate the alignment loss between cosine similarity and the true class label of the classification task features. The cross-entropy loss is optimized during model training to align visual features and text embeddings. The specific formula for the alignment loss is as follows:

[0160]

[0161] in and Representing the features of the classification task respectively and text embedding features The number of The probability that the true class label and the predicted class are the same as the true class label in the classification task. Represents the probability of the predicted category;

[0162] Figure 5 This is a schematic diagram illustrating the implementation process and subsequent processing of the semantic refinement module of the visual prototype. First, a memory prototype is generated for each category by averaging the memory items in each memory queue of the memory module. Then, the classification task features are calculated. With this memory prototype The scaled dot product attention is applied between the attention points to obtain the attention-weighted features for the classification task, as shown in the following formula:

[0163]

[0164] in This indicates that the feature belongs to the first... Categories For activation function, Indicates the weighted number of... One characteristic, Indicates the number of memory prototypes. The dimension representing the prototype of memory.

[0165] In a specific embodiment, visual fault datasets under different categories of smart meters were used for testing. The dataset collected image data of a total of 10 types of smart meters. Due to the influence of human and environmental factors during the collection process, there were cases of missing features and incorrect labels in the collected images. If used directly, the model would have difficulty learning the correct features. After data cleaning, a total of 1163 fault sample data were obtained. The dataset was divided into training and test sets in a 6:4 ratio.

[0166] Table 1. Datasets used in specific embodiments.

[0167] serial number Class 1 Class 2 Class 3 Class 4 Class 5 Class 6 Class 7 Class 8 Category 9 Category 10 Sample size 68 100 239 190 124 190 140 53 25 34 Sample type Watch damaged Burn-in Lacking seal The screen does not display Alarm light malfunctions Carrier module missing Battery undervoltage Wiring error Screen fogging Electricity meter tripped

[0168] To verify the effectiveness of the proposed algorithm, four mainstream few-sample target detection methods were compared in this embodiment of the invention; as shown in Table 2, the method in this embodiment of the invention is represented by MDRS (Multi-task Dynamic Routing and Retrieval Strategy).

[0169] Table 2. Algorithms compared in specific embodiments.

[0170] Method Name Publications and Years DeFRCN ICCV, 2021 MFDC ECCV, 2022 D&R AAAI, 2023 BSDet TMM, 2024

[0171] In this embodiment of the invention, the category AP index is used to evaluate the classification performance of the algorithm. The value ranges from 0 to 100. The larger the value, the better the classification performance of the model.

[0172] Table 3 shows a comparison of the experimental results of the AP of the embodiments of the present invention and the mainstream few-sample object detection methods under various fault categories of smart meters. It can be seen that the visual fault detection method of smart meters based on vision-language cross-modal learning of the present invention outperforms the AP of other methods in all categories and achieves the best detection effect. The results in Table 3 show that the visual fault detection effect of the method of the embodiments of the present invention is better than the mainstream few-sample image detection method and can obtain a higher recognition accuracy.

[0173] Table 3 Experimental results of MDRS and mainstream methods on AP index under various fault categories of smart meters

[0174] DeFRCN MFDC D&R BSDet MDRS Category 1 86.6 87.1 88.3 87.3 89.2 Category 2 70.3 72.8 72.1 77.2 79.0 Category 3 69.2 70.3 68.7 76.0 77.5 Category 4 94.1 95.8 96.3 93.5 96.1 Category 5 93.5 95.1 94.5 95.3 95.9 Category 6 89.1 90.2 89.7 89.3 91.6 Category 7 92.0 92.5 92.0 100 100 Category 8 67.8 72.9 73.5 40.1 77.3 Category 9 71.1 75.4 75.9 82.3 85.5 Category 10 93.8 94.4 94.6 87.0 95.6

[0175] This application provides a smart meter visual fault detection system based on multi-task routing and memory retrieval, such as... Figure 6 As shown, it includes a data processing module 601, a model training module 602, and a recognition module 603;

[0176] The data processing module 601 is used to: perform normalization preprocessing on the smart meter image data to obtain a normalized image corresponding to the smart meter image data;

[0177] The model training module 602 is used to: input the normalized image corresponding to the smart meter data into the detection model, perform progressive training on the smart meter image data, and obtain a trained smart meter visual fault detection model.

[0178] The recognition module 603 is used to: input the image to be tested into the trained meter visual fault model to complete the fault category detection.

[0179] This application provides an electronic device, such as... Figure 7 As shown, the electronic device includes a bus, a processor, a memory, and a communication interface. It may also include input / output interfaces and a display device. The various functional units can communicate with each other via the bus. The memory stores computer programs, and the processor executes the programs stored in the memory, performing the meter fault classification method described in the above embodiment.

[0180] This application provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the method for classifying electricity meter faults. Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented in hardware or with the aid of software and necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or external hard drive), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various implementation scenarios of this application.

[0181] In summary, the embodiments of the present invention have the following beneficial effects:

[0182] In the technical solution implemented in this invention, based on the visual characteristics of visible faults in electricity meters, visual-language cross-modal learning is used to associate fault manifestations with professional diagnostic knowledge, and a scalable memory bank is constructed to store typical fault patterns. During actual deployment, memory-enhanced inference utilizes prototype features to constrain the current detection results, improving the model's generalization ability. Furthermore, a feature fusion mechanism based on scaling dot product attention adaptively adjusts the contribution of multi-source information, enabling the system to accurately identify novel fault patterns while maintaining stable detection capabilities for common faults, thereby significantly improving the accuracy and robustness of fault detection.

[0183] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A visual fault detection method for smart meters based on multi-task routing and memory retrieval, characterized in that, Includes the following steps: Normalization preprocessing is performed on the smart meter image data to obtain the normalized image corresponding to the smart meter image data; During the base class training phase, the normalized map corresponding to the smart meter image data is input into the backbone network, and then the feature map obtained by the backbone network is input into the proposal region network to obtain the features of the region to be tested. The proposed region features are input into the multi-task dynamic routing, which adaptively decouples the task information of the proposed region features, and further extracts the regression task features and classification task features of the proposed region features; The classification task features are input into the visual prototype memory regularization module to obtain a memory module composed of visual prototypes for each category. The regression task features and classification task features are input into the regression task head and the classification task head, respectively, and the predicted coordinate information and category information are output. The loss is calculated based on the true label and the predicted information. In the new fine-tuning stage, the proposed regional features are obtained from the regional proposal network of the first-stage smart meter visual fault detection model, and input into the multi-task dynamic routing to extract the regression task features and classification task features of the proposed regional features. The classification task features are input into the semantic refinement module of the visual prototype, the scaling dot product attention between the memory items of the memory module and the classification task features is calculated, and the attention-weighted classification task features are obtained. The regression task features and the attention-weighted classification task features are input into the regression task head and the classification task head, respectively, to output the predicted category information and coordinate information. The loss is calculated based on the true label and the predicted information, and the parameters of the smart meter visual fault detection model are iteratively optimized based on the loss value to obtain the final trained smart meter visual fault detection model. The image data of the meter under test is input into the finally trained smart meter visual fault detection model to complete the fault category detection.

2. The smart meter visual fault detection method based on multi-task routing and memory retrieval according to claim 1, characterized in that, The step of performing normalization preprocessing on the smart meter image data to obtain the normalized image corresponding to the smart meter image data includes: Each smart meter image in the data is labeled with a fault category to form a training set; The size of all samples in the training set is adjusted to R×R, where R is the pixel size of the image; Calculate the mean and standard deviation of pixel values ​​for each channel for all samples in the training set, so that the pixel values ​​satisfy a distribution with a mean of 0 and a standard deviation of 1, to obtain the normalized image corresponding to the smart meter image data.

3. The method for visual fault detection of smart meters based on multi-task routing and memory retrieval according to claim 1, characterized in that, The steps of inputting the proposed region features into the multi-task dynamic routing, adaptively decoupling the task information of the proposed region features, and further extracting the regression task features and classification task features of the proposed region features include: The suggested region features are input to a multi-task dynamic routing, which utilizes three five-layer deep residual networks to output regression intermediate features f reg , classification intermediate features f cls , and shared intermediate features f share ; The classification intermediate features and shared intermediate features are input into a gated fusion network. After passing through a fully connected layer (FC(·)) and an activation function (Sigmoid(·), the corresponding branch scores are output. The branch scores are multiplied by the intermediate features to obtain their respective task features. Finally, the shared task feature f is calculated. share_feat and classification task features f cls_feat The summation and output of the final classification task features f cls_head The specific formula is as follows: f share_feat =f share ·Sigmoid(FC(f share )) f cls_feat =f cls ·Sigmoid(FC(f cls )) f cls_head =f share_feat +f cls_feat The regression intermediate feature f reg The input is fed into a gated fusion network, passes through a fully connected layer (FC(·)) and an activation function (Sigmoid(·)), and outputs corresponding branch scores. These branch scores are then multiplied by intermediate features to obtain their respective task features. Finally, the shared task feature f is calculated. share_feat and regression task features f reg_feat The summation and output of the final classification task features f reg_head The specific formula is as follows: f reg_feat =f reg ·Sigmoid(FC(f reg )) f reg_head =f share_feat +f reg_feat 。 4. The smart meter visual fault detection method based on multi-task routing and memory retrieval according to claim 1, characterized in that, The classification task features are input into the visual prototype memory regularization module to obtain a memory module composed of visual prototypes for each category. The regression task features and classification task features are input into the regression task head and classification task head to output predicted coordinate information and category information. The steps for calculating the loss based on the true label and predicted information are as follows: First, fill the category labels of the classification task features into the prompt word template "Aphoto of a{category}" to obtain the prompt words. Then, perform word segmentation on the filled-in prompt words to generate word tokens. The i-th token i The pre-trained CLIP model's text encoder (Text_Encoder(·)) generates semantic vectors (text_embedding). i The formula is as follows: text_embedding i =Text_Encoder(tokens i ) Embed the i-th text into text_embedding i The input is mapped to the linear layer Projector(·) to obtain the i-th text embedding text_norm. i Calculate the embedded text_norm of the j-th text after mapping. i With the i-th classification task feature vis i cosine similarity Sim i,j , ||·|| represents the magnitude of a vector, and the specific formula is as follows: text_norm j =Projector(text_embedding j ) Next, calculate the alignment loss L of the true class labels of the classification task features. align The cross-entropy loss is optimized during model training to align visual features and text embeddings. The specific formula for the alignment loss is as follows: Where M and N represent the number of classification task features (vis) and text embedding features (text_norm), respectively, e kii e represents the probability that the true class label and the predicted class are the same for the features of a classification task. kij Represents the probability of the predicted category; In each round of training, the classification task features are stored as visual prototypes in a memory matrix of size [C,D], where C is the number of categories and D is the category vector dimension. When the loss approaches convergence, the final memory matrix is ​​the memory module composed of the visual prototypes of each category. Input the regression task features and classification task features into the classification task header and regression task header to obtain the predicted coordinates and predicted class labels; Using L reg As the true label y in regression loss calculation true and predicted coordinates y pred The loss value between them is calculated using the following formula: Where x = y pred -y true Represents the difference between the true label and the predicted coordinates; Using the multivariate cross-entropy loss function L cls The loss value used to calculate the classification loss is the loss between the true label and the predicted label of the category, and the formula is as follows: Where y i It's a real label. It is the output class probability.

5. The smart meter visual fault detection method based on multi-task routing and memory retrieval according to claim 4, characterized in that, The steps for first filling the category labels of the classification task features into the prompt word template "Aphoto of a{category}" to obtain prompt words, and then performing word segmentation on the filled prompt words to generate tokens are as follows: First, fill the category labels into the prompt word template to obtain prompt words, such as "A photo of a cat"; Next, a special token is added to the prompt template "Aphoto of a cat" to obtain the preprocessed text I, whose formula is as follows: I="[CLS],'A','photo','of','a','cat',[SEP]" [CLS] and [SEP] represent the global token and segmented token, respectively. The preprocessed text is segmented to obtain several tokens composed of characters. The model's vocabulary is queried, and each token is converted into a corresponding ID sequence. Finally, the attention mask is used to generate the final ID sequence.

6. The smart meter visual fault detection method based on multi-task routing and memory retrieval according to claim 1, characterized in that, The steps for inputting the classification task features into the semantic refinement module of the visual prototype, calculating the scaled dot product attention between the memory items of the memory module and the classification task features, and obtaining the attention-weighted classification task features are as follows: First, a memory prototype is generated for each category by averaging the number of memory items in each memory queue of the memory module. Then, the classification task features are calculated. With this memory prototype The scaled dot product attention is applied between the attention points to obtain the attention-weighted features for the classification task, as shown in the following formula: Where i represents that the feature belongs to the i-th category, Softmax(·) is the activation function, and attention_memory j Let d represent the j-th feature after weighting, K represent the number of memory prototypes, and d represent the number of prototypes. m The dimension representing the prototype of memory.

7. The method for visual fault detection of smart meters based on multi-task routing and memory retrieval according to claim 1, characterized in that, The step of inputting the image data of the meter under test into the trained smart meter visual fault detection model to complete the fault category detection includes: The image data of the meter under test is input into the trained smart meter visual fault detection model. After feature extraction, regression and classification tasks in the feature learning stage, the predicted probability distribution and bounding box coordinates are obtained. The category with the highest probability is the predicted category, and the output bounding box coordinates are the location coordinates of the meter fault.

8. A visual fault detection system for smart meters based on multi-task routing and memory retrieval, characterized in that, It includes a data processing module, a model training module, and a recognition module; The data processing module is used to preprocess the smart meter image data to obtain the normalized image corresponding to the smart meter image data; The model training module uses the normalized meter image as input to the detection model to progressively train the smart meter visual fault detection model, thereby obtaining a trained smart meter visual fault detection model. The recognition module is used to input the image data of the meter under test into the trained smart meter visual fault detection model to complete the detection of fault categories.

9. An electronic device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images