Inspection image recognition method, device and equipment of power system and storage medium

Abstract

The invention discloses an inspection image recognition method, device and equipment for a power system and a storage medium. The method comprises the following steps: acquiring an inspection image of a to-be-recognized power system area; inputting the inspection image into an image category identification model, extracting to obtain a first image feature, determining a first similarity between the first image feature and each category prototype according to each category prototype in the image category identification model, determining a corresponding category prototype according to the first similarity, and determining a second similarity between the first image feature and each category prototype according to the corresponding category prototype; taking the information as an equipment state category of the inspection image; wherein the image category recognition model is obtained by training an initial image category recognition model through a power system region inspection image sample set. The technical problems that in the prior art, the accuracy of electric power system area inspection image recognition is low, the inspected image cannot be directly recognized and analyzed, and the electric power system area inspection image cannot be effectively recognized are solved.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a method, device, equipment and storage medium for inspection image recognition of a power system. Background Art

[0002] With the advancement of technology, drones have been widely used in many fields, including power system inspection. Drones provide an efficient and safe inspection method. High-quality image data captured by drones can help engineers inspect equipment and diagnose faults from remote locations.

[0003] For the captured image data, models such as convolutional neural networks (CNNs) and visual transformers (ViTs) are usually used for training to obtain recognition models for image recognition. However, when training, existing recognition models need to be trained and fine-tuned with a large amount of labeled data to improve the accuracy and generalization of image recognition. They cannot directly recognize and analyze the inspected images. They need to identify and process the corresponding feature devices of the images before they can recognize the corresponding states. At the same time, when the training data is small, the model will be overfitted, resulting in insufficient generalization of the model. In the existing field of power system inspection image recognition, the data scale of the training samples is small, and the recognition model obtained after training with a small number of sample data cannot effectively recognize the power system inspection images. Summary of the invention

[0004] The present invention provides a method, device, equipment and storage medium for recognizing inspection images of an electric power system, so as to solve the technical problems in the prior art of low accuracy of regional inspection image recognition of the electric power system, inability to directly recognize and analyze the inspected images, and inability to effectively recognize regional inspection images of the electric power system.

[0005] In order to solve the above technical problems, an embodiment of the present invention provides an inspection image recognition method for a power system, comprising:

[0006] Acquire inspection images of the power system area to be identified;

[0007] Input the inspection image into an image category recognition model to extract a first image feature, and determine a first similarity between the first image feature and each category prototype according to each category prototype in the image category recognition model, and determine a corresponding category prototype according to the first similarity as the device status category of the inspection image;

[0008] The image category recognition model is obtained by training an initial image category recognition model using a sample set of power system area inspection images.

[0009] As a preferred solution, the image category recognition model is obtained by training the initial image category recognition model through the power system area inspection image sample set, specifically including:

[0010] Acquire a sample set of inspection images of a power system area; wherein the sample set of inspection images includes a plurality of sample images and a device status category label corresponding to each sample image;

[0011] Based on the multimodal pre-training model structure, build an initial image category recognition model;

[0012] Iteratively training the initial image category recognition model with a power system area inspection image sample set, so that during the iterative training process, the first category loss and the deviation loss of the initial image category recognition model are calculated in sequence;

[0013] An image category recognition model is generated until the first category loss and the deviation loss are minimized as a whole.

[0014] As a preferred solution, the initial image category recognition model is constructed based on the multimodal pre-training model structure, specifically including:

[0015] Constructing an initial multimodal pre-training model; wherein the initial multimodal pre-training model includes a first visual encoder and a text encoder;

[0016] The initial multimodal pre-trained model is pre-trained using a sample set of power system area inspection images, so that the first visual encoder learns and extracts a second image feature of the sample image, the text encoder extracts text features of the state category label, generates prototypes of each category according to the text features, and calculates the second category loss of the initial multimodal pre-trained model;

[0017] Until the loss of the second category is minimized, generating a pre-trained multimodal pre-trained model;

[0018] Constructing a deviation feature classifier according to the pre-trained multimodal pre-trained model;

[0019] Based on the pre-trained multimodal pre-trained model and the deviation feature classifier, an initial image category recognition model is constructed.

[0020] As a preferred solution, constructing a deviation feature classifier based on the pre-trained multimodal pre-training model specifically includes:

[0021] Determining, according to the pre-trained multimodal pre-trained model, context features that are irrelevant to each category prototype as deviation feature prototypes;

[0022] According to the deviation feature prototype, a deviation feature classifier is constructed.

[0023] As a preferred solution, the initial image category recognition model is constructed based on the pre-trained multimodal pre-trained model and the deviation feature classifier, specifically including:

[0024] Freeze the model parameters of the text encoder in the current pre-trained multimodal pre-trained model;

[0025] Based on the model parameters of the first visual encoder in the currently pre-trained multimodal pre-trained model, construct a second visual encoder;

[0026] An initial image category recognition model is constructed based on the first visual encoder, the second visual encoder, the text encoder and the deviation feature classifier.

[0027] As a preferred solution, the text encoder extracts text features of the state category label and generates prototypes of each category according to the text features, specifically including:

[0028] Extracting text features corresponding to the category state labels of all sample images through the text encoder;

[0029] Based on all text features, determine the mean of all text features;

[0030] According to the mean of the text features, a prototype of each category is generated.

[0031] As a preferred solution, before the text encoder extracts the text features corresponding to the category status labels of all sample images, it also includes:

[0032] Obtain a number of text prompt templates; wherein each text prompt template corresponds to a category state label;

[0033] Generate a number of text prompt labels according to the device status category labels corresponding to the various sample images and a number of text prompt templates;

[0034] A plurality of text prompt tags are input into a text editor so that the text editor extracts text features of all the text prompt tags.

[0035] As a preferred solution, the calculating the second category loss of the initial multimodal pre-training model specifically includes:

[0036] The second image features of the sample image are learned and extracted by the first visual encoder, and the second similarities between each second image feature and each category prototype are calculated by combining the text features of the state category label extracted by the text encoder to generate each category prototype;

[0037] According to the second similarity between each second image feature and each category prototype, a first probability distribution of the sample image belonging to each category is calculated;

[0038] According to the first probability distribution, a second category loss of the initial multimodal pre-training model is calculated.

[0039] As a preferred solution, the calculation of the first category loss of the initial image category recognition model specifically includes:

[0040] extracting a third image feature of the sample image through a second visual encoder in the initial image category recognition model;

[0041] Extract text features of state category labels through the text editor in the initial image category recognition model;

[0042] Generate prototypes of each category based on the text features of the extracted state category labels;

[0043] The first category loss is calculated based on the third image feature and the prototypes of each category.

[0044] As a preferred solution, the first category loss is calculated according to the third image feature and each category prototype, specifically including:

[0045] Calculating a third similarity between each third image feature and each category prototype;

[0046] According to the third similarity between each third image feature and each category prototype, a second probability distribution of the sample image belonging to each category is calculated;

[0047] According to the second probability distribution, a first category loss of the initial image category recognition model is calculated.

[0048] As a preferred solution, the calculation obtains the deviation loss of the initial image category recognition model, specifically including:

[0049] extracting a third image feature of the sample image through a second visual encoder in the initial image category recognition model;

[0050] The deviation loss is calculated according to the second image feature and the third image feature.

[0051] As a preferred solution, during the iterative training process, when unclassified text features are used as deviation feature prototypes, the deviation loss is adjusted according to the current deviation feature prototype through the deviation feature classifier.

[0052] Accordingly, the present invention also provides a power system area inspection image recognition device, comprising: an acquisition module and a recognition module

[0053] The acquisition module is used to acquire the inspection image of the power system area to be identified;

[0054] The recognition module is used to input the inspection image into the image category recognition model, extract the first image feature, and determine the first similarity between the first image feature and each category prototype according to each category prototype in the image category recognition model, and determine the corresponding category prototype according to the first similarity as the device status category of the inspection image;

[0055] The image category recognition model is obtained by training an initial image category recognition model using a sample set of power system area inspection images.

[0056] Correspondingly, the present invention also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, it implements the inspection image recognition method of the power system as described in any one of the above.

[0057] Correspondingly, the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the inspection image recognition method of the power system as described in any one of the above.

[0058] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0059] The technical solution of the present invention obtains inspection images of the power system area to be identified, and then inputs the inspection images into an image category recognition model obtained by training an initial image category recognition model through a sample set of inspection images of the power system area, thereby extracting a first image feature, and calculating a corresponding first similarity based on each category prototype, so as to determine a final proposed category prototype, which is output as the equipment status category of the inspection image, and finally realizes accurate recognition and analysis of the inspection images of the power system area to be identified, thereby avoiding the inability to effectively identify the inspection images of the power system area.

[0060] Furthermore, by acquiring a sample set of power system area inspection images, an initial image category recognition model is constructed based on a multimodal pre-training model, and the initial image category recognition model is trained with the sample set of power system area inspection images. During the training process, the first category loss and deviation loss of the initial image category recognition model are calculated, and when the first category loss and deviation loss are minimized as a whole, an image category recognition model is generated; by introducing the calculation of deviation loss in the process of constructing the image category recognition model, the calculation based on deviation loss can avoid the problem of model overfitting in the case of few samples, and combined with the calculation of the first category loss, the trained image category recognition model has both recognition accuracy and generalization. When the power system area inspection images to be identified are identified based on the trained image category recognition model, the accuracy of power system area inspection image recognition can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 : A flowchart of a method for inspection image recognition of a power system provided by an embodiment of the present invention;

[0062] Figure 2 : A flowchart of the steps of a method for constructing an image category recognition model in a method for recognizing inspection images of a power system provided by an embodiment of the present invention;

[0063] Figure 3 : A flowchart of the steps of a method for constructing an initial image category recognition model in a method for recognizing inspection images of a power system provided by an embodiment of the present invention;

[0064] Figure 4 : A schematic diagram of the structure of the initial image category recognition model provided by an embodiment of the present invention;

[0065] Figure 5 : A structural diagram of a power system area inspection image recognition device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] Embodiment 1

[0068] See also Figure 1 , a method for recognizing inspection images of a power system provided by an embodiment of the present invention, comprising the following steps S101-S102:

[0069] Step S101: obtaining an inspection image of a power system area to be identified.

[0070] In this embodiment, the equipment in the power system area can be inspected in sequence by inspection robots, drones, PDA mobile terminals or manual inspections, so that the power system area inspection images required for identification can be obtained.

[0071] In this embodiment, the power system area includes substations, booster stations, power stations, transmission lines, distribution networks, step-down stations, etc. The embodiment of the present invention can collect inspection images of power electronic equipment of substations, booster stations, power stations, transmission lines, distribution networks, step-down stations, etc. in the power system area, and then determine the status category of the power electronic equipment.

[0072] Step S102: Input the inspection image into an image category recognition model, extract the first image feature, and determine the first similarity between the first image feature and each category prototype according to each category prototype in the image category recognition model, and determine the corresponding category prototype according to the first similarity as the equipment status category of the inspection image; wherein the image category recognition model is obtained by training an initial image category recognition model through a sample set of power system area inspection images.

[0073] In this embodiment, the image category recognition model is obtained by training the initial image category recognition model with the power system area inspection image sample set, and the initial image category recognition model can be constructed based on the CLIP (Contrastive Language-Image Pre-Training, multimodal pre-training model) model structure. Further, the initial image category recognition model can also be constructed based on model structures such as ALIGN, Florence, FLAVA, ViT and ALBEF, among which the ALIGN (Alar-scaleImaGe and Noisy-text embedding) model is similar to the CLIP multimodal pre-training model, but uses a larger scale of training data; the Florence model is a visual base model proposed by Microsoft, which performs well in multiple downstream tasks; FLAVA is a unified visual, language and multimodal model proposed by Facebook; ViT is a pure Transformer architecture visual model that can be used as a visual feature extractor; the ALBEF model can be used to align the multimodal model of text and image representation.

[0074] It should be noted that the CLIP model can process image and text data at the same time. It consists of two parts: a visual encoder and a text encoder. Among them, the visual encoder is used to extract image features, and the text encoder is used to extract text features. The CLIP model can align visual and text representations into a unified embedding space through training methods such as contrastive learning, so that the CLIP model can achieve tasks such as zero-sample image classification. Therefore, due to the excellent zero-sample and few-sample capabilities of the CLIP model, the application scenario in this embodiment is more excellent.

[0075] In this embodiment, the power system area inspection image to be identified is input into the image category recognition model, so that the image category recognition model extracts the image features (i.e., the first image features mentioned above) in the power system area inspection image to be identified, wherein the identified and extracted image features may be the overall visual features, local area features, foreground / background information, low-level features (including texture, color, shape, etc.) and high-level features at the semantic level in the inspection image. At the same time, each category prototype in the image category recognition model is obtained, and the similarity between the extracted image features and each category prototype is calculated, and the category prototype with the highest similarity is used as the equipment status category of the power system area inspection image to be identified, and the image category recognition model outputs the equipment status category. Preferably, the category prototype can be a device and its corresponding current state. For example, for transformer equipment, the category prototype can be the degree of damage to the transformer, including: normal transformer, slightly damaged transformer, moderately damaged transformer and severely damaged transformer, etc. It can also be the environmental state of the transformer, including: dry transformer, water droplets on the transformer, ice / frost on the transformer, snow accumulation on the transformer and transformer pollution (oil, dust, etc.), or it can be the load state of the transformer, including: normal temperature of the transformer, slightly overheated transformer and severely overheated transformer; for circuit breakers, the category prototype can be the degree of damage to the circuit breaker, including: normal circuit breaker, slightly damaged circuit breaker, moderately damaged circuit breaker and severely damaged circuit breaker, etc. It can also be the mechanical state of the circuit breaker, including: normal circuit breaker, loose circuit breaker, deformed circuit breaker and broken circuit breaker, etc.

[0076] Furthermore, the similarity between the first image feature and the prototypes of each category may be quantified based on the cosine similarity between the first image feature and the prototypes of each category, or may be other vector distance calculation methods, such as Euclidean distance.

[0077] Furthermore, the equipment status categories may include, but are not limited to, damage degree, environmental status, load status, operating status, insulation status, mechanical status, and corrosion status; wherein, damage degree includes normal, slight damage, and severe damage; environmental status includes: dry, water droplets, ice / frost, snow accumulation, and pollution (such as dust, oil, etc.); load status includes: normal temperature, slight overheating, and severe overheating; operating status includes: normal operation, abnormal vibration, and abnormal sound; insulation status includes: normal, slight discharge, severe discharge, and insulation damage; mechanical status includes: normal, loose, deformed, and broken; corrosion status includes: no corrosion, slight corrosion, and severe corrosion. The above categories are pre-defined, and each category corresponds to a specific status of a device, that is, by identifying the inspection image of the power system area to be identified, the specific status of the device in the inspection image is determined.

[0078] Exemplarily, inspection images of the power system area to be identified are acquired by photographing with a drone or other equipment, the images are input into a visual encoder of an image category recognition model, a first image feature is extracted, and the cosine similarity between the first image feature and each category prototype in the model is calculated, wherein the image category recognition model stores several category prototypes, and then the category prototype with the highest cosine similarity is found, the corresponding category prototype is used as the final recognition result, and the recognized equipment status category is output.

[0079] It can be understood that by identifying the inspection image, the image features in the corresponding inspection image can be determined, and the similarities between the image features and each category prototype can be determined by the category prototype in the image features, and then the category prototype with the highest similarity is output, that is, the results including device information and device status information are directly output, and there is no need to perform device identification, device status analysis, etc. on a large amount of image data. Moreover, through the similarity between each category prototype and the image feature, the accurate identification and analysis of the inspection image of the power system area to be identified can be improved, and the user experience is improved.

[0080] Implementing the above embodiments has the following effects:

[0081] The technical solution of the present invention obtains inspection images of the power system area to be identified, and then inputs the inspection images into an image category recognition model obtained by training an initial image category recognition model through a sample set of inspection images of the power system area, thereby extracting a first image feature, and calculating a corresponding first similarity based on each category prototype, so as to determine a final proposed category prototype, which is output as the equipment status category of the inspection image, and finally realizes accurate recognition and analysis of the inspection images of the power system area to be identified, thereby avoiding the inability to effectively identify the inspection images of the power system area.

[0082] Embodiment 2

[0083] See also Figure 2 , which is a method for constructing an image category recognition model provided by an embodiment of the present invention, wherein the image category recognition model is obtained by training an initial image category recognition model through a power system area inspection image sample set, and specifically includes the following steps S201-S204:

[0084] Step S201: Acquire a sample set of inspection images of a power system area; wherein the sample set of inspection images includes a plurality of sample images and a device status category label corresponding to each sample image.

[0085] In this embodiment, the equipment in the power system area can be inspected in sequence by means of inspection robots, drones, PDA mobile terminals or manual inspections, so that inspection image samples of the power system area required for identification can be obtained, and the inspection image samples can be annotated with equipment status category labels by manual labeling, thereby constructing a complete inspection image sample set of the power system area.

[0086] Step S202: constructing an initial image category recognition model based on the multimodal pre-trained model structure.

[0087] In this embodiment, based on the multimodal pre-trained model structure, the initial model of the multimodal pre-trained model structure can be pre-trained, and the visual encoder and text encoder in the initial model are mobilized to pre-train the inspection image sample set, so that the corresponding initial model is set according to the actual model parameters of the pre-trained visual encoder and text encoder, and the initial model of the multimodal pre-trained model structure with the model parameters set is used as the constructed initial image category recognition model.

[0088] Furthermore, the initial image category recognition model includes a visual encoder, a text encoder and a deviation feature classifier. Specifically, the visual encoder includes a first visual encoder and a second visual encoder. The first visual encoder is generated by pre-training the image sample by the initial image category recognition model, and the corresponding model parameters are fine-tuned to generate and set the second visual encoder, so that the initial image category recognition model has better recognition accuracy and generalization.

[0089] Step S203: iteratively training the initial image category recognition model with the power system area inspection image sample set, so that during the iterative training process, the first category loss and the deviation loss of the initial image category recognition model are calculated in sequence.

[0090] In this embodiment, the constructed initial image category recognition model, that is, the first visual encoder, the second visual encoder, the deviation feature classifier and the text editor in the initial image category recognition model, is iteratively trained according to the power system area inspection image sample set, so that in the process of iterative training, the first category loss and the deviation loss of the corresponding initial image category recognition model are calculated in turn, and then the first category loss and the deviation loss of the initial image category recognition model are trained to the minimum, that is, the model has good classification ability, and at the same time can ensure the robustness of the model deviation association, and finally when the first category loss and the deviation loss are both minimized, that is, the overall minimum, an image category recognition model is generated.

[0091] As a preferred solution of this embodiment, the calculation of the first category loss of the initial image category recognition model specifically includes:

[0092] The third image features of the sample image are extracted through the second visual encoder in the initial image category recognition model; the text features of the state category labels are extracted through the text editor in the initial image category recognition model; each category prototype is generated according to the extracted text features of the state category labels; and the first category loss is calculated according to the third image features and each category prototype.

[0093] As a preferred solution of this embodiment, the first category loss is calculated according to the third image feature and each category prototype, specifically including:

[0094] Calculate the third similarity between each third image feature and each category prototype respectively; calculate the second probability distribution that the sample image belongs to each category based on the third similarity between each third image feature and each category prototype respectively; calculate the first category loss of the initial image category recognition model based on the second probability distribution.

[0095] In this embodiment, the third image feature of the input sample image is extracted by the second visual encoder in the initial image category recognition model, and the text encoder is based on its frozen parameters. Therefore, the text features extracted by the text encoder here are the same as the text features extracted during pre-training, and the determined prototypes of each category are also the same. Based on this, the first category loss between the text encoder and the visual encoder can be determined according to the third image feature and each category prototype. The calculation of this category loss can be determined by calculating the cosine similarity between the third image feature and each category prototype. Specifically, the functional expression of the first category loss is:

[0096]

[0097] Where D represents the training data set, xi represents the input image sample, yi represents the true category label corresponding to xi, and f t represents the second visual encoder, f t (x i ) represents the third image feature extracted by the second visual encoder, Represents the prototype of category y, s(,) represents the cosine similarity function, and Y represents the set of all possible categories.

[0098] It should be noted that the above formula calculates the cross entropy loss, which is used to measure the gap between the category distribution predicted by the model and the true label. By minimizing this loss function, the classification accuracy of the model can be improved.

[0099] It should be noted that cosine similarity is an indicator that measures the degree of directional matching between two non-zero vectors. The closer its value is to 1, the more similar the two vectors are. That is, the closer the cosine similarity between an image feature and a prototype of any category is to 1, the more similar the image feature is to the prototype of the category.

[0100] Furthermore, the category distribution of the input image can be determined by calculating the cosine similarity between the third image feature and each category prototype. Specifically, by taking the cosine similarity between each category prototype and the image feature as the probability that the image belongs to the category, the category distribution of the image can be obtained, which describes the probability that the image belongs to each category. Finally, the first category loss is calculated based on the cross entropy loss, so that the initial image category recognition model can have better classification ability through the category loss calculation and the model adjusted to the second visual encoder.

[0101] As a preferred solution of this embodiment, the calculation of the deviation loss of the initial image category recognition model specifically includes:

[0102] A third image feature of the sample image is extracted through a second visual encoder in the initial image category recognition model; and a deviation loss is calculated based on the second image feature and the third image feature.

[0103] As a preferred solution of this embodiment, during the iterative training process, when unclassified text features are used as deviation feature prototypes, the deviation loss is adjusted according to the current deviation feature prototype by the deviation feature classifier.

[0104] In this embodiment, in order to maintain the stability of the initial image category recognition model to the deviation feature, it can also be achieved by a feature identification alignment method, that is, by a deviation feature classifier. Through the deviation feature classifier, it can be ensured that the classification probability distribution of the deviation feature remains consistent during the fine-tuning process. Among them, the deviation loss of the model is determined according to the second image feature output by the first visual encoder and the third image feature output by the second visual encoder. In the model training process, the text-visual alignment capability in the multimodal pre-training model structure is used, and the unclassified text feature is used as the deviation feature prototype. Preferably, the unclassified text feature can be a text feature with an unclear description of the category (i.e., context), and then the image feature (the third image feature and the second image feature) and the deviation feature prototype are measured for similarity to determine the probability distribution of the current image on the deviation feature. By constraining the probability distribution of the second image feature extracted before fine-tuning and the probability distribution of the third image feature extracted after fine-tuning, the consistency of the deviation feature extracted by the model before and after fine-tuning is ensured, and the deviation loss of the image feature before and after fine-tuning is reduced, that is, the deviation loss between the second image feature output by the first visual encoder and the third image feature output by the second visual encoder is reduced, and the robustness of the model to deviation association after fine-tuning is ensured.

[0105] In this embodiment, the deviation loss L spurious The function expression is:

[0106]

[0107] Where D is the training data set, KL is the Kullback-Leibler divergence, P spurious (x; f 0 ) is the original CLIP model f 0 The deviation feature distribution for image x, P spurious (x; f t ) is the fine-tuned model f t The deviation feature distribution of image x, and P spurious The calculation formula can be expressed as:

[0108]

[0109] Where f can be the first visual encoder f 0 Or the second visual encoder f t , s is the cosine similarity function, is the jth deviation feature prototype, and the deviation feature prototype μ spurious The calculation formula is:

[0110]

[0111] Where: Y is the category set, Pj is the jth prompt word template, g 0 It is a text encoder for CLIP

[0112] Finally, the total loss function is:

[0113] L total =αL class +βL spurious

[0114] Wherein α and β are weight coefficients, preferably, α=1, β=20.

[0115] Furthermore, in the process of training the initial image category recognition model, the first category loss and the bias loss need to be optimized simultaneously to ensure that the bias-related robustness of the model is maintained while improving the classification ability of the image category recognition model.

[0116] It should be noted that the category loss measures the gap between the category distribution predicted by the model and the true label. In each training iteration, the category loss can be determined based on the true label of the image and the category distribution predicted by the model. The parameters of the model are adjusted according to the category loss obtained in each iteration to minimize the category loss, even if the model converges when the category loss is minimized. Among them, the deviation loss is used to maintain the stability of the model to the deviation feature. Therefore, in each training iteration, the powerful alignment ability of CLIP's text and visual features is used to use text features with unclear category descriptions (i.e., context) as deviation feature prototypes. Then, the image features and the deviation feature prototypes are similarity-measured to determine the probability distribution of the current image on the deviation feature. The deviation loss measures the difference in the probability distribution of the image features extracted by the model before and after fine-tuning. Minimizing the deviation loss can make the model more stable. In summary, when the category loss and deviation loss of the model are minimized as a whole, the model has better classification performance and stability. Therefore, when the trained image category recognition model is applied to the recognition of power system area inspection images, based on the high classification accuracy and generalization of the model, it can better adapt to the recognition of power system area inspection images.

[0117] Furthermore, the embodiment of the present invention also introduces a deviation prototype correction module for correcting the deviation feature prototype. Since the introduced prompt template may contain redundant or unreasonable prompts, the deviation feature prototype calculated by these inaccurate and redundant prompt templates may lack accuracy. Therefore, it is necessary to filter and process these deviation feature prototypes. Preferably, the embodiment of the present invention uses an isolation forest algorithm to remove meaningless deviation feature prototypes, and then uses a k-means algorithm to merge redundant deviation features from similar prompts, specifically including:

[0118] For each prompt template, a text encoder is used to extract the text features of the prompt template, and the feature mean is calculated based on the text features of all prompt templates to obtain the deviation feature prototype. For the deviation feature prototype of the outlier, the isolation forest algorithm is used to filter the deviation feature prototype. The isolation forest algorithm is an unsupervised learning algorithm that is often used to detect outliers. By inputting the deviation feature prototype into the isolation forest algorithm, the isolation forest algorithm outputs an anomaly score, which indicates whether the input deviation feature prototype is an outlier. By setting a filtering threshold in the isolation forest algorithm, only the deviation feature prototypes with anomaly scores lower than the filtering threshold are retained, and the meaningless deviation feature prototypes can be removed. For redundant deviation feature prototypes, the k-means algorithm is used to merge redundant deviation feature prototypes. k-means is an unsupervised learning algorithm that is often used for data clustering. The deviation feature prototype filtered by the isolation forest algorithm is input into k-means, and a cluster label is output by k-means. The cluster label indicates which cluster the deviation feature prototype belongs to. The deviation feature prototypes belonging to the same cluster are merged into the same deviation feature prototype to remove redundant deviation feature prototypes.

[0119] Step S204: until the first category loss and the deviation loss are minimized as a whole, an image category recognition model is generated.

[0120] Implementing the above embodiments has the following effects:

[0121] By obtaining a sample set of power system area inspection images, an initial image category recognition model is constructed based on a multimodal pre-training model, and the initial image category recognition model is trained with the sample set of power system area inspection images. During the training process, the first category loss and deviation loss of the initial image category recognition model are calculated. When the first category loss and deviation loss are minimized as a whole, an image category recognition model is generated. By introducing the calculation of deviation loss in the process of building the image category recognition model, the calculation based on deviation loss can avoid the problem of model overfitting in the case of few samples, and combined with the calculation of the first category loss, the trained image category recognition model has both recognition accuracy and generalization. When the power system area inspection images to be identified are identified based on the trained image category recognition model, the accuracy of power system area inspection image recognition can be improved.

[0122] Embodiment 3

[0123] See also Figure 3 , which is a method for constructing an initial image category recognition model provided by an embodiment of the present invention, based on a multimodal pre-training model structure, constructing an initial image category recognition model, specifically comprising steps S301-S305:

[0124] Step S301: construct an initial multimodal pre-training model; wherein the initial multimodal pre-training model includes a first visual encoder and a text encoder.

[0125] In this embodiment, the initial multimodal pre-trained model includes a first visual encoder f0 and a text encoder g0, and specifically, also includes a second visual encoder ft and a deviation feature classifier, wherein the first visual encoder is generated by pre-training a sample set of power system area inspection images to generate a corresponding model parameter setting, and then the model parameters are fine-tuned to generate a second visual encoder, specifically, as Figure 4 As shown, it is an initial multimodal pre-training model (CLIP, Contrastive Language-Image Pre-Training) including a first visual encoder f0 and a text encoder g0. The initial CLIP model is trained using a sample set of power system area inspection images.

[0126] Step S302: Pre-train the initial multimodal pre-trained model using a sample set of power system area inspection images, so that the first visual encoder learns and extracts the second image features of the sample image, the text encoder extracts the text features of the state category label, generates prototypes of each category based on the text features, and calculates the second category loss of the initial multimodal pre-trained model.

[0127] As a preferred solution of this embodiment, the text encoder extracts text features of the state category label, and generates prototypes of each category according to the text features, specifically including:

[0128] The text encoder is used to extract text features corresponding to the category status labels of all sample images; the mean of all text features is determined based on all text features; and the prototypes of each category are generated based on the mean of the text features.

[0129] As a preferred solution of this embodiment, before the text encoder extracts the text features corresponding to the category status labels of all sample images, it also includes:

[0130] Acquire a plurality of text prompt templates; wherein each text prompt template corresponds to a category status label; generate a plurality of text prompt labels according to the device status category labels corresponding to each type of sample image and the plurality of text prompt templates; input the plurality of text prompt labels into a text editor so that the text editor extracts text features of all text prompt labels.

[0131] In this embodiment, the CLIP text-visual alignment capability is utilized to use the text features of each category name as the category prototype. For any category name y, obtain M text prompt templates (P1,...,PM); combine the M text prompt templates (P1,...,PM) with the category name y to obtain M prompt labels [P1,y],...,[PM,y]. Input the obtained M prompt labels into the text encoder g0, so that the text encoder g0 extracts the text features of the M prompt labels. Calculate the mean of the text features of the extracted M prompt labels to obtain the prototype of the corresponding category name, that is, the prototype of the category name y is μclass_y=1 / M*Σj=1...M g0([Pj,y]). The above calculation can determine the prototype of each category name, which represents the average feature of the category.

[0132] As a preferred solution of this embodiment, the calculating the second category loss of the initial multimodal pre-training model specifically includes:

[0133] According to the first visual encoder learning and extracting the second image features of the sample image, and in combination with the text encoder extracting the text features of the state category label to generate each category prototype, the second similarity between each second image feature and each category prototype is calculated; based on the second similarity between each second image feature and each category prototype, the first probability distribution that the sample image belongs to each category is calculated; based on the first probability distribution, the second category loss of the initial multimodal pre-training model is calculated.

[0134] In this embodiment, during the pre-training process, the first visual encoder f0 extracts the image features (i.e., the above-mentioned second image features) x of the input sample image; the text encoder g0 extracts the text features of the state category label, and generates prototypes of each category based on the text features. During the training process, the second category loss of the CLIP model is calculated. Specifically, based on the second image features output by the first visual encoder f0 and the matching degree between the image and text between the prototypes of each category determined by the text encoder g0, the second category loss between the first visual encoder f0 and the text encoder g0 can be determined. When the second category loss is minimum, that is, when the initial CLIP model reaches the convergence condition, the pre-trained CLIP model can be obtained. Preferably, in order to reduce the amount of calculation for training the CLIP model, the existing pre-trained CLIP model can be directly obtained.

[0135] It should be noted that the expression of the second category loss is:

[0136]

[0137] Where D is the power system area inspection image sample set, and |D| is the number of image samples.

[0138] In this embodiment, the second image feature x of the input image sample is extracted by the first visual encoder f0; the text features of each category name are extracted by the text encoder g0, and the mean is calculated to obtain each category prototype μclass_y, thereby calculating the cosine similarity s(x, μclass_y) between the second image feature x and each category prototype μclass_y, and calculating the probability distribution P(y|x) that the image belongs to each category based on the cosine similarity, and finally calculating the second category loss through the cross entropy loss.

[0139] Step S303: until the second category loss is minimized, a pre-trained multimodal pre-training model is generated.

[0140] Step S304: construct a deviation feature classifier according to the pre-trained multimodal pre-trained model.

[0141] As a preferred solution, constructing a deviation feature classifier based on the pre-trained multimodal pre-training model specifically includes:

[0142] According to the pre-trained multimodal pre-training model, context features that are independent of each category prototype are determined as deviation feature prototypes; and according to the deviation feature prototypes, a deviation feature classifier is constructed.

[0143] In this embodiment, a spurious feature classifier is a mechanism for maintaining the stability of the model to spurious features, and can be used to classify and evaluate spurious features in images. By utilizing the text-visual alignment capability of the CLIP model, contextual text features that are not related to the category are used as prototypes of spurious features, and similarity measurements are performed between image features and prototypes of spurious features to determine the probability distribution of the image on the spurious features. By constraining the consistency of the probability distribution of image features extracted by the model on the spurious features before and after fine-tuning, the stability of the model to the spurious features is maintained, thereby ensuring that the model maintains robustness to spurious associations during the fine-tuning process, and while improving the classification capability of the model, the out-of-domain (OOD) performance of the model is maintained, while preventing the model from overfitting to the spurious features in the training data, thereby improving the generalization capability of the model.

[0144] In the present embodiment, by obtaining a text prompt template, that is, using a plurality of predefined general text prompt templates, the text prompt template is combined with the category name to generate a plurality of prompt labels, the text features of these prompt labels are extracted using the text encoder of CLIP, the mean of all text features is calculated, and the deviation feature prototype is generated. Then the deviation feature prototype is corrected, the abnormal deviation feature prototype is removed by using the isolation forest algorithm, and the similar deviation feature prototype is clustered and merged using the k-means algorithm. Then a classifier is constructed, that is, a network structure is designed first, the input is the image feature, the output is the probability distribution of the image on each deviation feature prototype, the similarity between the image feature and the deviation feature prototype is calculated using the cosine similarity, and the similarity is converted into a probability distribution by the softmax function. Finally, a loss function is designed, and the KL divergence of the probability distribution of the model on the deviation feature before and after fine-tuning is used as the deviation loss, so that the deviation loss is combined with the classification loss as the objective function of the model optimization.

[0145] Step S305: constructing an initial image category recognition model based on the pre-trained multimodal pre-trained model and the deviation feature classifier.

[0146] As a preferred solution, the initial image category recognition model is constructed based on the pre-trained multimodal pre-trained model and the deviation feature classifier, specifically including:

[0147] Freeze the model parameters of the text encoder in the currently pre-trained multimodal pre-trained model; construct a second visual encoder based on the model parameters of the first visual encoder in the currently pre-trained multimodal pre-trained model; and construct an initial image category recognition model based on the first visual encoder, the second visual encoder, the text encoder and the deviation feature classifier.

[0148] In this embodiment, the first visual encoder f0 is the visual encoder part in the initial CLIP model, which is used to extract the second image feature of the sample image in the pre-training CLIP stage; in the fine-tuning stage, its parameters are used as the initial parameters of the second visual encoder; in the iterative training process, it is used to extract the second image feature, which is used to calculate the deviation loss. The second visual encoder ft is constructed based on the first visual encoder f0, with the parameters of f0 as the initial parameters, and is used to extract the third image feature of the sample image in the fine-tuning stage. The third image feature is used to calculate the first category loss to improve the classification ability of the model. At the same time, the third image feature is also used to compare with the second image feature to calculate the deviation loss to maintain the stability of the model to the deviation feature.

[0149] It should be noted that the parameters of the first visual encoder f0 remain unchanged during the fine-tuning stage, while the parameters of the second visual encoder ft are continuously updated during the training process; at the same time, the second image features extracted by the first visual encoder f0 are mainly used to calculate the deviation loss, while the third image features extracted by the second visual encoder ft are used to calculate the category loss and the deviation loss. The first visual encoder f0 and the second visual encoder ft enable the model to maintain robustness to deviation features while improving classification accuracy, so that in the case of few samples, the classification ability of the model can be improved while maintaining the generalization of the model, thereby improving the accuracy and stability of regional inspection image recognition in the power system.

[0150] Furthermore, regarding the first image feature and the second image feature, the first image feature refers to the image feature extracted by the image category recognition model from the inspection image of the power system area to be identified, and the feature used for actual identification is extracted by the finally trained model. The second image feature refers to the image feature learned by the first visual encoder f0 from the sample image when constructing the initial CLIP model, that is, as an intermediate feature in the model training process, used to calculate the loss function and optimize the model parameters.

[0151] In this embodiment, on the basis of obtaining a pre-trained CLIP model, the image category recognition model is made to have better recognition accuracy and generalization by fine-tuning the pre-trained CLIP model. During the fine-tuning process, the text encoder g0 of CLIP is frozen so that the model parameters of the text encoder g0 no longer change; based on the first visual encoder f0, a second visual encoder ft is constructed, and the model parameters of the first visual encoder f0 are used as the initial parameters of the second visual encoder ft. The second visual encoder extracts the image features of the input sample image (that is, the third image features mentioned above). Combined with a deviation feature classifier, an initial image category recognition model including the first visual encoder f0, the second visual encoder ft, the text encoder g0 and the deviation feature classifier can be obtained. The specific structure of the model is as follows Figure 4 As shown, in Figure 4 The right side of is the deviation feature classifier, which is used to adjust and determine the deviation loss.

[0152] Implementing the above embodiments has the following effects:

[0153] By obtaining a sample set of power system area inspection images, an initial image category recognition model is constructed based on a multimodal pre-training model, and the initial image category recognition model is trained with the sample set of power system area inspection images. During the training process, the first category loss and deviation loss of the initial image category recognition model are calculated. When the first category loss and deviation loss are minimized as a whole, an image category recognition model is generated. By introducing the calculation of deviation loss in the process of building the image category recognition model, the calculation based on deviation loss can avoid the problem of model overfitting in the case of few samples, and combined with the calculation of the first category loss, the trained image category recognition model has both recognition accuracy and generalization. When the power system area inspection images to be identified are identified based on the trained image category recognition model, the accuracy of power system area inspection image recognition can be improved.

[0154] Embodiment 4

[0155] See also Figure 5 , which is a power system area inspection image recognition device provided by the present invention, comprising: an acquisition module 01 and a recognition module 02;

[0156] The acquisition module 01 is used to acquire the inspection image of the power system area to be identified;

[0157] The recognition module 02 is used to input the inspection image into the image category recognition model, extract the first image feature, and determine the first similarity between the first image feature and each category prototype according to each category prototype in the image category recognition model, and determine the corresponding category prototype according to the first similarity as the device status category of the inspection image;

[0158] The image category recognition model is obtained by training an initial image category recognition model using a sample set of power system area inspection images.

[0159] As a preferred solution, the image category recognition model is obtained by training the initial image category recognition model through the power system area inspection image sample set, specifically including:

[0160] Acquire a sample set of inspection images of a power system area; wherein the sample set of inspection images includes a plurality of sample images and a device status category label corresponding to each sample image;

[0161] Based on the multimodal pre-training model structure, build an initial image category recognition model;

[0162] Iteratively training the initial image category recognition model with a power system area inspection image sample set, so that during the iterative training process, the first category loss and the deviation loss of the initial image category recognition model are calculated in sequence;

[0163] An image category recognition model is generated until the first category loss and the deviation loss are minimized as a whole.

[0164] As a preferred solution, the initial image category recognition model is constructed based on the multimodal pre-training model structure, specifically including:

[0165] Constructing an initial multimodal pre-training model; wherein the initial multimodal pre-training model includes a first visual encoder and a text encoder;

[0166] The initial multimodal pre-trained model is pre-trained using a sample set of power system area inspection images, so that the first visual encoder learns and extracts a second image feature of the sample image, the text encoder extracts text features of the state category label, generates prototypes of each category according to the text features, and calculates the second category loss of the initial multimodal pre-trained model;

[0167] Until the loss of the second category is minimized, generating a pre-trained multimodal pre-trained model;

[0168] Constructing a deviation feature classifier according to the pre-trained multimodal pre-trained model;

[0169] Based on the pre-trained multimodal pre-trained model and the deviation feature classifier, an initial image category recognition model is constructed.

[0170] As a preferred solution, constructing a deviation feature classifier based on the pre-trained multimodal pre-training model specifically includes:

[0171] Determining, according to the pre-trained multimodal pre-trained model, context features that are irrelevant to each category prototype as deviation feature prototypes;

[0172] According to the deviation feature prototype, a deviation feature classifier is constructed.

[0173] As a preferred solution, the initial image category recognition model is constructed based on the pre-trained multimodal pre-trained model and the deviation feature classifier, specifically including:

[0174] Freeze the model parameters of the text encoder in the current pre-trained multimodal pre-trained model;

[0175] Based on the model parameters of the first visual encoder in the currently pre-trained multimodal pre-trained model, construct a second visual encoder;

[0176] An initial image category recognition model is constructed based on the first visual encoder, the second visual encoder, the text encoder and the deviation feature classifier.

[0177] As a preferred solution, the text encoder extracts text features of the state category label and generates prototypes of each category according to the text features, specifically including:

[0178] Extracting text features corresponding to the category state labels of all sample images through the text encoder;

[0179] Based on all text features, determine the mean of all text features;

[0180] According to the mean of the text features, a prototype of each category is generated.

[0181] As a preferred solution, before the text encoder extracts the text features corresponding to the category status labels of all sample images, it also includes:

[0182] Obtain a number of text prompt templates; wherein each text prompt template corresponds to a category state label;

[0183] Generate a number of text prompt labels according to the device status category labels corresponding to the various sample images and a number of text prompt templates;

[0184] A plurality of text prompt tags are input into a text editor so that the text editor extracts text features of all the text prompt tags.

[0185] As a preferred solution, the calculating the second category loss of the initial multimodal pre-training model specifically includes:

[0186] The second image features of the sample image are learned and extracted by the first visual encoder, and the second similarities between each second image feature and each category prototype are calculated by combining the text features of the state category label extracted by the text encoder to generate each category prototype;

[0187] According to the second similarity between each second image feature and each category prototype, a first probability distribution of the sample image belonging to each category is calculated;

[0188] According to the first probability distribution, a second category loss of the initial multimodal pre-training model is calculated.

[0189] As a preferred solution, the calculation of the first category loss of the initial image category recognition model specifically includes:

[0190] extracting a third image feature of the sample image through a second visual encoder in the initial image category recognition model;

[0191] Extract text features of state category labels through the text editor in the initial image category recognition model;

[0192] Generate prototypes of each category based on the text features of the extracted state category labels;

[0193] The first category loss is calculated based on the third image feature and the prototypes of each category.

[0194] As a preferred solution, the first category loss is calculated according to the third image feature and each category prototype, specifically including:

[0195] Calculating a third similarity between each third image feature and each category prototype;

[0196] According to the third similarity between each third image feature and each category prototype, a second probability distribution of the sample image belonging to each category is calculated;

[0197] According to the second probability distribution, a first category loss of the initial image category recognition model is calculated.

[0198] As a preferred solution, the calculation obtains the deviation loss of the initial image category recognition model, specifically including:

[0199] extracting a third image feature of the sample image through a second visual encoder in the initial image category recognition model;

[0200] The deviation loss is calculated according to the second image feature and the third image feature.

[0201] As a preferred solution, during the iterative training process, when unclassified text features are used as deviation feature prototypes, the deviation loss is adjusted according to the current deviation feature prototype through the deviation feature classifier.

[0202] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0203] Implementing the above embodiments has the following effects:

[0204] The technical solution of the present invention obtains inspection images of the power system area to be identified, and then inputs the inspection images into an image category recognition model obtained by training an initial image category recognition model through a sample set of inspection images of the power system area, thereby extracting a first image feature, and calculating a corresponding first similarity based on each category prototype, so as to determine a final proposed category prototype, which is output as the equipment status category of the inspection image, and finally realizes accurate recognition and analysis of the inspection images of the power system area to be identified, thereby avoiding the inability to effectively identify the inspection images of the power system area.

[0205] Furthermore, by acquiring a sample set of power system area inspection images, an initial image category recognition model is constructed based on a multimodal pre-training model, and the initial image category recognition model is trained with the sample set of power system area inspection images. During the training process, the first category loss and deviation loss of the initial image category recognition model are calculated, and when the first category loss and deviation loss are minimized as a whole, an image category recognition model is generated; by introducing the calculation of deviation loss in the process of constructing the image category recognition model, the calculation based on deviation loss can avoid the problem of model overfitting in the case of few samples, and combined with the calculation of the first category loss, the trained image category recognition model has both recognition accuracy and generalization. When the power system area inspection images to be identified are identified based on the trained image category recognition model, the accuracy of power system area inspection image recognition can be improved.

[0206] Embodiment 5

[0207] Correspondingly, the present invention also provides a terminal device, comprising: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the inspection image recognition method of the power system as described in any one of Embodiments 1 to 3 above is implemented.

[0208] The terminal device of this embodiment includes: a processor, a memory, and a computer program and a computer instruction stored in the memory and executable on the processor. When the processor executes the computer program, each step in the above-mentioned embodiment 1 is implemented, for example: Figure 1 Alternatively, when the processor executes the computer program, the functions of each module / unit in the above device embodiment are implemented, such as the identification module 02.

[0209] Exemplarily, the computer program may be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal device. For example, the recognition module 02 is used to input the inspection image into the image category recognition model, extract the first image feature, and determine the first similarity between the first image feature and each category prototype according to each category prototype in the image category recognition model, and determine the corresponding category prototype according to the first similarity as the device status category of the inspection image.

[0210] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the schematic diagram is merely an example of a terminal device and does not constitute a limitation on the terminal device. The terminal device may include more or fewer components than shown in the diagram, or may combine certain components, or different components. For example, the terminal device may also include an input / output device, a network access device, a bus, etc.

[0211] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device.

[0212] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the terminal device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the mobile terminal, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0213] Wherein, if the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0214] Embodiment 6

[0215] Correspondingly, the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the inspection image recognition method of the power system as described in any one of Embodiments 1 to 3 above.

[0216] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for inspection image recognition of a power system, characterized in that: include: Acquire inspection images of the power system area to be identified; Input the inspection image into an image category recognition model to extract a first image feature, and determine a first similarity between the first image feature and each category prototype according to each category prototype in the image category recognition model, and determine a corresponding category prototype according to the first similarity as the device status category of the inspection image; The image category recognition model is obtained by training an initial image category recognition model through a sample set of inspection images in the power system area.

2. The inspection image recognition method for a power system according to claim 1, characterized in that: The image category recognition model is obtained by training the initial image category recognition model through the inspection image sample set of the power system area, specifically including: Acquire a sample set of inspection images of a power system area; wherein the sample set of inspection images includes a plurality of sample images and a device status category label corresponding to each sample image; Based on the multimodal pre-training model structure, build an initial image category recognition model; Iteratively training the initial image category recognition model with a power system area inspection image sample set, so that during the iterative training process, the first category loss and the deviation loss of the initial image category recognition model are calculated in sequence; An image category recognition model is generated until the first category loss and the deviation loss are minimized as a whole.

3. The inspection image recognition method for a power system according to claim 1, characterized in that: The method for constructing the initial image category recognition model specifically includes: Constructing an initial multimodal pre-training model; wherein the initial multimodal pre-training model includes a first visual encoder and a text encoder; The initial multimodal pre-trained model is pre-trained using a sample set of power system area inspection images, so that the first visual encoder learns and extracts a second image feature of the sample image, the text encoder extracts text features of the state category label, generates prototypes of each category according to the text features, and calculates the second category loss of the initial multimodal pre-trained model; Until the loss of the second category is minimized, generating a pre-trained multimodal pre-trained model; Constructing a deviation feature classifier according to the pre-trained multimodal pre-trained model; Based on the pre-trained multimodal pre-trained model and the deviation feature classifier, an initial image category recognition model is constructed.

4. The inspection image recognition method for a power system according to claim 3, characterized in that: The step of constructing a deviation feature classifier based on the pre-trained multimodal pre-training model specifically includes: Determining, according to the pre-trained multimodal pre-trained model, context features that are irrelevant to each category prototype as deviation feature prototypes; According to the deviation feature prototype, a deviation feature classifier is constructed.

5. The inspection image recognition method for a power system according to claim 3, characterized in that: The constructing of an initial image category recognition model based on the pre-trained multimodal pre-trained model and the deviation feature classifier specifically includes: Freeze the model parameters of the text encoder in the current pre-trained multimodal pre-trained model; Based on the model parameters of the first visual encoder in the currently pre-trained multimodal pre-trained model, construct a second visual encoder; An initial image category recognition model is constructed based on the first visual encoder, the second visual encoder, the text encoder and the deviation feature classifier.

6. A method for inspection image recognition of a power system according to any one of claims 3 to 5, characterized in that: The text encoder extracts text features of the state category label and generates prototypes of each category according to the text features, specifically including: Extracting text features corresponding to the category state labels of all sample images through the text encoder; Based on all text features, determine the mean of all text features; According to the mean of the text features, a prototype of each category is generated.

7. The inspection image recognition method for a power system according to claim 6, characterized in that: Before the text encoder extracts the text features corresponding to the category status labels of all sample images, it also includes: Obtain a number of text prompt templates; wherein each text prompt template corresponds to a category state label; Generate a number of text prompt labels according to the device status category labels corresponding to the various sample images and a number of text prompt templates; A plurality of text prompt tags are input into a text editor so that the text editor extracts text features of all the text prompt tags.

8. A method for inspection image recognition of a power system according to any one of claims 3 to 5, characterized in that: The calculating the second category loss of the initial multimodal pre-training model specifically includes: The second image features of the sample image are learned and extracted by the first visual encoder, and the second similarities between each second image feature and each category prototype are calculated by combining the text features of the state category label extracted by the text encoder to generate each category prototype; According to the second similarity between each second image feature and each category prototype, a first probability distribution of the sample image belonging to each category is calculated; According to the first probability distribution, a second category loss of the initial multimodal pre-training model is calculated.

9. A method for inspection image recognition of a power system according to any one of claims 2 to 5, characterized in that: The calculating of the first category loss of the initial image category recognition model specifically includes: extracting a third image feature of the sample image through a second visual encoder in the initial image category recognition model; Extract text features of state category labels through the text editor in the initial image category recognition model; Generate prototypes of each category based on the text features of the extracted state category labels; The first category loss is calculated based on the third image feature and the prototypes of each category.

10. The inspection image recognition method for a power system according to claim 9, characterized in that: The first category loss is calculated according to the third image feature and each category prototype, specifically including: Calculating a third similarity between each third image feature and each category prototype; According to the third similarity between each third image feature and each category prototype, a second probability distribution of the sample image belonging to each category is calculated; According to the second probability distribution, a first category loss of the initial image category recognition model is calculated.

11. A method for inspection image recognition of a power system according to any one of claims 2 to 5, characterized in that: The calculation obtains the deviation loss of the initial image category recognition model, specifically including: extracting a third image feature of the sample image through a second visual encoder in the initial image category recognition model; The deviation loss is calculated according to the second image feature and the third image feature.

12. The inspection image recognition method for a power system according to claim 11, characterized in that: During the iterative training process, when unclassified text features are used as deviation feature prototypes, the deviation loss is adjusted according to the current deviation feature prototype through the deviation feature classifier.

13. An image recognition device for regional inspection of a power system, characterized in that: include: Get Modules and Identify Modules The acquisition module is used to acquire the inspection image of the power system area to be identified; The recognition module is used to input the inspection image into the image category recognition model, extract the first image feature, and determine the first similarity between the first image feature and each category prototype according to each category prototype in the image category recognition model, and determine the corresponding category prototype according to the first similarity as the device status category of the inspection image; The image category recognition model is obtained by training an initial image category recognition model using a power system area inspection image sample set.

14. A terminal device, characterized in that: The invention comprises a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the inspection image recognition method for the power system as claimed in any one of claims 1 to 12 is implemented.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the inspection image recognition method for the power system according to any one of claims 1 to 12.

Images