Multi-label electrical equipment oil leakage identification method based on scene perception

Abstract

The invention belongs to the technical field of computer image processing, particularly discloses a scene perception-based multi-label oil leakage identification method for power equipment, and provides a scene perception-based multi-label oil leakage identification model based on the characteristic that the co-occurrence relationship of different types of labels in a picture is closely related to a picture scene. According to the model, through scene category detection, label co-occurrence modeling and scene classification loss function calculation, fine-grained modeling of a scene level label co-occurrence relation is realized, then a label graph is constructed, and message propagation and feature updating are performed on the label graph, so that more feature interaction exists between labels with high co-occurrence probability in different scenes, and the scene classification loss function is calculated. Therefore, the visual characterization of each other is improved, and the identification of leaked oil labels is promoted. According to the multi-label electric power equipment oil leakage identification method based on scene perception, the oil contamination area on the electric power equipment can be identified, different oil leakage types can be accurately judged, and therefore multi-label oil leakage identification is achieved.

Description

Technical Field

[0001] The invention belongs to the technical field of computer image processing and relates to a multi-label electric equipment oil leakage identification method based on scene perception. Background Art

[0002] As an important device used in power generation, transmission, transformation, and distribution, power equipment is also an indispensable infrastructure for energy production and conversion. Common power equipment includes transformers, gas relays, switchgear, submersible pumps, etc. Whether these power equipment can operate stably and reliably is directly related to the safety of power production and transmission. However, power equipment is prone to oil leakage after high-intensity and long-term operation, especially under extreme weather conditions such as severe cold, high temperature, and high humidity. Since power equipment is often exposed to the outdoors, it is more likely to induce oil leakage. Once a large-scale oil leakage occurs in power equipment, it will not only reduce the insulation performance of the equipment, causing safety accidents such as short circuits and fires, but also cause land and vegetation pollution and damage the local ecological environment. Therefore, identifying oil leakage in power equipment based on pictures taken by cameras around the facilities is of great value in preventing power safety accidents and environmental pollution.

[0003] Once oil leakage occurs in power equipment, it often spreads across multiple parts, resulting in oil stains in multiple parts. Therefore, the pictures of power equipment taken by the camera may contain multiple types of oil leakage. Figure 2 and Figure 3 The pictures of electrical equipment with two or more types of oil leakage are shown respectively. Figure 2 Including two types of oil leakage: ground oil pollution and pipeline oil pollution. Figure 3 There are three types of oil leakage: ground oil pollution, transformer oil pollution, and valve oil pollution. For the pictures of power equipment taken by the camera, it is necessary to use machine vision technology to identify the different types of oil leakage in the pictures, so as to locate the oil leakage part of the power equipment, so as to prompt maintenance personnel to intervene and deal with it accurately.

[0004] Existing methods for identifying oil leakage in power equipment based on machine vision can often only detect whether there is oil leakage in a given image, that is, a binary classification task, and it is difficult to accurately identify the multiple types of oil leakage contained in the image. If each type of oil leakage in the power equipment image is regarded as a label, then the multi-label power equipment oil leakage identification problem based on the image can be regarded as an image multi-label classification problem.

[0005] In real life, the label co-occurrence relationship of an image is closely related to the scene information. For example, in the pictures of the oil storage cabinets of the substation, "pipeline" often appears with "valve"; and in the pictures of the main transformer, "transformer" often appears with "ground", which shows that the power equipment scene contained in the picture directly affects the co-occurrence probability of different oil leakage labels. However, when using recurrent neural networks, graph neural networks and other methods to solve the problem of multi-label classification of images, the co-occurrence relationship of different labels is rarely considered, and the interpretability is poor; some methods based on label co-occurrence probability matrices have good interpretability, but they often only consider the label co-occurrence probability at the data set level and instance level, lack of fine-grained mining of label co-occurrence relationships, and are difficult to be well applied to multi-label power equipment oil leakage identification. Summary of the invention

[0006] The purpose of the present invention is to propose a multi-label oil leakage identification method for power equipment based on scene perception, which realizes multi-label oil leakage identification through a multi-label oil leakage identification model based on scene perception for power equipment pictures containing multiple oil leakage scenes.

[0007] In order to achieve the above object, the present invention adopts the following technical scheme:

[0008] Step 1. Build a scene-aware multi-label oil leakage recognition model;

[0009] The multi-label oil leakage recognition model includes a label feature extraction module, a visual feature extraction module, a cross-modal feature alignment module, a scene-aware label co-occurrence module, a semantic interaction module, and a probability distribution calculation module.

[0010] Step 2. Build a dataset of oil leakage images of power equipment with annotations;

[0011] For the oil leakage image dataset of power equipment, the label feature extraction module is used to extract the text semantic features of each oil leakage label, and the visual feature extraction module is used to extract the spatial visual features of each oil leakage image of power equipment;

[0012] Step 3. Establish a cross-modal alignment relationship between images and labels through a cross-modal feature alignment module;

[0013] Step 4. The scene-aware tag co-occurrence module models the scene-aware tag co-occurrence relationship and constructs the entropy loss function through scene category detection, tag co-occurrence modeling, and scene classification loss function calculation;

[0014] Step 5. The semantic interaction module constructs a tag graph and uses the tag co-occurrence probability matrix to guide the semantic interaction between tags and improve the visual feature representation of tags with high co-occurrence probability, i.e., visual representation;

[0015] Step 6. The probability distribution calculation module inputs the improved label visual feature representation into the feedforward neural network to calculate the probability distribution of different labels on the image and construct the multi-label classification loss;

[0016] Step 7. Based on the entropy loss function and the multi-label classification loss given in Step 4 and Step 6, a joint loss function is constructed, and the scene-aware multi-label oil leakage recognition model is trained based on the joint loss function;

[0017] Step 8. Based on the trained scene-aware multi-label oil leakage recognition model, the input image is subjected to multi-label oil leakage recognition of power equipment to obtain a multi-label oil leakage recognition result of power equipment.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] As described above, the present invention relates to a multi-label oil leakage identification method for power equipment based on scene perception. The method proposes a multi-label oil leakage identification model based on scene perception based on the characteristic that the co-occurrence relationship of different types of labels in the picture is closely related to the picture scene. The model realizes fine-grained modeling of label co-occurrence relationships at the scene level through scene category detection, label co-occurrence modeling, and scene classification loss function calculation; by constructing a label graph and performing message propagation and feature update on the label graph, there are more feature interactions between labels with high co-occurrence probabilities in different scenes, thereby improving each other's visual representation and promoting subsequent oil leakage label recognition. The multi-label oil leakage identification method for power equipment based on scene perception of the present invention can not only identify oily areas on power equipment, but also accurately judge different types of oil leakage. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0021] Figure 1 It is a flow chart of a method for identifying oil leakage of multi-label electric equipment based on scene perception in an embodiment of the present invention;

[0022] Figure 2 A picture of an electric power device including two types of oil leakage in an embodiment of the present invention;

[0023] Figure 3 The picture of the power equipment including three types of oil leakage in the embodiment of the present invention;

[0024] Figure 4 Schematic diagram of the structure of the scene-aware multi-label oil leakage recognition model in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0026] Example

[0027] In this embodiment, a scene-aware multi-label oil leakage identification method for power equipment is proposed. For power equipment pictures containing multiple oil leakage scenes, based on the characteristics that label co-occurrence relationships are closely related to picture scenes, a scene-aware multi-label oil leakage identification model is designed, and multi-label oil leakage identification of power equipment is performed based on the model.

[0028] like Figure 1 As shown, the multi-label power equipment oil leakage identification method based on scene perception specifically includes the following steps:

[0029] Step 1. Build a scene-aware multi-label oil leakage recognition model.

[0030] Among them, the multi-label oil leakage identification model includes a label feature extraction module, a visual feature extraction module, a cross-modal feature alignment module, a scene-aware label co-occurrence module, a semantic interaction module and a probability distribution calculation module.

[0031] Step 2. Extract the label semantic features and spatial visual features of the image.

[0032] First, we construct a dataset of oil leakage images of power equipment with annotations. The process is as follows:

[0033] M pictures of oil leakage from power equipment are collected from power plants, substations and other places, and each picture is manually labeled. The label category set is in Generate annotated data set of oil leakage images of power equipment Among them, c1 to c N They represent the 1st to Nth oil leakage labels respectively, N represents the number of label categories, and M is a natural number.

[0034] In this embodiment, a total of 8 types of oil leakage are marked, namely ground oil pollution, transformer oil pollution, oil storage cabinet oil pollution, pipeline oil pollution, valve oil pollution, switch cabinet oil pollution, submersible pump oil pollution, and gas relay oil pollution, that is, N=8.

[0035] Then, for the data set of oil leakage images of power equipment, the label feature extraction module is used to extract the text semantic features of each oil leakage label, and the visual feature extraction module is used to extract the spatial visual features of each oil leakage image of power equipment.

[0036] The label feature extraction module in this embodiment is implemented, for example, by using a Bert pre-trained language model, and the visual feature extraction module in this embodiment is implemented, for example, by using a ResNet pre-trained visual backbone network.

[0037] A dataset of oil leakage images of power equipment containing multiple labels For any picture I of oil leakage from power equipment, we first use the Bert pre-trained language model to extract the text semantic features of its oil leakage label.

[0038] If image I contains label c i , then label c i The embedding representation of is: i =Φ T (c i ).

[0039] Among them, i∈{1,2,…,N}, c i Represents a collection of label categories The i-th oil leakage label in t i is the label embedding representation, that is, the extracted label c i The semantic features of the text, d is the dimension of the label embedding representation; Φ T (·) is the Bert pre-trained language model, which is used to convert the input text label c i Mapped to embedding representation t i .

[0040] Secondly, the ResNet pre-trained visual backbone network is used to extract the spatial visual features V of the oil leakage image I of the power equipment:

[0041] V=Φ V (I); Φ V (·) represents the ResNet pre-trained visual backbone network, h and w represent the length and width of the visual feature in the spatial dimension respectively, and c represents the dimension of the visual feature. The image spatial visual feature extracted by the ResNet pre-trained visual backbone network is actually a 3D matrix, and c is the dimension of each element of the matrix.

[0042] The spatial visual feature V is expanded to be expressed as

[0043] in, v j The visual features representing the j-th spatial region on the power equipment oil leakage picture I are visual feature representations, j∈{1,2,…,m}, m represents the number of spatial regions of the power equipment oil leakage picture I, and m=h×w.

[0044] Step 3. Establish a cross-modal alignment relationship between images and labels through the cross-modal feature alignment module.

[0045] In this embodiment, the cross-modal feature alignment module adopts a low-rank bilinear pooling model.

[0046] The cross-modal alignment relationship between equipment oil leakage pictures and text labels is established through a low-rank bilinear pooling model. The low-rank bilinear pooling model has good feature fusion and alignment performance, and can significantly reduce the amount of calculation by reducing the dimension of the joint feature space; it represents the visual feature v j and label embedding representation t i Mapped to a unified joint feature space:

[0047]

[0048] Among them, x i,j Represents the visual feature representation v j and label embedding representation t i Feature representation in a unified joint feature space. P, U, W represent learnable parameter matrices, b represents the learnable parameter vector, tanh(·) represents the hyperbolic tangent function, and ⊙ represents the element-by-element multiplication operation.

[0049] Among them, d1 and d2 represent the dimension of the joint feature space and the dimension of the output space respectively.

[0050] Then, label c i The normalized correlation with the jth region on image I is calculated as follows:

[0051]

[0052] Among them, x i,j′ is the feature representation of the visual features of the image and the textual semantic features of the label in a unified joint feature space, j′∈{1,2,…,m}; α i,j represents the normalized cross-modal semantic relevance weight, Φ a (·) is a learnable feedforward fully connected neural network, which is used to map the input vector into a probability value.

[0053] Then, based on the normalized cross-modal semantic relevance weights, the visual features v of all regions on the image I are j Perform weighted summation to get label c i Visual representation f on image I i :

[0054] Step 4. The scene-aware tag co-occurrence module models the scene-aware tag co-occurrence relationship and constructs the entropy loss function through scene category detection, tag co-occurrence modeling and scene classification loss function calculation.

[0055] Step 4.1. Perform scene category detection. Assume that the oil leakage image dataset of power equipment It contains K scene categories in total. Each scene category can actually be regarded as a type of power equipment picture. For example, the same type of power equipment such as the main transformer is photographed from different angles, which corresponds to only one scene category, namely the main transformer scene.

[0056] The global context features of the image, i.e., picture I, are used to predict the probability distribution of the oil leakage picture I of the power equipment in K scene categories; the global context features of the image It is obtained by a global average pooling operation, and the calculation process is as follows:

[0057]

[0058] The probability π that image I belongs to the kth scene category k The calculation process is as follows:

[0059]

[0060] Among them, w k 、w k′ is the learnable parameter vector, Then we can get the probability distribution π∈{π1,π2,…,π k ,…,π K}.

[0061] Select the scene category with the highest probability in π as the scene category of image I, that is:

[0062] s = argmax k∈{1,2,…,K} π k ;

[0063] Among them, s is the scene category to which image I belongs, that is, the prediction result obtained by scene category detection.

[0064] Step 4.2. Perform label co-occurrence modeling. Maintain a label co-occurrence frequency matrix for each of the K scene categories, and continuously update these label co-occurrence frequency matrices during the training process based on the prediction results obtained from the scene category detection.

[0065] The label co-occurrence frequency matrix of scene category s is updated as follows:

[0066] Z s =Z s +yy T .

[0067] Among them, Z s Represents the label co-occurrence frequency matrix of scene category s, Z sThe initial value of all elements in is 0; y represents the binary vector representation of the label of image I, that is, y = [y1, y2, ..., y N ], y1 to y N are the 1st to Nth components of y respectively. If image I has the i-th label, then y i =1, otherwise y i =0.

[0068] As the training process continues and the label co-occurrence frequency matrix is ​​continuously updated, Z s The diagonal elements in Counted the label c i The number of times it appears in scene category s, not the diagonal elements Then the label c is counted i and label c r The number of times label c co-occurs in scene category s; therefore, in scene category s, when label c r When it appears, label c i The probability of occurrence The calculation process is as follows:

[0069] Among them, r∈[1,N]; through the above method to achieve label co-occurrence modeling, with continuous learning, the label co-occurrence frequency matrix {P 1 ,P 2 ,…,P K} will eventually converge to a stable distribution where P 1 To P K Represent the label co-occurrence frequency matrices of the 1st to Kth scene categories respectively.

[0070] Step 4.3. Calculate the scene classification loss function.

[0071] For each training batch of multi-label power equipment oil leakage picture sample set, the scene probability distribution predicted by scene category detection on the g-th power equipment oil leakage picture is expressed as: g ∈{π g1 ,π g2 ,…,π gK}.

[0072] Among them, g∈{1,2,…,G}, G represents the number of image samples in the current batch; π g1 to π gK They represent the probability that the image g belongs to the 1st to the Kth scene categories respectively. Then the entropy loss function L1 of the training batch samples is expressed as:

[0073]

[0074] Among them, when the loss function L1 is smaller, the scene probability distribution of each image sample is sharper, which means that each image in the batch of samples is clearly classified into a certain scene category.

[0075] During the training process, as the loss L1 continues to decrease, the scene category to which each picture belongs can be identified more accurately.

[0076] Step 5. Improve the visual representation between high co-occurrence probability tags through semantic interactions.

[0077] This embodiment constructs a tag graph through a semantic interaction module, and uses a tag co-occurrence probability matrix to guide semantic interaction between tags, thereby improving the visual feature representation, that is, visual representation, of tags with high co-occurrence probability.

[0078] Based on the label co-occurrence frequency matrix {P 1 ,P 2 ,…,P K} and the image scene probability distribution {π1,π2,…,π K}, calculate the total label co-occurrence probability matrix P of the power equipment oil leakage image I by weighted average I :

[0079] On this basis, construct the label map Node Collection With the dataset Collection of tags Correspondingly, each node v i The initial features are composed of label c i Visual representation f on image I i Initialize, the edge set ε represents the co-occurrence relationship between neighbor nodes, and its weight is the co-occurrence probability, which is calculated by the total label co-occurrence probability matrix P I Initialize.

[0080] Then, the label graph propagation learning is performed to allow each node to learn the representation of the context; specifically, the node v i The embedding representation at round t is express, For label c i The visual representation of for The initial value of is the initial representation, It aggregates context information from neighboring nodes at time t The calculation process is as follows:

[0081] Where N represents the number of labels, that is, the number of graph nodes in the label graph; Represents the total label co-occurrence probability matrix P I The elements in represents the embedded representation of neighbor node r at round t-1.

[0082] The method of the present invention is based on a gated cyclic update mechanism, which is a widely used neural network. The gated neural network includes two gating units, an update gate and a reset gate, which are used to control a specific proportion of past information to be forgotten.

[0083] The feature of the current node is updated using the context information aggregated from neighboring nodes. The calculation process is as follows:

[0084]

[0085] in, is the intermediate hidden vector of the gated recurrent update mechanism, σ(·) represents the Sigmoid function; W z and U z is the parameter matrix of the update gate, where W r and U r is the parameter matrix of the reset gate, where And W h and U h is the parameter matrix used to calculate the hidden state, where Through this message propagation and feature updating mechanism, there will be more feature interactions between tags with high co-occurrence probability in the current scenario, thereby improving each other's visual representation and facilitating subsequent oil spill tag identification.

[0086] Step 6. The probability distribution calculation module inputs the improved label visual feature representation into the feedforward neural network to calculate the probability distribution of different labels on the oil leakage picture of the power equipment and construct a multi-label classification loss.

[0087] After t rounds of feature updates, label c i Visual representation of Its initial characterization Concatenate and input into a feedforward neural network to obtain and The joint feature representation o i :

[0088] Among them, [·;·] represents the concatenation operation, Φ C (·) represents a learnable feedforward neural network.

[0089] On this basis, a linear binary classifier is further used to calculate the label c iThe probability p of appearing on the input image, that is, the oil leakage image I of the power equipment i : Among them, w i represents the learnable parameter vector, σ(·) represents the Sigmoid function, which is used to map real values ​​to probability values ​​in the [0,1] interval.

[0090] The following focal loss is used to calculate the multi-label classification loss L2:

[0091]

[0092] Step 7. Optimize the joint loss function for model training.

[0093] Based on the entropy loss function and the multi-label classification loss given in step 4 and step 6, a joint loss function is constructed, and the scene-aware multi-label oil leakage recognition model is trained based on the joint loss function.

[0094] Based on steps 2 to 6, the overall loss function L of the model is defined as: L=L1+λL2; where λ represents a hyperparameter used to control the weights of the two loss functions L1 and L2.

[0095] By solving the minimum value of the above joint loss function L, the optimal parameter combination of the model can be obtained, thereby obtaining a trained scene-aware multi-label oil leakage recognition model f. This model can actually be regarded as a mapping function f:x→y from the image space to the label category space. Where x represents the input image, and y is the binary vector representation of the label of image x, that is, y=[y1,y2,…,y N ], if image x has the i-th label, then y i =1, otherwise y i =0.

[0096] In the training phase, fully supervised multi-label image classification aims to In the above, we learn the mapping function f:I→Y from the image space to the label category space. Here, the mapping function f(·) usually refers to a parameterized neural network model.

[0097] In the test phase, given any image I′ to be recognized, the model will output y′=f(I′) as the predicted label of the image.

[0098] Step 8. Perform multi-label oil leakage recognition of power equipment based on the trained model: Perform multi-label oil leakage recognition of power equipment on the input image based on the trained scene-aware multi-label oil leakage recognition model to obtain the multi-label oil leakage recognition result of power equipment. That is, input any picture of power equipment containing oil leakage incidents, and use the model to identify the distribution of the picture on N labels.

[0099] Based on the scene-aware multi-label oil leakage recognition model trained in step 7, given any picture I′ of power equipment containing an oil leakage event, the picture I′ is input into the model f to obtain the binary vector representation of the oil leakage label [y′1,y′2,…,y′ N ]; if y′ i =1, it means that there is a label c in the power equipment image I′ i Corresponding oil leakage type; if y′ i = 0, it means that there is no label c in the power equipment image I′ i In the corresponding oil leakage type.

[0100] It should be noted that the multi-label power equipment oil leakage identification method based on scene perception in this embodiment is also applicable to single-label power equipment oil leakage identification, because single-label oil leakage can actually be used as a special case of multi-label oil leakage. If there is only one type of oil leakage in the power equipment image, the multi-label power equipment oil leakage identification problem degenerates to the single-label power equipment oil leakage identification problem, which will not be elaborated here.

[0101] Of course, the above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above embodiments. It should be noted that all equivalent substitutions and obvious deformation forms made by any technician familiar with the field under the guidance of this specification fall within the essential scope of this specification and should be protected by the present invention.

Claims

1. A multi-label power equipment oil leakage identification method based on scene perception, characterized in that: The steps include: Step 1. Build a scene-aware multi-label oil leakage recognition model; The multi-label oil leakage recognition model includes a label feature extraction module, a visual feature extraction module, a cross-modal feature alignment module, a scene-aware label co-occurrence module, a semantic interaction module, and a probability distribution calculation module. Step 2. Build a dataset of annotated oil leakage images of power equipment; For the oil leakage image dataset of power equipment, the label feature extraction module is used to extract the text semantic features of each oil leakage label, and the visual feature extraction module is used to extract the spatial visual features of each oil leakage image of power equipment; Step 3. Establish a cross-modal alignment relationship between images and labels through a cross-modal feature alignment module; Step 4. The scene-aware tag co-occurrence module models the scene-aware tag co-occurrence relationship and constructs the entropy loss function through scene category detection, tag co-occurrence modeling, and scene classification loss function calculation; Step 5. The semantic interaction module constructs a tag graph and uses the tag co-occurrence probability matrix to guide the semantic interaction between tags and improve the visual feature representation of tags with high co-occurrence probability, i.e., visual representation; Step 6. The probability distribution calculation module inputs the improved label visual feature representation into the feedforward neural network to calculate the probability distribution of different labels on the image and construct the multi-label classification loss; Step 7. Based on the entropy loss function and the multi-label classification loss given in Step 4 and Step 6, a joint loss function is constructed, and the scene-aware multi-label oil leakage recognition model is trained based on the joint loss function; Step 8. Based on the trained scene-aware multi-label oil leakage recognition model, the input image is subjected to multi-label oil leakage recognition of power equipment to obtain a multi-label oil leakage recognition result of power equipment.

2. The method for identifying oil leakage of power equipment based on multi-tag scene perception according to claim 1 is characterized in that: In step 2, the process of constructing a dataset of oil leakage images of power equipment with annotations is specifically as follows: M pictures of oil leakage from power equipment are collected from power plants and substations, and each picture is manually labeled. The label category set is Generate annotated data set of oil leakage images of power equipment Among them, c1 to c N They represent the 1st to Nth oil leakage labels respectively, N represents the number of label categories, and M is a natural number.

3. The method for identifying oil leakage of electric equipment based on multi-label scene perception according to claim 2 is characterized in that: In step 2, the label feature extraction module uses the Bert pre-trained language model, and the visual feature extraction module uses the ResNet pre-trained visual backbone network. The process of extracting text semantic features and spatial visual features is specifically as follows: A dataset of oil leakage images of power equipment containing multiple labels For any picture I of oil leakage from power equipment, use the Bert pre-trained language model to extract the text semantic features of its oil leakage label; If image I contains label c i , then label c i The embedding representation of is: i =Φ T (c i ); Among them, i∈{1,2,…,N}, c i Represents a collection of label categories The i-th oil leakage label in t i is the label embedding representation, that is, the extracted label c i The semantic features of the text, d is the dimension of the label embedding representation; Φ T (·) is the Bert pre-trained language model, which is used to convert the input text label c i Mapped to embedding representation t i ; Use the ResNet pre-trained visual backbone network to extract the spatial visual features V of the power equipment oil leakage image I: V=Φ V (I); where Φ V (·) represents the ResNet pre-trained visual backbone network, h and w represent the length and width of the visual feature in the spatial dimension, respectively, and c represents the dimension of the visual feature; The spatial visual feature V is expanded to be expressed as in, v j The visual features representing the j-th spatial region on the power equipment oil leakage picture I are visual feature representations, j∈{1,2,…,m}, m represents the number of spatial regions of the power equipment oil leakage picture I, and m=h×w.

4. The method for identifying oil leakage of power equipment based on multi-tag scene perception according to claim 1 is characterized in that: In step 2, the categories of the marked oil leakage labels include ground oil stains, transformer oil stains, oil storage cabinet oil stains, pipeline oil stains, valve oil stains, switch cabinet oil stains, submersible pump oil stains, and gas relay oil stains.

5. The method for identifying oil leakage of electric equipment based on multi-tags based on scene perception according to claim 3 is characterized in that: The step 3 is specifically as follows: The cross-modal feature alignment module uses a low-rank bilinear pooling model to establish a cross-modal alignment relationship between equipment oil leakage images and text labels, and represents the visual features v j and label embedding representation t i Mapped to a unified joint feature space: x i,j =P T (tanh((U T v j )⊙(W T t i )))+b; Among them, x i,j Represents the visual feature representation v j and label embedding representation t i Feature representation in a unified joint feature space; P, U, W represent the learnable parameter matrices, b represents the learnable parameter vector, tanh(·) represents the hyperbolic tangent function, ⊙ represents the element-by-element multiplication operation; Where d1 and d2 represent the dimension of the joint feature space and the dimension of the output space respectively; Tag c i The normalized correlation with the jth region on the oil leakage image I of the power equipment is calculated as follows: Among them, x i,j′ is the feature representation of the visual features of the oil leakage picture of the power equipment and the textual semantic features of the label in a unified joint feature space, j′∈{1,2,…,m}; α i,j represents the normalized cross-modal semantic relevance weight, Φ a (·) is a learnable feedforward fully connected neural network, which is used to map the input vector into a probability value; Based on the normalized cross-modal semantic relevance weights, the visual features v of all regions on the image I are j Perform weighted summation to get label c i Visual representation f on image I i :

6. The method for identifying oil leakage of electric equipment based on multi-tags based on scene perception according to claim 5 is characterized in that: The step 4 is specifically as follows: Step 4.

1. Perform scene category detection; The global context features of the image are used to predict the probability distribution of the oil leakage picture I of the power equipment in K scene categories. It is obtained by a global average pooling operation, and the calculation process is as follows: The probability π that image I belongs to the kth scene category k The calculation process is as follows: Among them, w k 、w k′ is the learnable parameter vector, Then we can get the probability distribution π∈{π1,π2,…,π k ,…,π K }; Select the scene category with the highest probability in π as the scene category of image I, that is: s=argmax k∈{1,2,…,K} π k ; Where s is the scene category to which image I belongs, that is, the prediction result obtained by scene category detection; Step 4.

2. Perform tag co-occurrence modeling; Maintain a label co-occurrence frequency matrix for each of the K scene categories, and update the label co-occurrence frequency matrix during training based on the prediction results obtained from the scene category detection; the label co-occurrence frequency matrix of scene category s is updated as follows: Z s =Z s +yy T ; Among them, Z s Represents the label co-occurrence frequency matrix of scene category s, Z s The initial value of all elements in is 0; y represents the binary vector representation of the label of image I, y = [y1, y2, ..., y N ], where y1 to y N are the 1st to Nth components of y respectively. If image I has the i-th label, then y i =1, otherwise y i =0; In scene category s, when label c r When it appears, label c i The probability of occurrence The calculation process is as follows: Among them, r∈[1,N], is the diagonal element of the label co-occurrence frequency matrix of scene category s, used to count label c i The number of times it appears in scene category s; is the non-diagonal element of the label co-occurrence frequency matrix of scene category s, used to count label c i and label c r The number of co-occurrences in scene category s; As the model continues to learn, the label co-occurrence frequency matrix {P 1 ,P 2 ,…,P K } eventually converges to a stable distribution where P 1 To P K Respectively represent the label co-occurrence frequency matrix of the 1st to the Kth scene categories; Step 4.

3. Calculate the scene classification loss function; For each training batch of multi-label power equipment oil leakage picture sample set, the predicted scene probability distribution π on the g-th power equipment oil leakage picture g Expressed as: π g ∈{π g1 ,π g2 ,…,π gK }; Among them, g∈{1,2,…,G}, G represents the number of image samples in the current batch; π g1 to π gK They represent the probability that the image g belongs to the 1st to the Kth scene categories respectively; the entropy loss function L1 of the training batch samples is expressed as:

7. The method for identifying oil leakage of electric equipment based on multi-tags based on scene perception according to claim 6 is characterized in that: The step 5 is specifically as follows: Based on the label co-occurrence frequency matrix {P 1 ,P 2 ,…,P K } and the image scene probability distribution {π1,π2,…,π K }, calculate the total label co-occurrence probability matrix P of the power equipment oil leakage image I by weighted average I : Building a label graph Node Collection With the dataset Collection of tags Correspondingly, each node v i The initial features are composed of label c i Visual representation f on image I i Initialize; the edge set ε represents the co-occurrence relationship between neighbor nodes, and its weight is the co-occurrence probability, which is obtained by the total label co-occurrence probability matrix P I Initialize; Perform label graph propagation learning to enable each node to learn the representation of the context; i The embedding representation at round t is express, For label c i The visual representation of for The initial value of is the initial representation, Context information aggregated from neighboring nodes at time t The calculation process is: Where N is the number of labels, that is, the number of graph nodes in the label graph; Represents the total label co-occurrence probability matrix P I The elements in represents the embedding representation of neighbor node r in round t-1; based on the gated loop update mechanism, the context information aggregated from neighbor nodes is used to update the features of the current node. The calculation process is as follows: in, is the intermediate hidden vector of the gated recurrent update mechanism, σ(·) represents the Sigmoid function; W z and U z is the parameter matrix of the update gate, where W r and U r is the parameter matrix of the reset gate, where W h and U h is the parameter matrix used to calculate the hidden state, where Through message propagation and feature update mechanisms, there are more feature interactions between tags with high co-occurrence probability in the current scenario, thereby improving each other's visual representation and obtaining an improved visual feature representation of the tags.

8. The method for identifying oil leakage of electric equipment based on multi-tags based on scene perception according to claim 7 is characterized in that: The step 6 is specifically as follows: After t rounds of feature updates, label c i Visual representation of Its initial characterization Concatenate and input into a feedforward neural network to obtain and The joint feature representation o i : Among them, [·;·] represents the concatenation operation, Φ C (·) represents a learnable feedforward neural network; Compute the label c using a linear binary classifier i The probability p of appearing on the input image, that is, the oil leakage image I of the power equipment i : Among them, w i represents the learnable parameter vector, σ(·) represents the Sigmoid function, which is used to map real values ​​to probability values ​​in the interval [0,1]; The multi-label classification loss L2 is calculated using the following focal loss:

9. The scene-aware multi-tag power equipment oil leakage identification method according to claim 8 is characterized in that: In step 7, based on the entropy loss function L1 in step 4 and the multi-label classification loss L2 in step 6, an overall loss function L is constructed, and the overall loss function L of the model is defined as: L=L1+λL2; Among them, λ represents the hyperparameter used to control the proportion of L1 and L2 loss functions; by solving the minimum value of the joint loss function L, the optimal parameter combination of the model is obtained, thereby obtaining the trained scene-aware multi-label oil leakage recognition model f, that is, the mapping function f:x→y from the image space to the label category space is obtained, where x represents the input picture.

10. The method for identifying oil leakage of electric equipment based on multi-tags based on scene perception according to claim 9 is characterized in that: The step 8 is specifically as follows: Based on the scene-aware multi-label oil leakage recognition model trained in step 7, the power equipment image I′ is input into the model f to obtain the binary vector representation of the oil leakage label [y′1,y′2,…,y′ N ]; If y′ i =1, it means that there is a label c in the power equipment image I′ i Corresponding oil leakage type; If y′ i = 0, it means that there is no label c in the power equipment image I′ i In the corresponding oil leakage type.

Images