A multi-label method for identifying oil leakage in power equipment based on scene perception

Abstract

The present invention belongs to the field of computer image processing technology, and specifically discloses a multi-label oil leakage identification method for power equipment based on scene perception. The method is based on the characteristic that the co-occurrence relationship of different types of labels in an image is closely related to the image scene, and proposes a multi-label oil leakage identification model based on scene perception. The model realizes fine-grained modeling of the co-occurrence relationship of labels at the scene level through scene category detection, label co-occurrence modeling, and scene classification loss function calculation. Then, a label graph is constructed, and message propagation and feature update are performed on the label graph, so that there are more feature interactions between labels with high co-occurrence probabilities in different scenes, thereby improving each other's visual representation and promoting the identification of oil leakage labels. The multi-label oil leakage identification method for power equipment based on scene perception proposed by the present invention can not only identify the oil-stained area on the power equipment, but also accurately judge different types of oil leakage, thereby realizing multi-label oil leakage identification.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer image processing and relates to a multi-label power equipment oil leakage identification method based on scene perception. Background Art

[0002] Power equipment, as a crucial component of power generation, transmission, transformation, and distribution, is essential infrastructure for energy production and conversion. Common examples include transformers, gas relays, switchgear, and submersible oil pumps. The stable and reliable operation of these devices is directly related to the safety of power production and transmission. However, after prolonged, high-intensity operation, power equipment is prone to oil leakage. This is particularly true in extreme weather conditions, such as severe cold, high temperature, and high humidity. Since power equipment is often exposed outdoors, oil leakage is more likely to occur. Extensive oil leakage from power equipment not only degrades the equipment's insulation performance, leading to safety incidents such as short circuits and fires, but also contaminates land and vegetation, damaging the local ecological environment. Therefore, identifying oil leakage from power equipment based on images captured by cameras surrounding the facility is crucial for preventing power safety incidents and environmental pollution.

[0003] Once oil leakage occurs in power equipment, it often spreads across multiple parts, resulting in oil pollution in multiple parts. Therefore, the pictures of power equipment taken by the camera may contain multiple types of oil leakage. Figure 2 and Figure 3 Pictures of electrical equipment with two or more types of oil leakage are shown. Figure 2 Including two types of oil leakage: ground oil pollution and pipeline oil pollution. Figure 3 There are three types of oil leakage: ground oil stains, transformer oil stains, and valve oil stains. For the pictures of power equipment taken by the camera, machine vision technology is needed to identify the different types of oil leakage in the pictures, so as to locate the oil leakage parts of the power equipment, so as to prompt maintenance personnel to intervene and deal with them accurately.

[0004] Existing machine vision-based methods for identifying oil leaks in power equipment are often limited to detecting the presence of oil leaks in a given image—a binary classification task—and struggle to accurately identify the multiple types of leaks contained in an image. If each type of oil leak in a power equipment image is considered a label, then the multi-label image-based oil leak detection problem can be considered a multi-label image classification problem.

[0005] In real life, the co-occurrence relationship of image labels is closely related to the scene information. For example, in photos of substation oil storage cabinets, "pipes" often appear together with "valves." In photos of main transformers, "transformers" often appear together with "ground." This indicates that the power equipment scene contained in the image directly affects the co-occurrence probability of different oil leakage labels. However, current methods such as recurrent neural networks and graph neural networks for solving multi-label image classification problems rarely consider the co-occurrence relationship of different labels, resulting in poor interpretability. Some methods based on label co-occurrence probability matrices, while having good interpretability, often only consider label co-occurrence probabilities at the dataset and instance levels, lacking fine-grained mining of label co-occurrence relationships, making them difficult to apply to multi-label power equipment oil leakage identification. Summary of the Invention

[0006] The purpose of the present invention is to propose a multi-label oil leakage identification method for power equipment based on scene perception. It realizes multi-label oil leakage identification for power equipment pictures containing multiple oil leakage scenes through a multi-label oil leakage identification model based on scene perception.

[0007] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions:

[0008] Step 1. Build a scene-aware multi-label oil leakage recognition model;

[0009] The multi-label oil leakage recognition model includes a label feature extraction module, a visual feature extraction module, a cross-modal feature alignment module, a scene-aware label co-occurrence module, a semantic interaction module, and a probability distribution calculation module.

[0010] Step 2. Build a dataset of labeled images of oil leakage from power equipment.

[0011] For the power equipment oil leakage image dataset, the label feature extraction module is used to extract the text semantic features of each oil leakage label, and the visual feature extraction module is used to extract the spatial visual features of each power equipment oil leakage image;

[0012] Step 3. Establish a cross-modal alignment relationship between images and labels through the cross-modal feature alignment module;

[0013] Step 4. The scene-aware tag co-occurrence module models the scene-aware tag co-occurrence relationship and constructs the entropy loss function through scene category detection, tag co-occurrence modeling, and scene classification loss function calculation;

[0014] Step 5. The semantic interaction module constructs a tag graph and uses the tag co-occurrence probability matrix to guide the semantic interaction between tags, improving the visual feature representation of tags with high co-occurrence probability, i.e., visual representation;

[0015] Step 6. The probability distribution calculation module inputs the improved label visual feature representation into the feedforward neural network to calculate the probability distribution of different labels on the image and construct the multi-label classification loss;

[0016] Step 7. Based on the entropy loss function and multi-label classification loss given in steps 4 and 6, a joint loss function is constructed, and the scene-aware multi-label oil spill recognition model is trained based on the joint loss function.

[0017] Step 8. Based on the trained scene-aware multi-label oil leakage recognition model, the input image is subjected to multi-label oil leakage recognition of power equipment to obtain a multi-label oil leakage recognition result of power equipment.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] As described above, the present invention describes a multi-label oil leakage identification method for power equipment based on scene perception. The method is based on the characteristic that the co-occurrence relationship of different types of labels in the picture is closely related to the picture scene, and proposes a multi-label oil leakage identification model based on scene perception. The model realizes fine-grained modeling of the label co-occurrence relationship at the scene level through scene category detection, label co-occurrence modeling, and scene classification loss function calculation; by constructing a label graph and performing message propagation and feature update on the label graph, there are more feature interactions between labels with high co-occurrence probability in different scenes, thereby improving each other's visual representation and promoting subsequent oil leakage label identification. The multi-label oil leakage identification method for power equipment based on scene perception of the present invention can not only identify the oily areas on the power equipment, but also accurately judge different types of oil leakage. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for describing the embodiments or the prior art.

[0021] Figure 1 Flowchart of a method for identifying oil leakage in power equipment using multiple tags based on scene perception according to an embodiment of the present invention;

[0022] Figure 2 This is a picture of an electric power equipment with two types of oil leakage in an embodiment of the present invention;

[0023] Figure 3 This is a picture of power equipment with three types of oil leakage in an embodiment of the present invention;

[0024] Figure 4 Schematic diagram of the structure of the scene-aware multi-label oil leakage recognition model in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0026] Example

[0027] In this embodiment, a scene-aware multi-label oil leakage identification method for power equipment is proposed. For power equipment pictures containing multiple oil leakage scenes, based on the characteristics that label co-occurrence relationships are closely related to image scenes, a scene-aware multi-label oil leakage identification model is designed, and multi-label oil leakage identification of power equipment is performed based on this model.

[0028] like Figure 1 As shown in FIG, the multi-label power equipment oil leakage identification method based on scene perception specifically includes the following steps:

[0029] Step 1. Build a scene-aware multi-label oil leak recognition model.

[0030] Among them, the multi-label oil leakage recognition model includes a label feature extraction module, a visual feature extraction module, a cross-modal feature alignment module, a scene-aware label co-occurrence module, a semantic interaction module, and a probability distribution calculation module.

[0031] Step 2. Extract the label semantic features and spatial visual features of the image.

[0032] First, we construct a dataset of labeled images of oil leakage from power equipment. The process is as follows:

[0033] M pictures of oil leakage from power equipment are collected from power plants, substations and other places, and each picture is manually labeled. The label category set is in Generate a labeled dataset of oil leakage images of power equipment Among them, c1 to c N They represent the 1st to Nth oil leakage labels respectively, N represents the number of label categories, and M is a natural number.

[0034] In this embodiment, a total of 8 types of oil leakage are marked, namely ground oil pollution, transformer oil pollution, oil storage cabinet oil pollution, pipeline oil pollution, valve oil pollution, switch cabinet oil pollution, submersible pump oil pollution, and gas relay oil pollution, that is, N=8.

[0035] Then, for the dataset of oil leakage images of power equipment, the label feature extraction module is used to extract the text semantic features of each oil leakage label, and the visual feature extraction module is used to extract the spatial visual features of each oil leakage image of power equipment.

[0036] In this embodiment, the label feature extraction module is implemented, for example, using a Bert pre-trained language model, and the visual feature extraction module in this embodiment is implemented, for example, using a ResNet pre-trained visual backbone network.

[0037] A dataset of oil leakage images of power equipment containing multiple labels For any picture I of oil leakage from power equipment, we first use the Bert pre-trained language model to extract the text semantic features of its oil leakage label.

[0038] If image I contains label c i , then label c i The embedding representation is: i =Φ T (c i ).

[0039] Where i∈{1,2,…,N}, c i Represents a collection of label categories The i-th oil leakage label, t i is the label embedding representation, that is, the extracted label c i The text semantic features, d is the dimension of the label embedding representation; Φ T (·) is the Bert pre-trained language model, which is used to label the input text c i Mapped to embedding representation t i .

[0040] Secondly, the ResNet pre-trained visual backbone network is used to extract the spatial visual features V of the power equipment oil leakage image I:

[0041] V=Φ V (I); Φ V (·) represents the ResNet pre-trained visual backbone network, h and w represent the length and width of the visual feature in the spatial dimension, respectively, and c represents the dimension of the visual feature. The image spatial visual feature extracted by the ResNet pre-trained visual backbone network is actually a 3D matrix, and c is the dimension of each element of the matrix.

[0042] The spatial visual feature V is expanded and expressed as

[0043] in, v j The visual feature representing the jth spatial region on the power equipment oil leakage image I is the visual feature representation, j∈{1,2,…,m}, m represents the number of spatial regions in the power equipment oil leakage image I, and m=h×w.

[0044] Step 3. Establish a cross-modal alignment relationship between images and labels through the cross-modal feature alignment module.

[0045] In this embodiment, the cross-modal feature alignment module adopts a low-rank bilinear pooling model.

[0046] The cross-modal alignment relationship between equipment oil leakage pictures and text labels is established through the low-rank bilinear pooling model. The low-rank bilinear pooling model has good feature fusion and alignment performance, and can significantly reduce the amount of calculation by reducing the dimension of the joint feature space; it represents the visual feature v j and label embedding representation t i Mapping to a unified joint feature space:

[0047]

[0048] Among them, x i,j Represents the visual feature representation v j and label embedding representation t i Feature representation in a unified joint feature space. P, U, W represent the learnable parameter matrix, b represents the learnable parameter vector, tanh(·) represents the hyperbolic tangent function, and ⊙ represents the element-by-element multiplication operation.

[0049] Where d1 and d2 represent the dimensions of the joint feature space and the output space, respectively.

[0050] Then, label c i The normalized correlation with the jth region on image I is calculated as follows:

[0051]

[0052] Among them, x i,j′ is the feature representation of the visual features of the image and the textual semantic features of the label in a unified joint feature space, j′∈{1,2,…,m}; α i,j represents the normalized cross-modal semantic relevance weight, Φ a (·) is a learnable feedforward fully connected neural network, which is used to map the input vector into a probability value.

[0053] Then, based on the normalized cross-modal semantic relevance weights, the visual features v of all regions on the image I are j Perform weighted summation to obtain label c i Visual representation f on image I i :

[0054] Step 4. The scene-aware tag co-occurrence module models the scene-aware tag co-occurrence relationship and constructs the entropy loss function through scene category detection, tag co-occurrence modeling, and scene classification loss function calculation.

[0055] Step 4.1. Perform scene category detection. Assume that the data set of oil leakage images of power equipment There are K scene categories in total. Each scene category can actually be regarded as a type of power equipment image. For example, the same type of power equipment such as the main transformer photographed from different angles only corresponds to one scene category, namely the main transformer scene.

[0056] The global context features of the image I are used to predict the probability distribution of the oil leakage image I of the power equipment in K scene categories; the global context features of the image It is obtained by a global average pooling operation, which is calculated as follows:

[0057]

[0058] The probability π that image I belongs to the kth scene category k The calculation process is as follows:

[0059]

[0060] Among them, w k 、w k′ is the learnable parameter vector, Then we can get the probability distribution π∈{π1,π2,…,π k ,…,π K}.

[0061] Select the scene category with the highest probability in π as the scene category of image I, that is:

[0062] s=argmax k∈{1,2,…,K} π k ;

[0063] Where s is the scene category to which image I belongs, that is, the prediction result obtained by scene category detection.

[0064] Step 4.2. Perform label co-occurrence modeling. Maintain a label co-occurrence frequency matrix for each of the K scene categories, and continuously update these label co-occurrence frequency matrices during training based on the prediction results obtained from the scene category detection.

[0065] The label co-occurrence frequency matrix of scene category s is updated as follows:

[0066] Z s =Z s +yy T .

[0067] Among them, Z s Represents the label co-occurrence frequency matrix of scene category s, Z sThe initial value of all elements in is 0; y represents the binary vector representation of the label of image I, that is, y=[y1,y2,…,y N ], y1 to y N are the 1st to Nth components of y respectively. If image I has the i-th label, then y i =1, otherwise y i =0.

[0068] As the training process continues and the label co-occurrence frequency matrix is ​​continuously updated, Z s The diagonal elements in Counted the label c i The number of times it appears in scene category s, not the diagonal elements Then the label c is counted i and label c r The number of times the label c co-occurs in the scene category s; therefore, in the scene category s, when the label c r When it appears, label c i The probability of occurrence The calculation process is as follows:

[0069] Among them, r∈[1,N]; through the above method to achieve label co-occurrence modeling, with continuous learning, the label co-occurrence frequency matrix {P 1 ,P 2 ,…,P K} will eventually converge to a stable distribution where P 1 To P K Represent the label co-occurrence frequency matrix of the 1st to Kth scene categories respectively.

[0070] Step 4.3. Calculate the scene classification loss function.

[0071] For each training batch of multi-label power equipment oil leakage picture sample set, the scene probability distribution predicted by scene category detection on the g-th power equipment oil leakage picture is expressed as: g ∈{π g1 ,π g2 ,…,π gK}.

[0072] Among them, g∈{1,2,…,G}, G represents the number of image samples in the current batch; π g1 to π gK Respectively represent the probability that the picture g belongs to the 1st to the Kth scene category, then the entropy loss function L1 of the training batch samples is expressed as:

[0073]

[0074] Among them, when the loss function L1 is smaller, the scene probability distribution of each image sample is sharper, which means that each image in the batch of samples is clearly classified into a certain scene category.

[0075] During the training process, as the loss L1 continues to decrease, the scene category to which each picture belongs can be identified more accurately.

[0076] Step 5. Improve the visual representation between high co-occurrence probability tags through semantic interaction.

[0077] This embodiment constructs a tag graph through a semantic interaction module and uses a tag co-occurrence probability matrix to guide semantic interaction between tags, thereby improving the visual feature representation, i.e., visual representation, of tags with high co-occurrence probability.

[0078] Based on the label co-occurrence frequency matrix {P 1 ,P 2 ,…,P K} and the image scene probability distribution {π1,π2,…,π K}, calculate the total label co-occurrence probability matrix P of the power equipment oil leakage image I by weighted average I :

[0079] On this basis, construct the label map Node Collection With the dataset Tag collection Correspondingly, each node v i The initial features are represented by label c i Visual representation f on image I i Initialize, the edge set ε represents the co-occurrence relationship between neighbor nodes, and its weight is the co-occurrence probability, which is determined by the total label co-occurrence probability matrix P I Initialize.

[0080] Then the label graph propagation learning is carried out to let each node learn the representation of the context; specifically, the node v i The embedding representation at round t is express, For label c i visual representation of for The initial value of is the initial representation, It aggregates context information from neighboring nodes at time t The calculation process is as follows:

[0081] Where N represents the number of labels, that is, the number of graph nodes in the label graph; Represents the total label co-occurrence probability matrix P I The elements in represents the embedding representation of neighbor node r at round t-1.

[0082] The method of the present invention is based on a gated cyclic update mechanism, which is a widely used neural network. The gated neural network includes two gating units, an update gate and a reset gate, which are used to control a specific proportion of past information to be forgotten.

[0083] The feature of the current node is updated using the context information aggregated from neighboring nodes. The calculation process is as follows:

[0084]

[0085] in, is the intermediate hidden vector of the gated loop update mechanism, σ(·) represents the Sigmoid function; W z and U z is the parameter matrix of the update gate, where W r and U r is the parameter matrix of the reset gate, where And W h and U h is the parameter matrix used to calculate the hidden state, where Through this message propagation and feature updating mechanism, there will be more feature interactions between tags with high co-occurrence probability in the current scenario, thereby improving each other's visual representation and facilitating subsequent oil spill tag recognition.

[0086] Step 6. The probability distribution calculation module inputs the improved label visual feature representation into the feedforward neural network to calculate the probability distribution of different labels on the oil leakage pictures of power equipment and construct a multi-label classification loss.

[0087] After t rounds of feature updates, the label c i Visual representation of Its initial characterization Splice and input into a feedforward neural network to obtain and The joint feature representation o i :

[0088] Among them, [·;·] represents the splicing operation, Φ C (·) represents a learnable feedforward neural network.

[0089] On this basis, we further use the linear binary classifier to calculate the label c iThe probability p of appearing on the input image, that is, the oil leakage picture I of the power equipment i : Among them, w i represents the learnable parameter vector, σ(·) represents the Sigmoid function, which is used to map real values ​​to probability values ​​in the [0, 1] interval.

[0090] The following focal loss is used to calculate the multi-label classification loss L2:

[0091]

[0092] Step 7. Optimize the joint loss function for model training.

[0093] Based on the entropy loss function and multi-label classification loss given in steps 4 and 6, a joint loss function is constructed, and the scene-aware multi-label oil leakage recognition model is trained based on the joint loss function.

[0094] Based on steps 2 to 6, the overall loss function L of the model is defined as: L = L1 + λL2; where λ represents a hyperparameter used to control the weights of the L1 and L2 loss functions.

[0095] By solving the minimum value of the above joint loss function L, we can obtain the optimal parameter combination of the model, thereby obtaining a trained scene-aware multi-label oil spill recognition model f. This model can actually be regarded as a mapping function f:x→y from image space to label category space. Where x represents the input image, and y is the binary vector representation of the label of image x, that is, y=[y1,y2,…,y N ], if the image x has the i-th label, then y i =1, otherwise y i =0.

[0096] In the training phase, fully supervised multi-label image classification aims to In

[15] , we learn the mapping function f:I→Y from the image space to the label category space. Here, the mapping function f(·) usually refers to a parameterized neural network model.

[0097] In the test phase, given any image I′ to be identified, the model will output y′=f(I′) as the predicted label of the image.

[0098] Step 8. Perform multi-label oil leakage detection on power equipment based on the trained model: Based on the trained scene-aware multi-label oil leakage detection model, perform multi-label oil leakage detection on the input image to obtain the multi-label oil leakage detection result. Specifically, input any image of power equipment containing an oil leakage event, and use the model to identify the distribution of the image across N labels.

[0099] Based on the scene-aware multi-label oil leakage recognition model trained in step 7, given any picture I′ of power equipment containing an oil leakage event, the picture I′ is input into the model f to obtain the binary vector representation of the oil leakage label [y′1,y′2,…,y′ N ]; if y′ i =1, it means that there is a label c in the power equipment image I′ i Corresponding oil leakage type; if y′ i = 0, it means that there is no label c in the power equipment image I′ i In the corresponding oil leakage type.

[0100] It should be noted that the multi-label power equipment oil leakage identification method based on scene perception in this embodiment is also applicable to single-label power equipment oil leakage identification, because single-label oil leakage can actually be regarded as a special case of multi-label oil leakage. If there is only one type of oil leakage in the power equipment picture, the multi-label power equipment oil leakage identification problem degenerates into a single-label power equipment oil leakage identification problem, which will not be elaborated here.

[0101] Of course, the above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above-mentioned embodiments. It should be noted that all equivalent substitutions and obvious deformation forms made by any technician familiar with this field under the guidance of this specification fall within the substantive scope of this specification and should be protected by the present invention.

Claims

1. A multi-label method for identifying oil leakage in power equipment based on scene perception, characterized in that: The steps include: Step 1. Build a scene-aware multi-label oil leakage recognition model; The multi-label oil leakage recognition model includes a label feature extraction module, a visual feature extraction module, a cross-modal feature alignment module, a scene-aware label co-occurrence module, a semantic interaction module, and a probability distribution calculation module. Step 2. Build a dataset of labeled images of oil leakage from power equipment. For the power equipment oil leakage image dataset, the label feature extraction module is used to extract the text semantic features of each oil leakage label, and the visual feature extraction module is used to extract the spatial visual features of each power equipment oil leakage image; Step 3. Establish a cross-modal alignment relationship between images and labels through the cross-modal feature alignment module; Step 4. The scene-aware tag co-occurrence module models the scene-aware tag co-occurrence relationship through scene category detection and tag co-occurrence modeling; and constructs the entropy loss function through scene classification loss function calculation. Step 5. The semantic interaction module constructs a tag graph and uses the tag co-occurrence probability matrix to guide the semantic interaction between tags, improving the visual feature representation of tags with high co-occurrence probability, i.e., visual representation; Step 6. The probability distribution calculation module inputs the improved label visual feature representation into the feedforward neural network to calculate the probability distribution of different labels on the image and construct the multi-label classification loss; Step 7. Based on the entropy loss function and multi-label classification loss given in steps 4 and 6, a joint loss function is constructed, and the scene-aware multi-label oil spill recognition model is trained based on the joint loss function. Step 8. Based on the trained scene-aware multi-label oil leakage recognition model, the input image is subjected to multi-label oil leakage recognition of power equipment to obtain a multi-label oil leakage recognition result of power equipment.

2. The multi-tag power equipment oil leakage identification method based on scene perception according to claim 1 is characterized in that: In step 2, the process of constructing a dataset of labeled images of oil leakage from power equipment is as follows: M pictures of oil leakage from power equipment are collected from power plants and substations, and each picture is manually labeled. The label category set is Generate a labeled dataset of oil leakage images of power equipment M is a natural number; Among them, c1 to c N They represent the 1st to Nth oil leakage labels respectively, and N represents the number of label categories.

3. The multi-tag power equipment oil leakage identification method based on scene perception according to claim 2 is characterized in that: In step 2, the label feature extraction module uses the Bert pre-trained language model, and the visual feature extraction module uses the ResNet pre-trained visual backbone network. The process of extracting text semantic features and spatial visual features is as follows: A dataset of oil leakage images of power equipment containing multiple labels For any image I of oil leakage from power equipment, use the Bert pre-trained language model to extract the text semantic features of its oil leakage label; If image I contains label c i , then label c i The embedding representation is: i =Φ T (c i ); Where i∈{1,2,…,N}, c i Represents a collection of label categories The i-th oil leakage label, t i is the label embedding representation, that is, the extracted label c i The text semantic features, d is the dimension of the label embedding representation; Φ T (·) is the Bert pre-trained language model, which is used to label the input text c i Mapped to embedding representation t i ; Use the ResNet pre-trained visual backbone network to extract the spatial visual features V of the power equipment oil leakage image I: V=Φ V (I); where Φ V (·) represents the ResNet pre-trained visual backbone network, h and w represent the length and width of the visual feature in the spatial dimension, respectively, and c represents the dimension of the visual feature; The spatial visual feature V is expanded and expressed as in, v j The visual feature representing the jth spatial region on the power equipment oil leakage image I is the visual feature representation, j∈{1,2,…,m}, m represents the number of spatial regions in the power equipment oil leakage image I, and m=h×w.

4. The multi-tag method for identifying oil leakage in power equipment based on scene perception according to claim 1 is characterized in that: In step 2, the categories of the oil leakage labels marked include ground oil stains, transformer oil stains, oil storage cabinet oil stains, pipeline oil stains, valve oil stains, switch cabinet oil stains, submersible pump oil stains, and gas relay oil stains.

5. The method for identifying oil leakage in power equipment based on multi-tag scene perception according to claim 3 is characterized in that: The step 3 is specifically as follows: The cross-modal feature alignment module uses a low-rank bilinear pooling model to establish a cross-modal alignment relationship between equipment oil leakage images and text labels, and represents the visual feature v j and label embedding representation t i Mapping to a unified joint feature space: Among them, x i,j Represents the visual feature representation v j and label embedding representation t i Feature representation in a unified joint feature space; P, U, W represent the learnable parameter matrix, b represents the learnable parameter vector, tanh(·) represents the hyperbolic tangent function, and ⊙ represents the element-by-element multiplication operation; Where d1 and d2 represent the dimensions of the joint feature space and the output space respectively; Tag c i The normalized correlation with the jth region on the power equipment oil leakage image I is calculated as follows: Among them, x i,j′ is the feature representation of the visual features of the oil leakage picture of power equipment and the textual semantic features of the label in a unified joint feature space, j′∈{1,2,…,m}; α i,j represents the normalized cross-modal semantic relevance weight, Φ a (·) is a learnable feedforward fully connected neural network used to map the input vector into a probability value; Based on the normalized cross-modal semantic relevance weights, the visual features v of all regions on the image I are j Perform weighted summation to obtain label c i Visual representation f on image I i :

6. The method for identifying oil leakage in power equipment based on multi-tag scene perception according to claim 5 is characterized in that: The step 4 is specifically as follows: Step 4.

1. Perform scene category detection; The global context features of the image are used to predict the probability distribution of the oil leakage picture I of the power equipment on K scene categories. The global context features of the image It is obtained by a global average pooling operation, and the calculation process is as follows: The probability π that image I belongs to the kth scene category k The calculation process is as follows: Among them, w k 、w k′ is the learnable parameter vector, k∈{1,2,…,K}; k′∈{1,2,…,K}; then we get the probability distribution π∈{π1,π2,…,π k ,…,π K }; Select the scene category with the highest probability in π as the scene category of image I, that is: s=argmax k∈{1,2,…,K} π k ; Where s is the scene category to which image I belongs, that is, the prediction result obtained by scene category detection; Step 4.

2. Perform tag co-occurrence modeling; Maintain a label co-occurrence frequency matrix for each of the K scene categories, and update the label co-occurrence frequency matrix during training based on the prediction results obtained from the scene category detection. The label co-occurrence frequency matrix of scene category s is updated as follows: Z s =Z s +yy T ; Among them, Z s Represents the label co-occurrence frequency matrix of scene category s, Z s The initial value of all elements in is 0; y represents the binary vector representation of the label of image I, y=[y1,y2,…,y N ], where y1 to y N are the 1st to Nth components of y respectively. If image I has the i-th label, then y i =1, otherwise y i =0; In scene category s, when label c r When it appears, label c i The probability of occurrence The calculation process is as follows: Among them, r∈[1,N], is the diagonal element of the label co-occurrence frequency matrix of scene category s, used to count label c i The number of times it appears in scene category s; is the non-diagonal element of the label co-occurrence frequency matrix of scene category s, used to count label c i and label c r The number of co-occurrences in scene category s; As the model continues to learn, the label co-occurrence frequency matrix {P 1 ,P 2 ,…,P K } eventually converges to a stable distribution where P 1 To P K Represents the label co-occurrence frequency matrix of the 1st to Kth scene categories respectively; Step 4.

3. Calculate the scene classification loss function; For each training batch of multi-label power equipment oil leakage picture sample set, the predicted scene probability distribution π on the g-th power equipment oil leakage picture g Expressed as: π g ∈{π g1 ,π g2 ,…,π gK }; Among them, g∈{1,2,…,G}, G represents the number of image samples in the current batch; π g1 to π gK They represent the probability that the image g belongs to the 1st to Kth scene categories respectively; the entropy loss function L1 of the training batch samples is expressed as:

7. The method for identifying oil leakage in power equipment based on multi-tag scene perception according to claim 6 is characterized in that: The step 5 is specifically as follows: Based on the label co-occurrence frequency matrix {P 1 ,P 2 ,…,P K } and the image scene probability distribution {π1,π2,…,π K }, calculate the total label co-occurrence probability matrix P of the power equipment oil leakage image I by weighted average I : Building a label graph Node Collection With the dataset Tag collection Correspondingly, each node v i The initial features are represented by label c i Visual representation f on image I i Initialize; the edge set ε represents the co-occurrence relationship between neighbor nodes, and its weight is the co-occurrence probability, which is determined by the total label co-occurrence probability matrix P I Initialize; Perform label graph propagation learning to enable each node to learn the representation of the context; i The embedding representation at round t is express, For label c i visual representation of for The initial value of is the initial representation, Context information aggregated from neighboring nodes at time t The calculation process is: Where N is the number of labels, that is, the number of graph nodes in the label graph; Represents the total label co-occurrence probability matrix P I The elements in represents the embedding representation of neighbor node r at round t-1; based on the gated loop update mechanism, the context information aggregated from neighbor nodes is used to update the features of the current node. The calculation process is as follows: in, is the intermediate hidden vector of the gated recurrent update mechanism, σ(·) represents the Sigmoid function; W z and U z is the parameter matrix of the update gate, where W r and U r is the parameter matrix of the reset gate, where W h and U h is the parameter matrix used to calculate the hidden state, where Through the message propagation and feature update mechanism, there are more feature interactions between tags with high co-occurrence probability in the current scenario, thereby improving each other's visual representation and obtaining an improved tag visual feature representation.

8. The method for identifying oil leakage in power equipment based on multi-tag scene perception according to claim 7 is characterized in that: The step 6 is specifically as follows: After t rounds of feature updates, the label c i Visual representation of Its initial characterization Splice and input into a feedforward neural network to obtain and The joint feature representation o i : Among them, [·;·] represents the splicing operation, Φ C (·) represents a learnable feedforward neural network; Compute the label c using a linear binary classifier i The probability p of appearing on the input image, that is, the oil leakage picture I of the power equipment i : Among them, w i represents the learnable parameter vector, σ(·) represents the Sigmoid function, which is used to map real values ​​to probability values ​​in the interval [0,1]; The multi-label classification loss L2 is calculated using the following focal loss:

9. The method for identifying oil leakage in power equipment based on multi-tag scene perception according to claim 8 is characterized in that: In step 7, based on the entropy loss function L1 in step 4 and the multi-label classification loss L2 in step 6, an overall loss function L is constructed, and the overall loss function L of the model is defined as: L = L1 + λL2; Here, λ represents a hyperparameter used to control the weight of the L1 and L2 loss functions. By solving the minimum value of the joint loss function L, the optimal parameter combination of the model is obtained, thereby obtaining a trained scene-aware multi-label oil leakage recognition model f, that is, a mapping function f:x→y from image space to label category space, where x represents the input image.

10. The method for identifying oil leakage in power equipment based on multi-tag scene perception according to claim 9 is characterized in that: The step 8 is specifically as follows: Based on the scene-aware multi-label oil leakage recognition model trained in step 7, the power equipment image I′ is input into the model f to obtain the binary vector representation of the oil leakage label [y′1,y′2,…,y′ N ]; If y′ i =1, it means that there is a label c in the power equipment image I′ i Corresponding oil leakage type; If y′ i = 0, it means that there is no label c in the power equipment image I′ i In the corresponding oil leakage type.

Images