Switch closing identification model construction method based on visual large model and switch closing identification method

Abstract

The invention discloses a large visual model-based switch closing identification model construction method and a switch closing identification method.The switch closing identification model construction method comprises the steps of S1, collecting switch images and making a data set which comprises an image file and a label file, and recording the type and state of the corresponding switch image in the label file; s2, building a ViT model, S3, building a CLIP model, and S4, training the model; training a switch type classifier by using a CLIP model, and training a switch closing identification classifier by using a ViT model; and S5, model fusion: fusing the switch type classifier based on the CLIP model and the switch closing identification classifier based on the ViT model so as to obtain a switch closing identification model. According to the method and the device, the closing state of the switch can be efficiently and accurately identified under different switch types.

Description

Technical Field

[0001] The present application belongs to the technical field of switch closure recognition, and in particular, relates to a switch closure recognition model construction method based on a large visual model and a switch closure recognition method. Background Art

[0002] The application of switch closure recognition technology is becoming more and more extensive, especially in the fields of power, transportation, industrial automation, etc. In the power system, switch status recognition can effectively ensure the safe operation of equipment.

[0003] At present, common switch closure recognition technologies include: image processing and computer vision technology, which can effectively remove noise and highlight the target area by separating switch areas and extracting edge features through image segmentation, edge detection and morphological processing. Deep learning methods use convolutional neural networks (CNN) to extract switch features, which are suitable for recognition in complex scenes, while target detection models such as YOLO and Faster R-CNN have high accuracy in real-time switch positioning and recognition, and transfer learning can also effectively improve model performance when there is less data; traditional machine learning methods such as support vector machines (SVM), random forests and K nearest neighbors (KNN) rely on manual features and can provide better classification effects for switch images with low feature dimensions; in addition, multi-sensor data fusion technology combines image, sound, vibration and infrared sensor information to further improve the accuracy and robustness of switch state recognition.

[0004] Most switch closure recognition methods are based on traditional machine learning algorithms or feature engineering methods, such as edge detection and color segmentation based on image processing. Although these methods can identify the switch state to a certain extent, their accuracy is limited by the influence of ambient lighting, differences in switch appearance, and shooting angles, resulting in low robustness and accuracy of recognition. In addition, traditional methods rely on manual feature extraction and annotation, which is difficult to adapt to various types of switches and lacks versatility and scalability. At present, new types of switches are constantly emerging, and existing switch closure recognition methods are difficult to identify these new types of switch closure states. Summary of the invention

[0005] The purpose of this application is to overcome the problems of the prior art and disclose a switch closure recognition model construction method based on a large visual model and a switch closure recognition method. Through this application, it is possible to efficiently and accurately identify the closed state of a switch under different switch types.

[0006] The purpose of this application is achieved through the following technical solutions:

[0007] A method for constructing a switch closure recognition model based on a visual large model, wherein the switch closure recognition model is constructed Method Package include:

[0008] S1: Collect switch images and create a data set, the data set includes: an image file and a label file, the label file records the type and state of the corresponding switch image;

[0009] S2: Build the ViT model,

[0010] S3: Build the CLIP model,

[0011] S4: Model training: Use the CLIP model to train a switch type classifier, and use the ViT model to train a switch closure recognition classifier;

[0012] S5: Model fusion, fusing the switch type classifier based on the CLIP model and the switch closure recognition classifier based on the ViT model to obtain a switch closure recognition model.

[0013] According to a preferred embodiment, the ViT model in step S2 is constructed based on the Transformer encoder module.

[0014] The Transformer encoder module consists of 3 encoder layers, each of which consists of two parts:

[0015] Self-attention mechanism and feed-forward neural networks;

[0016] In each encoder layer, self-attention calculation is first performed, and attention weight calculation is completed by querying query, key key, and value value, and the value is weighted and summed according to the corresponding weight.

[0017] After the self-attention calculation, each output vector is processed by a feed-forward neural network consisting of two linear transformations and a ReLU activation function.

[0018] Finally, the output of the encoder is obtained through residual connection and layer normalization operations.

[0019] According to a preferred embodiment, the ViT model is provided with an output layer, and the probability of each category is obtained by using a fully connected layer and performing a softmax operation.

[0020] According to a preferred embodiment, the process of weighted summing of values ​​according to weights is as follows:

[0021]

[0022] Among them, Q is the query matrix, which represents the current input, K is the key matrix, which represents the relevance of the input, V is the value matrix, which represents the output information corresponding to the key, and d k is the dimension of the key, used for scaling.

[0023] According to a preferred embodiment, the ViT model is configured to divide the input image into small patches of fixed size, and then flatten and map each patch to an embedding space of fixed dimension; and introduce position encoding to represent the absolute position of each small patch in the image.

[0024] According to a preferred embodiment, step CLIP adopts a dual encoder architecture, including an image encoder and a text encoder.

[0025] The image encoder uses the ViT model to extract image features. The image I is passed through the image encoder f I Generate image embedding vector v I ;

[0026] The text encoder uses the Transformer encoder module to encode the text label into an embedding vector, and the text description T is passed through the text encoder f T Generate text embedding vector v t .

[0027] According to a preferred embodiment, in step S4, the CLIP model is used to train the switch type classifier, and the contrast loss is used to measure the similarity between the image and the text. The training step includes:

[0028] Forward propagation: In each training step, images are passed through the image encoder to get image embeddings, and texts are passed through the text encoder to get text embeddings.

[0029] Compute Similarity: Compute the cosine similarity between the image and text embeddings:

[0030]

[0031] Calculate the loss: According to the contrast loss function, calculate the similarity between each image and its corresponding text, and minimize the similarity with other categories of text.

[0032] According to a preferred embodiment, in step S4, the ViT model is used to train a switch closure recognition classifier, a cross entropy loss function is used to perform image classification tasks, and before the features are input into the transfomer compiler, the type encoding of the switch is added to guide the model training.

[0033] On the other hand, the present application also discloses:

[0034] A switch closure recognition method based on a visual large model, characterized in that a switch closure recognition model constructed by the switch closure recognition model construction method according to any one of claims 1 to 8 is used,

[0035] The switch closure identification method comprises:

[0036] Take switch images and perform image preprocessing;

[0037] Output the processed image to the switch type classifier based on the CLIP model to obtain the type of switch in the corresponding image;

[0038] According to the obtained switch type, the type code is embedded in the switch closure recognition classifier based on the ViT model to identify whether the switch is closed;

[0039] The output is whether the switch is closed.

[0040] The aforementioned main scheme of the present application and its further options can be freely combined to form multiple schemes, all of which are schemes that can be adopted and claimed for protection in the present application. After understanding the scheme of the present application, those skilled in the art can understand that there are multiple combinations based on the prior art and common knowledge, all of which are technical schemes to be protected by the present application, and they are not exhaustively listed here.

[0041] Beneficial effects of this application:

[0042] 1. The switch closure recognition model based on the visual large model provided by this application uses CLIP as the switch type classifier. Due to the characteristics of zero-shot learning, the switch type classifier already has the ability to match new categories with images through text descriptions, and there is no need to retrain or fine-tune the model. Therefore, for newly emerging switch styles, there is no need to retrain a switch type classifier.

[0043] 2. The switch closure recognition model based on the visual large model provided in this application uses ViT as the switch closure recognition classifier. ViT can process high-dimensional features, especially when processing images with a large number of details or complex structures, it is easier to capture rich information in the image. Through the global self-attention mechanism, it can better focus on the key areas in the image, so that the model can still effectively identify the state of the switch in a complex background.

[0044] 3. In the switch closure recognition method based on the visual large model provided in the present application, the method first identifies the type of switch and then identifies whether the switch is closed. The design of this step enables the model to efficiently and accurately identify the closed state of the switch under different switch types. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 The figure is a flow chart of the switch closure recognition method based on the visual large model of the present invention.

[0046] Figure 2 This is the reasoning process of the switch closure recognition model based on the visual large model of the present invention.

[0047] Figure 3 Schematic diagram of the Transformer encoder structure of the present invention. DETAILED DESCRIPTION

[0048] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0049] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0050] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, or the positions or positional relationships in which the product of the application is usually placed when in use. They are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific position, be constructed and operated in a specific position, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0051] In addition, the terms "horizontal", "vertical", "overhanging" and the like do not mean that the components are required to be absolutely horizontal or overhanging, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0052] In the description of this application, it should also be noted that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0053] In addition, the present application would like to point out that, in the present application, unless the specific structure, connection relationship, positional relationship, power source relationship, etc. are specifically written out, the structure, connection relationship, positional relationship, power source relationship, etc. involved in the present application are all known by those skilled in the art on the basis of the prior art without creative work.

[0054] Example 1

[0055] refer to Figure 1 As shown in the figure, a switch closure recognition method based on a visual large model is shown, and a switch closure recognition model is constructed using a switch closure recognition model construction method. The switch closure recognition method includes:

[0056] Step 1: Take a switch image and perform image preprocessing.

[0057] Preferably, the image preprocessing process includes: performing the same normalization operation on the captured image as the data set to maintain the consistency of the input data.

[0058] Step 2: Output the processed image to the switch type classifier based on the CLIP model to obtain the type of switch in the corresponding image.

[0059] Specifically, the input image is passed through the image encoder and the similarity is calculated with each text embedding vector, and the text with the highest similarity is taken as the predicted value of the switch type.

[0060] Step 3: According to the obtained switch type, embed the type code in the switch closure recognition classifier based on the ViT model to identify whether the switch is closed.

[0061] Specifically, the predicted value of the switch type is encoded and embedded into the features between the transfomer encoders in the ViT model, so that the model pays more attention to the features of this type of switch style to improve the accuracy of the output results.

[0062] Step 4: Output the result of whether the switch is closed.

[0063] Example 2

[0064] Based on Example 1, the reasoning process of the switch closure recognition model of this example based on the visual large model includes the following parts:

[0065] 1. Input image: This image has been normalized in the same way as the dataset to maintain the consistency of the input data. Assume that the image size is H×W×C, where H is the image height, W is the image width, and C is the number of image channels.

[0066] 2. Block operation: Divide the input image into small patches of fixed size. Assume that the size of each patch is P×P, and the image is divided into

[0067] 3. Flatten and linearize: image data of each patch It will be flattened and mapped to a D-dimensional embedding vector space through linear transformation. The formula is expressed as:

[0068]

[0069] in, is the i-th patch, W e is the embedding matrix, b e is the bias term.

[0070] 4. Add position encoding: Position encoding is introduced to represent the absolute position of each patch in the image. The position encoding is added to the embedding vector of the image patch to help the model understand the location of the patch. The formula is:

[0071]

[0072] Among them, p i is the code of the ith position, generated by sine and cosine functions. The formula is as follows:

[0073]

[0074] Here, k is the dimension index in the positional encoding, D is the embedding dimension, and i is the position index.

[0075] 5. Transformer encoder: see Figure 3 , consists of multiple encoder layers, each of which includes two main parts: a self-attention mechanism and a feedforward neural network, including the following steps:

[0076] (1) Input: Position-encoded sequence Z0.

[0077] (2) Multi-head attention: The input vector Z0 is transformed through three linear transformations to obtain the query vector Q, key vector K and value vector V:

[0078] Q=Z0W Q ,K=Z0W K ,V=Z0W V

[0079] Among them, W Q , W K and W V is the learnable parameter matrix.

[0080] The attention weights are calculated by Q, K, and V, and the values ​​are weighted summed according to these weights. The formula is as follows:

[0081]

[0082] Among them, d k is the dimension of the key, used for scaling.

[0083] Assuming there are h attention heads, multiple attention results can be calculated in parallel and concatenated:

[0084] MultiHead(Q,K,V)=Concat(head1,…,head h )W O

[0085] (3) Residual connection and layer normalization: Residual connection and layer normalization are applied to maintain information flow and training stability. Assuming the input is Z0 and the output of multi-head attention is M, the residual connection and layer normalization are:

[0086] Z1=LayerNorm(Z0+M)

[0087] (4) Feedforward network: After the self-attention module is a feedforward neural network (FFN). This network consists of two linear layers and a nonlinear activation function (such as ReLU):

[0088] FFN(Z1)=ReLU(Z1W1+b1)W2+b2

[0089] (5) Residual connection and layer normalization: The operation is the same as step (3).

[0090] (6) Stacked encoder layer: It contains N such encoder layers stacked, and the output of each layer serves as the input of the next layer.

[0091] (7)Output.

[0092] 6. Similarity comparison: The image is passed through the image encoder to get the image embedding, and the text is passed through the text encoder to get the text embedding. Then the cosine similarity between the image and text embedding is calculated:

[0093]

[0094] 7. Predict switch type: Select the text corresponding to the text embedding with the highest similarity to the image embedding as the predicted switch type.

[0095] 8. Perform the ViT-based switch closure recognition classifier operation: Use the image patch obtained in step 2 to perform the same operation as steps 3 and 4, but the parameters are different. Perform the operation in step 5, but the input sequence contains the category encoding information of the switch type predicted from step 7. The other parts remain unchanged, but the parameters are different. Finally, through a fully connected layer and a sigmoid operation, the probability of whether the switch is closed is obtained.

[0096] 9. Get the final result: whether the switch is closed.

[0097] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for constructing a switch closure recognition model based on a large visual model, characterized in that: The switch closure identification model construction method comprises: S1: Collect switch images and create a data set, the data set includes: an image file and a label file, the label file records the type and state of the corresponding switch image; S2: Build the ViT model, S3: Build the CLIP model, S4: Model training: Use the CLIP model to train a switch type classifier, and use the ViT model to train a switch closure recognition classifier; S5: Model fusion, fusing the switch type classifier based on the CLIP model and the switch closure recognition classifier based on the ViT model to obtain a switch closure recognition model.

2. The switch closure identification model construction method according to claim 1, characterized in that: The ViT model described in step S2 is built based on the Transformer encoder module. The Transformer encoder module consists of three encoder layers, each of which includes two parts: a self-attention mechanism and a feedforward neural network; In each encoder layer, self-attention calculation is first performed, and attention weight calculation is completed by querying query, key key, and value value, and the value is weighted and summed according to the corresponding weight. After the self-attention calculation, each output vector is processed by a feed-forward neural network consisting of two linear transformations and a ReLU activation function. Finally, the output of the encoder is obtained through residual connection and layer normalization operations.

3. The switch closure identification model construction method according to claim 2, characterized in that: The ViT model has an output layer, which uses a fully connected layer and a softmax operation to obtain the probability of each category.

4. The switch closure identification model construction method according to claim 2, characterized in that: The process of weighted summation of values ​​according to weights is: Among them, Q is the query matrix, which represents the current input, K is the key matrix, which represents the relevance of the input, V is the value matrix, which represents the output information corresponding to the key, and d k is the dimension of the key, used for scaling.

5. The switch closure identification model construction method according to claim 2, characterized in that: The ViT model is configured to divide the input image into small patches of fixed size, and then flatten and map each patch to an embedding space of fixed dimension; and introduce position encoding to represent the absolute position of each small patch in the image.

6. The switch closure identification model construction method according to claim 3, characterized in that: Step CLIP adopts a dual encoder architecture, including an image encoder and a text encoder. The image encoder uses the ViT model to extract image features. The image I is passed through the image encoder f I Generate image embedding vector v I ; The text encoder uses the Transformer encoder module to encode the text label into an embedding vector, and the text description T is passed through the text encoder f T Generate text embedding vector v t .

7. The switch closure identification model construction method according to claim 6, characterized in that: In step S4, the CLIP model is used to train the switch type classifier, and the contrast loss is used to measure the similarity between the image and the text. The training steps include: Forward propagation: In each training step, images are passed through the image encoder to get image embeddings, and texts are passed through the text encoder to get text embeddings. Compute Similarity: Compute the cosine similarity between the image and text embeddings: Calculate the loss: According to the contrast loss function, calculate the similarity between each image and its corresponding text, and minimize the similarity with other categories of text.

8. The switch closure identification model construction method according to claim 7, characterized in that: In step S4, the ViT model is used to train the switch closure recognition classifier, and the cross entropy loss function is used to perform the image classification task. Before the features are input into the transfomer compiler, the type encoding of the switch is added to guide the model training.

9. A switch closure recognition method based on a visual large model, characterized in that: A switch closure identification model constructed by the switch closure identification model construction method according to any one of claims 1 to 8, The switch closure identification method comprises: Take switch images and perform image preprocessing; Output the processed image to the switch type classifier based on the CLIP model to obtain the type of switch in the corresponding image; According to the obtained switch type, the type code is embedded in the switch closure recognition classifier based on the ViT model to identify whether the switch is closed; The output is whether the switch is closed.

Images