Industrial equipment indicator light state recognition method based on image sequence and transformer network

Abstract

The application provides an industrial equipment indicator light state recognition method based on an image sequence and a Transformer network, a target detection network and a state recognition network are arranged, the target detection network combines all detected indicator light sub-images into an image sequence, and simultaneously outputs all indicator light states at one time; the state recognition network is based on a network with a Transformer structure, respectively calculates the relationship between features inside each indicator light sub-image and the relationship between features of all indicator light images, and comprehensively recognizes the state of the indicator light; the method is input into a neural network model in the form of an image sequence at one time, the image feature relationship inside a single indicator light and the image feature relationship between each indicator light are learned, the state of each indicator light is recognized, the global feature and the comparison between the indicator lights are considered in addition to the indicator light itself, and therefore the method has more accurate recognition effect under complex illumination conditions.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, specifically to a method for recognizing the status of indicator lights on industrial equipment based on image sequences and Transformer networks. Background Technology

[0002] Industrial production environments contain numerous indicator lights that indicate the operating status of equipment. Understanding the status of these lights allows for quick assessment of equipment operation, enabling timely troubleshooting and ensuring normal operation. Traditionally, indicator light identification required regular manual inspections to ensure the lights met expectations. With the development of computer vision, more and more systems are using cameras to capture real-time images of indicator lights and algorithms to automatically monitor their status. Currently, mainstream technologies utilize neural network models based on object detection, directly detecting the coordinates of the indicator lights in the image and obtaining the status of each light. Object detection is generally considered to possess "translation invariance," meaning that any object appearing at any position in the image should be equally detectable; therefore, each object in the image is independent. This is generally reasonable, and existing technologies achieve good results. However, real-world scenarios are subject to complex interference factors. For example, lighting conditions vary; a lit indicator light in a brightly lit environment looks highly similar to an off indicator light in normal lighting. An off indicator light can also be partially illuminated by a nearby lit indicator light, making it appear indistinguishable from other lit lights. These difficulties make it challenging to further improve the accuracy of existing technologies.

[0003] In the patented technologies, patent number CN202310914925.6, "A Method for Recognizing the Status of Indicator Lights in Industrial Equipment," uses a YOLOv7 target detection network model to directly detect the coordinates of all indicator lights on the image, as well as the classification of each indicator light (on / off binary classification); patent number CN202510003611.X, "A Method for Recognizing the Status of Switchgear Based on Small Target Perception," uses an improved YOLOv8 target detection network model to similarly directly detect the coordinates of all indicator lights on the image, as well as the on / off classification of each indicator light.By introducing SPD non-staggered convolution and the multi-scale feature extraction module MSFE-AC into the YOLOv8 network, this method improves the detection performance of small indicator lights. Patent No. CN202311051320.5, "A Method, Device, Terminal Equipment and Storage Medium for Recognizing the State of an Indicator Light," uses a two-stage method. The steps are: 1) Training a target detection network to directly detect indicator lights in the image, including the bounding box of each indicator light and its on / off state V1; 2) For each indicator light, calculating the average brightness of the pixels within the bounding box, denoted as V, and taking a background region around the bounding box with the same size as the indicator light's bounding box, calculating the average brightness of the pixels within the region. Let V' be the indicator light's on / off state. Then, the confidence level of its on / off state is V2 = V / (V + k * V'), where k is a preset threshold value. 3) The state of each indicator light is ultimately determined by the weighted average of V1 and V2. Patents CN202310914925.6 and CN202510003611.X directly use the target detection network to determine the coordinates and state of the indicator lights. The target detection network itself has the characteristic of translation invariance. When calculating, each indicator light in the image is treated as an independent object. It is easy to misjudge when the lit indicator light in the adjacent indicator lights illuminates the part of the off indicator light. Moreover, the accuracy will also decrease when the lighting changes. For example, the indicator light lit in a strong light environment is different from the normal indicator light. Indicator lights that are off during illumination have similar appearances, making it difficult for object detection networks to learn the correct features from such samples. Furthermore, object detection networks struggle to recognize small-sized indicator lights. While various improvement schemes can enhance detection performance, the relatively small area of ​​the indicator lights results in too few corresponding image features, making it difficult to further improve the accuracy of state recognition. Patent CN202311051320.5 uses a two-stage recognition method: first, an object detection network detects the indicator lights, obtaining the bounding box and first state value for each indicator light; then, it calculates the average brightness value within the indicator light's bounding box and the average brightness value of the nearby background area, using the ratio of these two brightness values ​​as the second state value. Clearly, this method... Average pixel brightness is an overly simplistic image feature. Using this feature, it is almost impossible to learn the various types of indicator lights in real-world scenes, as well as complex backgrounds and lighting conditions. For example, in a dark background, the average pixel brightness of a white indicator light that is off will be close to the average brightness of the background. However, in another scene, the average brightness of a blue indicator light that is on on a white panel may also be close to the background. Obviously, it is difficult to correctly determine the on / off state of the indicator light using a preset threshold value. To address this, the applicant proposes an industrial equipment indicator light status recognition method based on image sequences and Transformer networks, which has high adaptability and reliability under complex lighting conditions and for various types of indicator lights. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a method for recognizing the status of industrial equipment indicator lights based on image sequences and Transformer networks. By setting up a target detection network and a status recognition network, the target detection network combines all detected indicator light sub-images into an image sequence, simultaneously outputting the status of all indicator lights at once. The status recognition network, based on a Transformer structure, calculates the relationships between features within each indicator light sub-image and the relationships between features across all indicator light images, comprehensively recognizing the status of the indicator lights. The method inputs the image sequence into the neural network model all at once, learning the image feature relationships within a single indicator light and the image feature relationships between each indicator light to identify the status of each indicator light.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A method for recognizing the status of indicator lights on industrial equipment based on image sequences and Transformer networks is characterized by the following: the method includes a target detection network and a status recognition network; the target detection network combines all detected indicator light sub-images into an image sequence and outputs the status of all indicator lights simultaneously; the status recognition network, based on a Transformer structure, calculates the relationships between features within each indicator light sub-image and the relationships between features across all indicator light images, comprehensively recognizing the status of the indicator lights; the specific steps of the method are as follows:

[0007] Step 1: Use a camera to capture an image of the target to be identified. The image should contain at least one indicator light to be identified.

[0008] Step 2: Preprocess the acquired images by adjusting their size, normalizing pixel values, and normalizing channel order. Then, input the pre-trained object detection network to obtain the bounding box coordinates and confidence scores for all indicator lights to be identified. The result is: [(x 11 , y 11 , x 12 , y 12 , s1), (x 21 , y 21 , x 22 , y 22 , s2),..., (x n1 , y n1 , x n2 , y n2 , s n] represents the bounding box coordinates and confidence level of the n detected indicator lights;

[0009] Step 3: Input the n detection results after image preprocessing into the detection network to adjust the sub-images of the original image corresponding to the bounding box to the same size, and at the same time adjust the original image to the same size, and combine these n+1 images into an image sequence;

[0010] Step 4: Input the image sequence into the state recognition network, calculate the relationship between features within each indicator light sub-image and the relationship between features across all indicator light images, and comprehensively recognize the state of the indicator lights;

[0011] Step 5: Detection results of the output indicator light.

[0012] Furthermore, the industrial equipment indicator light status recognition method based on image sequences and Transformer networks sets up a status recognition network. The status recognition network is based on the Transformer neural network model. The input of the model is an image sequence of variable length, in which each image has the same size. Let the sequence length be s, and the width and height of the images be w and h, respectively. Then the input is a matrix of (s, 3, h, w). The output of the network is a sequence of length s-1, where each element is a decimal between 0 and 1.

[0013] Furthermore, the industrial equipment indicator light status recognition method based on image sequences and Transformer networks specifically involves inputting the image sequence into the status recognition network for processing as follows:

[0014] 1) The state recognition network contains an image encoder; the image encoder is based on a neural network with a Transformer structure. For each image in the input image sequence, it first divides the image into a grid of fixed size. Each unit does not overlap with the others, and all grids completely cover the entire image. The size of the grid is denoted as p*q.

[0015] 2) The image encoder further encodes each sub-image within each grid cell of the image into a vector of length d; for the input image sequence, it outputs n+1 matrices of size (p*q,d);

[0016] 3) The image encoder further uses the attention module in the Transformer structure to calculate the relationship between p×q d-dimensional vectors for each output matrix; the output is still n+1 matrices of size (p×q,d); the image encoder stacks several attention modules and repeats this step several times.

[0017] 4) The result obtained by the image encoder transforms n+1 matrices of size (p×q,d) into n+1 vectors of dimension d', satisfying d ≤ d'. <p×q×d;

[0018] 5) The image encoder further utilizes the attention mechanism in the Transformer structure to calculate the relationship between n+1 d'-dimensional vectors; the output is still n+1 vectors with dimension d', and this step is repeated several times;

[0019] 6) The state recognition network includes a decoder based on a Transformer neural network. The initial output sequence is a vector of length n+1, where the first element is a special number. <bos>The following n elements are the confidence scores of the on / off states of the n indicator lights in the input image sequence; the decoder uses the attention module in the Transformer structure to calculate the relationship between each element in the output sequence, as well as the relationship between each element in the output sequence and the n+1 d'-dimensional vectors output by the image encoder, and outputs a vector of length n+1; the decoder stacks several attention modules and repeats this step several times;

[0020] 7) Take the output vector after the decoder processing; the first number is... <bos>, the subsequent n digits respectively correspond to the state confidence of n indicator lights in the input image sequence, each digit ranges from (0, 1), representing the probability that the indicator light is in the "on" state. A confidence threshold t is set, where 0<t<1. All indicator lights with confidence lower than the threshold are determined as "off", and those with confidence higher than the threshold are determined as "on", thus the states of all indicator lights in the input image can be obtained.

[0021] Further, in the industrial equipment indicator light state recognition method based on image sequence and Transformer network, a network model based on Transformer structure is used, which takes the image sequence formed by all indicator light sub-images on one image as input, comprehensively uses intra-image attention and inter-image attention, and recognizes the states of all indicator lights on the original image at one time.

[0022] Benefits of the present application are:

[0023] 1. The industrial equipment indicator light state recognition method based on image sequence and Transformer network has high adaptability and recognition reliability for complex lighting conditions and various indicator lights of different models;

[0024] 2. The industrial equipment indicator light state recognition method based on image sequence and Transformer network can output the states of all indicator lights at the same time in one go, instead of separately recognizing the state of a single indicator light;

[0025] 3. In the attention module of the industrial equipment indicator light state recognition method based on image sequence and Transformer network, besides separately calculating the relationship between features inside each indicator light sub-image, it also calculates the relationship between features of all indicator light images, so as to comprehensively recognize the states of indicator lights;

[0026] 4. The industrial equipment indicator light state recognition method based on image sequence and Transformer network inputs all indicator lights in an image in the form of an image sequence into the neural network model at one time, predicts the state of each indicator light by learning the intra-indicator image feature relationship and the inter-indicator image feature relationship, which is closer to human habits. It not only focuses on the indicator light itself, but also considers global features and the comparison between indicator lights, so it has a more accurate prediction effect under complex lighting conditions. Description of Drawings

[0027] Figure 1 is a schematic flow diagram of the algorithm of the present invention;

[0028] Figure 2 is a schematic diagram of the main structure of the state recognition network of the present invention;

[0029] Figure 3 is a schematic diagram of actual image analysis of the present invention. Detailed Implementation

[0030] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0031] like Figure 1-2 The diagram illustrates a method for recognizing the status of indicator lights on industrial equipment based on image sequences and Transformer networks. The method is characterized by the inclusion of a target detection network and a status recognition network. The target detection network combines all detected indicator light sub-images into an image sequence, simultaneously outputting the status of all indicator lights. The status recognition network, based on a Transformer structure, calculates the relationships between features within each indicator light sub-image and the relationships between features across all indicator light images, comprehensively recognizing the indicator light status. The specific steps of the method are as follows:

[0032] Step 1: Use a camera to capture an image of the target to be identified. The image should contain at least one indicator light to be identified.

[0033] Step 2: Preprocess the acquired images by adjusting their size, normalizing pixel values, and normalizing channel order. Then, input the pre-trained object detection network to obtain the bounding box coordinates and confidence scores for all indicator lights to be identified. The result is: [(x 11 , y 11 , x 12 , y 12 , s1), (x 21 , y 21 , x 22 , y 22 , s2),..., (x n1 , y n1 , x n2 , y n2 , s n ] represents the bounding box coordinates and confidence level of the n detected indicator lights;

[0034] Step 3: Input the n detection results after image preprocessing into the detection network to adjust the sub-images of the original image corresponding to the bounding box to the same size, and at the same time adjust the original image to the same size, and combine these n+1 images into an image sequence;

[0035] Step 4: Input the image sequence into the state recognition network, calculate the relationship between features within each indicator light sub-image and the relationship between features across all indicator light images, and comprehensively recognize the state of the indicator lights;

[0036] Step 5: Detection results of the output indicator light.

[0037] The method for recognizing the status of industrial equipment indicator lights based on image sequences and Transformer networks is shown. The status recognition network is set up based on the Transformer neural network model. The input of the model is an image sequence of variable length, in which each image has the same size. Let the sequence length be s, and the width and height of the images be w and h, respectively. Then the input is a matrix (s, 3, h, w). The output of the network is a sequence of length s-1, in which each element is a decimal between 0 and 1.

[0038] The industrial equipment indicator light status recognition method based on image sequences and Transformer networks, as shown, uses a network model based on a Transformer structure. It takes an image sequence composed of all indicator light sub-images from a single image as input, and comprehensively utilizes intra-image attention and inter-image attention to recognize the status of all indicator lights in the original image at once. Figure 3 As shown, the two images represent indicator lights in different scenes. The ambient light in the left image is relatively dim. The "alarm" indicator light in the left image is actually off, but it is partially illuminated because it is close to a lit light. The "running" indicator light in the right image is lit, but because the ambient brightness is higher than in the left image, there is no glare like in the left image. It can be seen that the pixel features of the "alarm" light, which is actually off in the left image, and the light, which is actually lit in the right image, are very similar. If existing methods are used to judge the status of the indicator lights individually, it is difficult to distinguish them. This method uses the eight indicator light sub-images in the left image as an image sequence, learning not only the features of individual indicator lights but also the differences between the eight indicator lights. It can easily determine that the brightness of the "alarm" light and the "running" light are significantly different, thus determining that its status is off. Its recognition accuracy is higher and more reliable.

[0039] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any modifications or equivalent changes made based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.< / bos> < / bos>

Claims

1. A method for recognizing the status of indicator lights in industrial equipment based on image sequences and Transformer networks, characterized in that: The industrial equipment indicator light status recognition method based on image sequences and Transformer networks includes a target detection network and a status recognition network. The target detection network combines all detected indicator light sub-images into an image sequence and outputs the status of all indicator lights simultaneously. The status recognition network, based on a Transformer structure, calculates the relationships between features within each indicator light sub-image and the relationships between features across all indicator light images, comprehensively recognizing the indicator light status. The specific steps of the industrial equipment indicator light status recognition method based on image sequences and Transformer networks are as follows: Step 1: Use a camera to capture an image of the target to be identified. The image should contain at least two indicator lights to be identified. Step 2: Preprocess the collected images and input them into a pre-trained object detection network to obtain the bounding box coordinates and confidence scores of all indicator lights to be identified; Step 3: Input the n detection results after image preprocessing into the detection network to adjust the sub-images of the original image corresponding to the bounding box to the same size, and at the same time adjust the original image to the same size, and combine these n+1 images into an image sequence; Step 4: Input the image sequence into the state recognition network, calculate the relationship between features within each indicator light sub-image and the relationship between features across all indicator light images, and comprehensively recognize the state of the indicator lights; The specific processing procedure is as follows: 1) The state recognition network contains an image encoder based on a neural network with a Transformer structure, which divides each image in the input image sequence into a grid of size p×q; 2) The image encoder further encodes each sub-image within each grid cell of the image into a vector of length d; for the input image sequence, it outputs n+1 matrices of size (p×q,d); 3) The image encoder further uses the attention module in the Transformer structure to calculate the relationship between p×q d-dimensional vectors for each output matrix; the output is still n+1 matrices of size (p×q,d); the image encoder stacks at least two attention modules and repeats this step several times. 4) The result obtained by the image encoder transforms n+1 matrices of size (p×q,d) into n+1 vectors of dimension d', satisfying d ≤ d'. <p×q×d; 5) The image encoder further utilizes the attention mechanism in the Transformer structure to calculate the relationship between n+1 d'-dimensional vectors; the output is still n+1 vectors with dimension d', and this step is repeated several times; 6) The state recognition network includes a decoder based on a neural network with a Transformer structure. After processing by the decoder, the output vector after decoding is taken to obtain the state of all indicator lights in the input image. Step 5: Output indicator light test results.

2. The method for identifying the status of indicator lights in industrial equipment based on image sequences and Transformer networks according to claim 1, characterized in that: The industrial equipment indicator status recognition method based on image sequences and Transformer networks sets up a status recognition network. The status recognition network is based on the Transformer neural network model. The input of the model is an image sequence of variable length, in which each image has the same size. Let the sequence length be s, and the width and height of the images be w and h, respectively. Then the input is a matrix of (s, 3, h, w). The output of the network is a sequence of length s-1, where each element is a decimal between 0 and 1.

3. The method for identifying the status of indicator lights in industrial equipment based on image sequences and Transformer networks according to claim 1, characterized in that: The industrial equipment indicator light status recognition method based on image sequence and Transformer network uses a network model based on Transformer structure. It takes all the indicator light sub-images on an image as an image sequence as input, and comprehensively uses intra-image attention and inter-image attention to recognize the status of all indicator lights on the original image at once. The state recognition network includes a decoder based on a Transformer neural network. The initial output sequence is a vector of length n+1, where the first element of the vector is a special number. <bos> The following n elements are the confidence scores of the on / off states of the n indicator lights in the input image sequence; the decoder uses the attention module in the Transformer structure to calculate the relationship between each element in the output sequence, as well as the relationship between each element in the output sequence and the n+1 d'-dimensional vectors output by the image encoder, and outputs a vector of length n+1.< / bos> The decoder contains at least two attention modules, and this step is repeated several times.

Images