Industrial equipment indicating lamp state identification method based on image sequence and Transform network

Abstract

The invention provides an industrial equipment indicator lamp state identification method based on an image sequence and a Transform network, the method is provided with a target detection network and a state identification network, the target detection network combines all detected indicator lamp sub-images into an image sequence, and all indicator lamp states are simultaneously output at one time; the state recognition network is based on a Transform structure network, respectively calculates the relation between the internal characteristics of each indicator lamp sub-image and the relation between the characteristics of all indicator lamp images, and comprehensively recognizes the state of the indicator lamp; according to the method, images are input into a neural network model in a one-time mode in the form of an image sequence, the state of each indicator light is recognized by learning the image feature relation in a single indicator light and the image feature relation between the indicator lights, the indicator lights are concerned, and comparison between global features and the indicator lights is also considered; therefore, the method has a more accurate recognition effect under a complex illumination condition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, in particular to an industrial equipment indicator light state recognition method based on image sequences and a Transformer network. BACKGROUND

[0002] There are a large number of indicator lights in industrial production environments, which indicate the operating state of equipment. The operating state of equipment can be quickly understood through the state of the indicator light, and when a fault is found, it can be handled in time to ensure the normal operation of the equipment. Traditionally, indicator light recognition requires regular manual inspection to ensure that the state of the indicator light meets expectations. With the development of computer vision, more and more systems use cameras to collect images of indicator light equipment in real time and use algorithms to automatically monitor the state of the indicator light. The current mainstream technology uses a target detection neural network model to directly detect the coordinates of the indicator light on the image and directly obtain the state of each indicator light. The target detection task is generally considered to have "shift invariance", that is, the target should be equally detected regardless of its position in the image. Therefore, each target in the image is independent of each other. This is generally reasonable and the existing technology can achieve good results. However, in real-world scenarios, there are many complex interference factors. On the one hand, the lighting will change, and a bright indicator light in a strong light environment will look highly similar to a dim indicator light in normal lighting. A dim indicator light will also be partially illuminated by a bright indicator light next to it, making it indistinguishable from other bright indicator lights. These difficulties make it difficult for existing technology to further improve accuracy.

[0003] In the patent technology, the patent number CN202310914925.6 "An industrial equipment indicator light state recognition method" uses the yolov7 target detection network model to directly detect the coordinates of all indicator lights on the image and the classification (on / off two classification) of each indicator light; the patent number CN202510003611.X "Switch cabinet state recognition method based on small target perception" uses an improved yolov8 target detection network model to also directly detect the coordinates of all indicator lights on the image and the on / off classification of each indicator light.By introducing SPD non-stride convolution and multi-scale feature extraction module MSFE-AC into the yolov8 network, the detection effect of small indicator lights is improved; patent number CN202311051320.5 "Indicator light state recognition method and device, terminal equipment and storage medium", uses a two-stage method, the steps are: 1) train a target detection network to directly detect the indicator lights in the image, including the coordinate bounding box and the on-off state V1 of each indicator light; 2) for each indicator light, calculate the average brightness of the pixels in the bounding box, denoted as V, take a background area around the bounding box, which has the same size as the indicator light bounding box, calculate the average brightness of the pixels in the area, denoted as V', then the confidence of the on-off state is V2=V / (V+k*V'), k is a preset threshold value; 3) the final state of each indicator light is determined by the weighted average of V1 and V2; patents CN202310914925.6 and CN202510003611.X both directly use a target detection network to directly determine the coordinates and state of the indicator light, and the target detection network itself has the characteristic of translation invariance, treating each indicator light in the image as an independent object in the calculation, which is prone to misjudgment in the case of adjacent indicator lights being partially illuminated by the indicator light that is on; moreover, the accuracy will also decrease when the light changes, for example, in a strong light environment, the appearance of the indicator light that is on is similar to that of the indicator light that is off in general lighting, and the target detection network is difficult to learn correct features from such samples; on the other hand, the target detection network has difficulty in recognizing small size indicator lights, although various improvement schemes can improve the detection effect, but due to the small relative area of the indicator light itself, the corresponding image features are also too few, and the accuracy of state recognition is difficult to further improve; patent CN202311051320.5 uses a two-stage recognition method, first uses a target detection network to detect the indicator light, obtains the bounding box and the first state value of each indicator light, then calculates the average brightness value of the indicator light bounding box and the average brightness value of the nearby background area, and uses the ratio of the two brightness values as the second state value; obviously, the average brightness of the pixels is a too simple image feature, and it is almost impossible to learn various models of indicator lights in real scenes and complex backgrounds and lighting using this feature, such as a dark background with an off white indicator light, the average pixel brightness of the indicator light area will be close to the average brightness of the background, but in another scene, a bright blue indicator light on a white panel, the average brightness may also be close to the background, obviously it is difficult to correctly determine the on-off state of the indicator light with a preset threshold value; therefore, the applicant proposes an industrial equipment indicator light state recognition method based on image sequence and Transformer network according to the detection requirements of the indicator light, which has high adaptability and reliability to complex lighting conditions and various types of indicator lights. SUMMARY

[0004] To solve the above technical problems, the present application provides an industrial equipment indicator light state recognition method based on image sequence and Transformer network, which sets a target detection network and a state recognition network. The target detection network combines all detected indicator light sub-images into an image sequence and outputs all indicator light states at once. The state recognition network is based on a Transformer structure network, which calculates the relationship between the features inside each indicator light sub-image and the relationship between the features of all indicator light images, and comprehensively recognizes the state of the indicator light. The method inputs the image sequence into the neural network model at once, learns the image feature relationship inside a single indicator light and the image feature relationship between each indicator light, and recognizes the state of each indicator light.

[0005] To achieve the above purpose, the technical scheme adopted by the present application is:

[0006] The industrial equipment indicator light state recognition method based on image sequence and Transformer network is characterized in that it sets a target detection network and a state recognition network. The target detection network combines all detected indicator light sub-images into an image sequence and outputs all indicator light states at once. The state recognition network is based on a Transformer structure network, which calculates the relationship between the features inside each indicator light sub-image and the relationship between the features of all indicator light images, and comprehensively recognizes the state of the indicator light. The specific steps of the industrial equipment indicator light state recognition method based on image sequence and Transformer network are as follows:

[0007] Step one, use a camera to collect pictures of the target to be recognized. The pictures should contain at least one indicator light to be recognized.

[0008] Step two, preprocess the collected pictures, adjust the size, normalize the pixel value and normalize the channel order, and input them into the pre-trained target detection network to obtain the bounding box coordinates and confidence of all indicator lights to be recognized. The result is: [(x 11 , y 11 , x 12 , y 12 , s1), (x 21 , y 21 , x 22 , y 22 , s2),..., (x n1 , y n1 , x n2 , y n2 , s n)] represents the bounding box coordinates and confidence of the detected n indicator lights;

[0009] Step three, input the n detection results of the preprocessed picture into the detection network to adjust the sub-image of the original image corresponding to the bounding box to the same size, and also adjust the original image to the same size, and form an image sequence of n+1 images;

[0010] Step four, input the image sequence into the state recognition network to calculate the relationship between the features inside each indicator light sub-image and the relationship between the features of all indicator light images, and comprehensively recognize the state of the indicator light;

[0011] Step five, output the detection result of the indicator light.

[0012] Further, the industrial equipment indicator light state recognition method based on image sequence and Transformer network sets a state recognition network, which is based on a Transformer neural network model. The input of the model is an indefinite length image sequence, and the size of each image in the sequence is the same. Let the sequence length be s, the width and height of the image be w and h respectively, then the input is a (s, 3, h, w) matrix. The output of the network is a sequence of length s-1, and each element is a decimal between 0 and 1.

[0013] Further, the industrial equipment indicator light state recognition method based on image sequence and Transformer network inputs the image sequence into the state recognition network for processing, which is specifically:

[0014] 1) The state recognition network includes an image encoder. The image encoder is a neural network based on Transformer structure, which first divides each image in the input image sequence into fixed-size grids, each grid does not overlap, and all grids completely cover the entire image. Let the size of the grid be p*q;

[0015] 2) The image encoder further encodes each sub-image in each grid cell of the image into a length-d vector. For the input image sequence, n+1 matrices of size (p*q, d) are output;

[0016] 3) The image encoder further uses the attention module in the Transformer structure to calculate the relationship between the p×q d-dimensional vectors. The output is still n+1 matrices of size (p×q, d). Several attention modules are stacked in the image encoder, and this step is repeated several times;

[0017] 4) the result obtained by processing the image encoder, linearly transforming n+1 matrices of size (p x q, d) into n+1 vectors of dimension d', satisfying d≤ d' < p x q x d;

[0018] 5) the image encoder further utilizes the attention mechanism in the Transformer structure to calculate the relationship between the n+1 d' dimensional vectors; the output is still n+1 vectors of dimension d', and this step is repeated several times;

[0019] 6) the decoder of the neural network based on the Transformer structure contained in the state recognition network, the initial output sequence is a vector of length n+1, and the first element of the vector is a special number <bos>, the last n elements are the confidence of the on-off state of the n indicator lights in the input image sequence; the decoder uses the attention module in the Transformer structure to calculate the relationship between each element in the output sequence and the relationship between each element in the output sequence and the n+1 d' dimension vectors output by the image encoder, and outputs a vector with a length of n+1; a plurality of attention modules are stacked in the decoder, and this step is repeated several times;

[0020] 7) Take the output vector processed by the decoder, the first number is <bos>, the last n numbers are the state confidence of n indicator lights in the input image sequence respectively, the range of each number is (0, 1), which is the probability that the state of the indicator light is "on", and the confidence threshold is t, 0<t<1, all the confidence less than the threshold is "off", and the confidence greater than the threshold is "on", and the state of all indicator lights in the input image can be obtained.

[0021] Further, the industrial equipment indicator light state recognition method based on image sequence and Transformer network uses a network model based on Transformer structure, takes the image sequence composed of all indicator light sub-images on an image as input, and comprehensively uses image attention and inter-image attention to recognize the states of all indicator lights on the original image at one time.

[0022] The application brings the following benefits:

[0023] 1. The industrial equipment indicator light state recognition method based on image sequence and Transformer network has high adaptability and recognition reliability to complex lighting conditions and various types of indicator lights.

[0024] 2. The industrial equipment indicator light state recognition method based on image sequence and Transformer network can output all indicator light states at one time, rather than recognizing single indicator light states separately.

[0025] 3. The attention module of the industrial equipment indicator light state recognition method based on image sequence and Transformer network not only calculates the relationship between the features of each indicator light sub-image, but also calculates the relationship between the features of all indicator light images, and comprehensively recognizes the state of the indicator light.

[0026] 4. The industrial equipment indicator light state recognition method based on image sequence and Transformer network inputs all indicator lights in the image in the form of image sequence to the neural network model at one time, learns the image feature relationship of single indicator light and the image feature relationship between each indicator light, predicts the state of each indicator light, and is closer to human habits, not only pays attention to the indicator light itself, but also considers the global features and the comparison between the indicator lights, and has more accurate prediction effect in complex lighting conditions. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 It is an algorithm flowchart of the application.

[0028] Figure 2 It is a main structure diagram of the state recognition network of the application.

[0029] Figure 3 It is a practical image analysis diagram of the application. DETAILED DESCRIPTION

[0030] The application will be described in further detail below with reference to the drawings and specific embodiments:

[0031] As Figures 1-2 shown, it is an industrial equipment indicator light state recognition method based on image sequence and Transformer network, characterized in that: the industrial equipment indicator light state recognition method based on image sequence and Transformer network is provided with a target detection network and a state recognition network; the target detection network combines all the detected indicator light sub-images into an image sequence, and outputs all the indicator light states at one time; the state recognition network is based on a Transformer structure network, respectively calculates the relationship between the features inside each indicator light sub-image and the relationship between the features of all the indicator light images, and comprehensively recognizes the state of the indicator light; the specific steps of the industrial equipment indicator light state recognition method based on image sequence and Transformer network are:

[0032] Step one, use a camera to collect pictures of the target to be recognized, which should contain at least one indicator light to be recognized;

[0033] Step two, pre-process the collected pictures, adjust the size, normalize the pixel value and normalize the channel order, and input them into the pre-trained target detection network to obtain the bounding box coordinates and confidence of all the indicator lights to be recognized, and the result is: [(x 11 , y 11 , x 12 , y 12 , s1), (x 21 , y 21 , x 22 , y 22 , s2),..., (x n1 , y n1 , x n2 , y n2 , s n )], representing the bounding box coordinates and confidence of the n detected indicator lights;

[0034] Step three, input the n detection results of the pre-processed picture into the detection network to adjust the sub-image of the original image corresponding to the bounding box to the same size, and at the same time adjust the original image to the same size, and then form an image sequence with the n+1 images;

[0035] Step four, input the image sequence into the state recognition network, respectively calculate the relationship between the features inside each indicator light sub-image and the relationship between the features of all the indicator light images, and comprehensively recognize the state of the indicator light.

[0036] Step five, output the detection result of the indicator light.

[0037] The industrial equipment indicator light state recognition method based on the image sequence and the Transformer network sets a state recognition network, the state recognition network is based on a Transformer neural network model, the input of the model is an indefinite length image sequence, the size of each image in the sequence is the same, the sequence length is s, the width and height of the image are w and h respectively, and the input is a (s, 3, h, w) matrix; the output of the network is a sequence with a length of s-1, and each element is a decimal between 0 and 1.

[0038] The industrial equipment indicator light state recognition method based on the image sequence and the Transformer network uses a network model based on the Transformer structure, takes an image sequence composed of all indicator light sub-images on an image as input, comprehensively uses intra-image attention and inter-image attention, and identifies the states of all indicator lights on the original image at one time. Figure 3 As shown, the two images are indicator lights in different scenes, and the environment light of the left image is dark; the "alarm" indicator light in the left image is actually "off", but it is partially illuminated because it is close to the bright light; the "running" indicator light in the right image is bright, but the environment brightness is higher than that of the left image, and no glare is generated as in the left image; it can be found that the pixel features of the "alarm" light actually "off" in the left image and the light actually "on" in the right image are very close, and it is difficult to distinguish them if the existing method is used to judge the state of the indicator light alone; the method takes the eight indicator light sub-images in the left image as an image sequence, learns not only the features of a single indicator light but also the differences between the eight indicator lights, can easily judge that the brightness of the "alarm" light and the "running" light is significantly different, and thus the state of the "alarm" light is "off", and the recognition accuracy is higher and more reliable.

[0039] The above description is only a preferred embodiment of the present application, and does not limit the present application in any other form, and any modification or equivalent change made according to the technical essence of the present application still falls within the scope of the present application.< / bos> < / bos>

Claims

1. A method for industrial equipment indicator light status recognition based on image sequences and Transformer networks, characterized by: The specific steps are: Step 1: Use a camera to capture an image of the target to be identified. The image should contain at least one indicator light to be identified. Step 2: Preprocess the collected images, adjust the size, normalize the pixel values ​​and normalize the channel order, and input them into the pre-trained target detection network to obtain the coordinates and confidence of all indicator light frames to be identified. The result is: [(x 11 , y 11 , x 12 , y 12 , s1), (x 21 , y 21 , x 22 , y 22 , s2),..., (x n1 , y n1 , x n2 , y n2 , s n )], representing the bounding box coordinates and confidence of the n detected indicator lights; Step 3: Input the n detection results after image preprocessing into the detection network, adjust the sub-images of the original image corresponding to the bounding box to the same size, and adjust the original image to the same size at the same time, and form these n+1 images into an image sequence; Step 4: Input the image sequence into the state recognition network, calculate the relationship between the features within each indicator light sub-image and the relationship between the features of all indicator light images, and comprehensively identify the state of the indicator light; Step 5: Output the detection result of the indicator light.

2. The method for industrial equipment indicator light status recognition based on image sequences and Transformer networks according to claim 1 is characterized by: The industrial equipment indicator light status recognition method based on image sequence and Transformer network sets up a status recognition network. The status recognition network is based on the Transformer neural network model. The input of the model is an image sequence of variable length. Each image in the sequence has the same size. The sequence length is s, and the width and height of the image are w and h respectively. Then the input is a matrix of (s, 3, h, w); the output of the network is a sequence of length s-1, and each element is a decimal between 0 and 1.

3. The method for industrial equipment indicator light status recognition based on image sequences and Transformer networks according to claim 1 is characterized by: The industrial equipment indicator light state recognition method based on image sequence and Transformer network inputs the image sequence into the state recognition network and the processing process is specifically as follows: 1) The state recognition network includes an image encoder. The image encoder is based on a Transformer-structured neural network. For each image in the input image sequence, it first divides the image into a grid of fixed size, with each unit non-overlapping and all grids completely covering the entire image. The grid size is denoted as p*q. 2) The image encoder further encodes each sub-image within each grid cell of the image into a vector of length d; for the input image sequence, it outputs n+1 matrices of size (p*q,d); 3) The image encoder further uses the attention module in the Transformer structure to calculate the relationship between p×q d-dimensional vectors for each output matrix. The output is still n+1 matrices of size (p×q, d). Several attention modules are stacked in the image encoder, and this step is repeated several times. 4) The result obtained by image encoder processing is to transform the n+1 matrix lines of size (p×q,d) into n+1 vectors of dimension d', satisfying d≤ d' <p×q×d; 5) The image encoder further uses the attention mechanism in the Transformer structure to calculate the relationship between the n+1 d'-dimensional vectors; the output is still a vector of n+1 dimension d', and this step is repeated several times; 6) The state recognition network contains a decoder based on the Transformer structure neural network. The initial output sequence is a vector of length n+1, and the first element of the vector is a special number. <bos> , the following n elements are the confidence levels of the on / off states of the n indicator lights in the input image sequence; the decoder uses the attention module in the Transformer structure to calculate the relationship between each element in the output sequence, as well as the relationship between each element in the output sequence and the n+1 d'-dimensional vectors output by the image encoder, and outputs a vector of length n+1;< / bos> A number of attention modules are stacked in the decoder, and this step is repeated several times; 7) Take the output vector after the decoder processing, the first number is <bos>, the following n numbers are the state confidence levels of the n indicator lights in the input image sequence respectively. The range of each number is (0, 1), which is the probability that the state of the indicator light is "on". The confidence threshold is set as t, where 0 < t < 1. All confidences less than the threshold are "off", and those greater than the threshold are "on", so as to obtain the states of all indicator lights in the input picture.< / bos> 4. The method for industrial equipment indicator light status recognition based on image sequences and Transformer networks according to claim 1 is characterized by: The method for identifying the states of industrial equipment indicator lights based on an image sequence and a Transformer network uses a network model based on the Transformer structure. All the indicator light sub-images on an image are used to form an image sequence as the input, and both intra-image attention and inter-image attention are comprehensively used to identify the states of all indicator lights on the original image at one time.

Images