Universal detection method for pointer type square meters and LWYS model improved by application of universal detection method

Abstract

The invention relates to the technical field of image detection methods, and particularly discloses a universal detection method for a pointer type square meter and an improved LWYS model applied to the universal detection method, and the method comprises the steps: building an image data set, and obtaining a dial scale division value database of the square meter; improving to obtain an LWYS model; in the detection use work, a to-be-detected square table image is collected, a to-be-detected image is obtained, an input image data set is processed and trained in the LWYS model, the LWYS model is called to segment the to-be-detected image to obtain a square table mask, and then the mask is mapped to the to-be-detected image to obtain a square table mapping image only containing a square table; correcting the square table mapping image; based on a fine-grained classification model, comparing and detecting the square table front view with the image of the square table dial plate division value database to distinguish the type of the square table; and performing square meter pointer reading identification on the square meter front view. Through the above method and model detection, the universality is higher, the method is suitable for pointer reading detection of different types of square meters, and the reading is more accurate.

Description

Technical Field

[0001] The invention relates to the technical field of image detection methods. Background Art

[0002] Pointer-type square meters can display various key parameters (such as pressure, temperature, flow, etc.) in real time. Operators can quickly understand the current status through the position of the pointer on the meter, which helps to detect abnormal situations in time and take necessary measures to ensure the safety and stability of the production process. With the rapid development of industrial production automation and intelligence, inspection robots used in industrial production processes have emerged. Inspection robots can replace manual inspections of various instruments and other operating indicators of electrical and mechanical equipment to determine the operating status of the equipment. The reading detection of pointer-type square meters is one of the many detection items. Most of the existing related research is based on computer vision and digital image processing technology to complete the reading detection of pointer-type square meters.

[0003] For example, a method for identifying the readings of a square pointer instrument disclosed in Chinese patent application No. 202111094853.2 first captures the original image and extracts the rectangular frame of the instrument for correction. The corrected image is processed into a binary refined image, and then the center of the circle around which the pointer revolves, the center of mass of the upper left corner A, and the midpoint of the center line segment of the pointer are determined. The baseline and the needle line are defined by connecting these points, and finally the angle between the baseline and the needle line is calculated, and a linear equation is fitted to reflect the reading value. In this method, the target object extracted from the image using the binarization method is greatly affected by the illumination and background complexity, and is only applicable to scenes with a single background and a stable lighting environment. Therefore, this method cannot be deployed on inspection robots used in complex environments.

[0004] Another example is a pointer instrument detection method for a chemical inspection robot disclosed in Chinese patent application No. 202410233860.3. First, the pointer instrument image is obtained in the inspection image using the improved YOLOv7 network. The pointer instrument image is sequentially obtained through the CNN network and the G-RMI algorithm to obtain the key point set of the instrument's large scale. The key point set is obtained through the RANSAC algorithm to obtain the optimal affine transformation parameter matrix. Then, the coordinates of each pixel point on the uncorrected instrument image are multiplied by the optimal affine transformation parameter matrix to obtain the front view image. Finally, the pointer instrument reading recognition network based on deep regression obtains the reading value of the pointer instrument from the front view image. This method has strong anti-environmental interference ability, but its detection accuracy depends on the detection results of the key points of the instrument's large scale. If the coordinates of the key points deviate, the pointer readings finally identified will be inaccurate. When applied to the detection of the scale of the square table dial, it is inevitable that there will be some errors, which will lead to large errors in the detected readings and weak applicability.

[0005] In summary, the existing detection methods only consider one or two types, and do not fully consider the actual application needs of industrial inspection robots. They have weak versatility and are not applicable to the detection of pointer-type square meters with a large variety of ranges and graduation values, and the detection results are unreliable. In addition, the segmentation model parameters and calculation amount used in the existing methods are large, which cannot meet the performance requirements of rapid detection of inspection robots. Summary of the invention

[0006] An object of the present invention is to provide a universal detection method for pointer-type square meters which has higher versatility and is applicable to the detection of pointer readings of different types of square meters and has more accurate readings. Another object of the present invention is to provide an improved LWYS model for applying the above-mentioned universal detection method for pointer-type square meters.

[0007] To achieve the above object, the technical solution of the present invention is: a universal detection method for pointer-type square meters, characterized in that the method steps are as follows:

[0008] S1. Establish an image dataset for deep learning; establish a database of square watch dial graduation values; improve the YOLOv5s-Seg model to obtain the LWYS model;

[0009] S2. During the detection work, the square table image to be detected is collected to obtain the image to be detected, the input image data set and the training in the LWYS model are processed, and the LWYS model is called to segment the image to be detected to obtain a square table mask, and then the mask is mapped to the image to be detected to obtain a square table mapping image containing only the square table;

[0010] S3, correcting the square table mapping image;

[0011] S4, based on the fine-grained classification model, the square watch front view is compared with the image of the square watch dial graduation value database to detect and distinguish the square watch category;

[0012] S5. Identify the pointer reading of the square meter on the front view of the square meter.

[0013] In step S3, the square table mapping image is corrected, and the correction process is as follows: first, Blob analysis is performed according to the characteristics of the square table mapping image, and the square table outer frame, internal pointer and dividing line areas are extracted by threshold segmentation, and then the internal pointer and dividing line areas are removed by area screening to obtain the square table outer frame area; secondly, the edge of the square table outer frame area is detected by the Canny algorithm, and then the four edge straight lines of the outer frame are extracted by Hough transform; finally, the intersection of the four straight lines is used as the input of perspective transformation, and the square table front view is obtained by perspective transformation.

[0014] In the four edge lines of the outer frame extracted by Hough transform, the Hough line detection algorithm is improved to extract the line, if the line width is greater than 1, then the outermost edge of the line is taken.

[0015] In step S4, based on the fine-grained classification model, the square watch front view is compared with the image of the square watch dial graduation value database to detect and distinguish the square watch category. This step performs classification training through the fine-grained classification model. First, the square watch front view is classified and labeled and the data is enhanced using the Cutmix method; secondly, the data set is placed in the fine-grained classification model for training, and the classification accuracy of the model is improved by adjusting parameters during the training process; finally, the trained model is used for the classification of the square watch front view.

[0016] In step S5, the square meter pointer reading is recognized on the front view of the square meter. First, the LWYS model is called to segment the front view of the square meter to obtain the pointer mask. Secondly, the pointer mask is detected based on the Hough transform to obtain the rotation angle of the pointer. Finally, the dial division value is found according to the square meter category, and the square meter pointer reading is calculated according to the proportional relationship between the dial division value and the rotation angle.

[0017] A universal detection method for pointer-type square tables uses an improved LWYS model, characterized in that the improvements include the following: 1) Based on the ShuffleUnit1 module in the ShuffleNet V2 network, an E-SN module is designed to replace the C3 network layer of the backbone feature extraction network in the YOLOv5-Seg model by fusing the SE channel attention mechanism. In the network structure of the E-SN module, K1 and K2 are defined as the channel numbers of the input and output feature maps respectively, and K1 is equal to K2. The input feature map is first subjected to a Channel Split operation to separate the channel dimensions to obtain two groups of feature maps with the number of channels being K1 / 2. One group is first subjected to a deep convolution and a point-by-point convolution operation, and then a new feature map is obtained by SE channel attention weight optimization. The new feature map is concat-operated with the other group of feature maps, and then channel shuffled to obtain a feature map containing rich information; 2) GhostConv is used to replace the ordinary convolution layer; 3) SIOU is used as the loss function of boundary regression.

[0018] By adopting the above technical scheme, the beneficial effects of the present invention are as follows: the present invention firstly segments the square table area in the image through the improved LWYS model; then uses the Blob analysis method to obtain the outer frame area of ​​the square table, and then performs contour detection, Hough line detection, line intersection detection and other steps to obtain the four corner points of the outer frame of the square table. After that, the square table image is corrected based on the four corner points; finally, the pointer in the front view of the square table is segmented and the rotation angle of the pointer is detected, and the proportional relationship between its division value and the rotation angle is obtained according to the type of square table detected by the fine-grained classification model, so as to calculate the pointer reading. The present invention obtains the front view of the square table through a lightweight improved instance segmentation model and a traditional digital image processing flow, which is not only high in precision and fast in processing speed, but also lays the foundation for the subsequent pointer reading calculation. In addition, the present invention also introduces a square table category database and a fine-grained classification model, and the type of the detected square table can be obtained through model classification, and then the proportional relationship between the division value and the rotation angle is obtained in the database, and the pointer reading is calculated. The versatility of the method of the present invention is improved, which brings convenience to the practical application of the inspection robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The present invention relates to a flowchart of a universal detection method for a pointer-type square meter

[0020] Figure 2 It is a network structure diagram of the E-SN module involved in the present invention.

[0021] Figure 3 It is a network structure diagram of the LWYS model involved in the present invention.

[0022] Figure 4a and Figure 4b The present invention relates to a square meter before and after correction. DETAILED DESCRIPTION

[0023] In order to further explain the technical solution of the present invention, the present invention is described in detail below through specific embodiments.

[0024] The embodiment of the present invention discloses a universal detection method for a pointer-type square meter, and the method flow is as follows: Figure 1 As shown, the method steps are as follows:

[0025] S1. Establish an image dataset for deep learning. Specifically, capture images containing a pointer-type square meter in an industrial on-site factory environment. The number of images should be large. For example, in this embodiment, the experiment collects no less than 5,000 images. Each image is processed. Then, the data is manually calibrated using annotation software. The square meter and the pointer are marked in each image. The data is randomly divided into a training set, a validation set, and a test set according to the ratio (8:1.5:0.5).

[0026] A database of square meter dial graduation values ​​is established; a variety of existing square meter images are collected, and the square meter categories and corresponding dial graduation values ​​in the images are processed and marked. Through this database, the applicability of the method of the present invention to the ability to recognize pointer readings of various square meters can be improved.

[0027] The YOLOv5s-Seg model is improved to obtain the LWYS model (Light Weight YOLOv5s-Seg, LWYS); the improvements include the following: 1. Based on the ShuffleUnit1 module in the ShuffleNet V2 network, the E-SN module is designed to replace the C3 network layer (C3 module) of the backbone feature extraction network in the YOLOv5-Seg model. The network structure of the E-SN module is as follows: Figure 2 As shown in the figure, K1 and K2 are the number of channels of the input and output feature maps respectively, and K1 is equal to K2. The input feature map is first subjected to a Channel Split operation to separate it in the channel dimension to obtain two groups of feature maps with the number of channels being K1 / 2. One group is first subjected to a deep convolution and a point-by-point convolution operation, and then a new feature map is obtained through SE channel attention weight optimization. The new feature map is concat-operated with the other group of feature maps, and then channel shuffled to obtain a feature map containing rich information. This improvement can not only reduce the number of model parameters, but also improve the ability to extract information from the image; 2. Using GhostConv instead of ordinary convolution layers can further reduce the number of model parameters; 3. Using SIOU as the loss function for boundary regression, this improvement increases the regression constraints, which can improve the segmentation accuracy of the model, and improve the accuracy and convergence speed of model training. The network structure of the improved LWYS model is shown below: Figure 3 As shown, through the above improvements, the segmentation model can be lightweight according to the segmentation characteristics of the square table and the pointer, which can effectively solve the problems encountered in the prior art that industrial inspections have certain requirements on detection speed, the image processing platform used due to cost considerations has low computing power, and the commonly used instance segmentation model has large parameters and slow reasoning speed.

[0028] S2. During the detection work, the square table image to be detected is collected to obtain the image to be detected, the input image data set and the LWYS model are processed for training, and the LWYS model is called to segment the image to be detected to obtain a square table mask, and then the mask is mapped to the image to be detected to obtain a square table mapping image containing only the square table. The principle expression of this process is Iresult(x,y)=I1(x,y)&I2(x,y), where I1 and I2 are input images and Iresult is the output result.

[0029] S3. Correct the square table mapping image. The correction process is as follows: first, perform Blob analysis based on the features of the square table mapping image, and use threshold segmentation to extract the square table outer frame, internal pointers, and dividing lines. Then, use area screening to remove the internal pointers and dividing lines to obtain the square table outer frame area. Secondly, use the Canny algorithm to detect the edge of the square table outer frame area, and then use Hough transform to extract the four edge straight lines of the outer frame. To ensure that the outer straight lines of the outer frame area are accurately extracted, the present invention improves the Hough straight line detection algorithm. When extracting a straight line, if the straight line width is greater than 1, take the outermost edge of the straight line. Finally, use the intersection of the four straight lines as the input of the perspective transformation, and use the perspective transformation method to obtain the front view of the square table. Comparison before and after correction: Figure 4a and Figure 4b shown.

[0030] S4. Based on the fine-grained classification model, the square watch front view is compared with the image of the square watch dial graduation value database to detect and distinguish the square watch category; after the square watch category is detected by the fine-grained classification model, the dial graduation value of the corresponding category can be automatically obtained from the square watch dial graduation value database for pointer reading calculation. Here, the fine-grained classification model (Object-Part Attention Model) is trained to classify the square watch. First, the square watch front view is labeled and data enhancement is performed using the Cutmix method; secondly, the data set is put into the fine-grained classification model for training, and the classification accuracy of the model is improved by adjusting parameters during the training process; finally, the trained model is used for the classification of the square watch front view. In the application, different dial graduation values ​​can be assigned to the pointer reading calculation formula according to the classification results.

[0031] S5. Identify the pointer reading of the square meter on the front view of the square meter. First, call the LWYS model to segment the front view of the square meter to obtain the pointer mask. Second, perform line detection on the pointer mask based on the Hough (Hough line detection) transform to obtain the rotation angle of the pointer. Finally, find the dial division value according to the square meter category, and calculate the pointer reading of the square meter according to the proportional relationship between the dial division value and the rotation angle.

[0032] In summary, the above method of the present invention, first, segments the square table mask in the image through the improved YOLOv5-Seg model (Light WeightYOLOv5s-Seg, LWYS), maps the square table mask on the original image to obtain an image of only the square table, and then corrects the square table image to a front view; secondly, since different types of square tables have different graduation values, the square table front view is classified by a fine-grained image classification model, and different graduation values ​​are assigned according to the category of the square table, so that the pointer angle can be directly converted into a pointer reading after it is obtained; finally, the LWYS model is used to segment the pointer in the square table front view, and then the pointer is subjected to Hough straight line detection to obtain the deflection angle of the pointer, so that the pointer reading is calculated according to the proportional relationship between the graduation value and the deflection angle.

[0033] Compared with the prior art, the above detection method includes the following innovations: 1. The common instance segmentation model has large parameters and calculation amount, which is difficult to meet the performance requirements of rapid detection during the movement of the inspection robot. Therefore, the present invention makes a lightweight improvement to the YOLOv5s-Seg model, which is less affected by the accuracy of the model reasoning result during segmentation and has high robustness. While reducing its parameters and calculation amount, it ensures the segmentation accuracy of the model and improves the model reasoning speed, which can meet the inspection requirements in any environment; 2. In order to improve the quality of square table image correction, the present invention proposes an improved Hough line detection method, adds the judgment conditions when extracting Hough lines, and extracts the periphery of the line when the line width is greater than 1, avoiding interference from other lines; 3. For the first time, it is proposed to create a square table dial division value database and apply a fine-grained classification network to the detection of pointer-type square tables. In the detection process, the types of square tables are distinguished and assigned corresponding division values, which are used for the calculation of the pointer readings of the square tables. The calculated pointer readings are more accurate and suitable for the detection of pointer readings of different types of square tables. It has higher versatility, so that the method of the present invention becomes a universal pointer-type square table detection method.

[0034] The above embodiments and drawings do not limit the product form and style of the present invention. Any appropriate changes or modifications made by ordinary technicians in the relevant technical field should be deemed to be within the patent scope of the present invention.

Claims

1. A universal detection method for pointer-type square meters, characterized in that: The steps are as follows: S1. Establish an image dataset for deep learning; establish a database of square watch dial graduation values; improve the YOLOv5s-Seg model to obtain the LWYS model; S2. During the detection work, the square table image to be detected is collected to obtain the image to be detected, the input image data set and the training in the LWYS model are processed, and the LWYS model is called to segment the image to be detected to obtain a square table mask, and then the mask is mapped to the image to be detected to obtain a square table mapping image containing only the square table; S3, correcting the square table mapping image; S4, based on the fine-grained classification model, the square watch front view is compared with the image of the square watch dial graduation value database to detect and distinguish the square watch category; S5. Identify the pointer reading of the square meter on the front view of the square meter.

2. A universal detection method for a pointer-type square meter as claimed in claim 1, characterized in that: In step S1, the YOLOv5s-Seg model is improved to obtain the LWYS model. The improvements include the following: 1) Based on the ShuffleUnit1 module in the ShuffleNet V2 network and the SE channel attention mechanism, the E-SN module is designed to replace the C3 network layer of the backbone feature extraction network in the YOLOv5-Seg model. In the network structure of the E-SN module, K1 and K2 are defined as the number of channels of the input and output feature maps, respectively, and K1 is equal to K2. The input feature map is first separated by the ChannelSplit operation in the channel dimension to obtain two groups of feature maps with the number of channels of K1 / 2. One group is first subjected to deep convolution and point-by-point convolution operations, and then a new feature map is obtained by SE channel attention weight optimization. The new feature map is concat-operated with the other group of feature maps, and then channel shuffle is performed to obtain a feature map containing rich information; 2) Use GhostConv to replace the ordinary convolution layer; 3) Use SIOU as the loss function for boundary regression.

3. A universal detection method for a pointer-type square meter as claimed in claim 2, characterized in that: In step S3, the square table mapping image is corrected, and the correction process is as follows: first, Blob analysis is performed according to the characteristics of the square table mapping image, and the square table outer frame, internal pointer and dividing line areas are extracted by threshold segmentation, and then the internal pointer and dividing line areas are removed by area screening to obtain the square table outer frame area; secondly, the edge of the square table outer frame area is detected by the Canny algorithm, and then the four edge straight lines of the outer frame are extracted by Hough transform; finally, the intersection of the four straight lines is used as the input of perspective transformation, and the square table front view is obtained by perspective transformation.

4. A universal detection method for a pointer-type square meter as claimed in claim 3, characterized in that: In the four edge lines of the outer frame extracted by Hough transform, the Hough line detection algorithm is improved to extract the line, if the line width is greater than 1, then the outermost edge of the line is taken.

5. A universal detection method for a pointer-type square meter as claimed in any one of claims 2 to 4, characterized in that: Step S4 compares the square watch front view with the image of the square watch dial graduation value database based on the fine-grained classification model to detect and distinguish the square watch category. This step uses the fine-grained classification model for classification training. First, the square watch front view is classified and labeled and data enhancement is performed using the Cutmix method; secondly, the data set is put into the fine-grained classification model for training, and the classification accuracy of the model is improved by adjusting parameters during the training process; finally, the trained model is used for the classification of the square watch front view; And / or, in step S5, in identifying the square meter pointer reading of the square meter front view, first, the LWYS model is called to segment the square meter front view to obtain the pointer mask; secondly, the pointer mask is subjected to straight line detection based on the Hough transform to obtain the rotation angle of the pointer, and finally, the dial division value is found according to the square meter category, and the square meter pointer reading is calculated according to the proportional relationship between the dial division value and the rotation angle.

6. A universal detection method for pointer-type square meters using an improved LWYS model, characterized in that: Improvements include: 1) Based on the ShuffleUnit1 module in the ShuffleNet V2 network and the SE channel attention mechanism, the E-SN module is designed to replace the C3 network layer of the backbone feature extraction network in the YOLOv5-Seg model. In the network structure of the E-SN module, K1 and K2 are defined as the number of channels of the input and output feature maps, respectively, and K1 is equal to K2. The input feature map is first separated by the ChannelSplit operation in the channel dimension to obtain two groups of feature maps with the number of channels of K1 / 2. One group is first subjected to deep convolution and point-by-point convolution operations, and then a new feature map is obtained by SE channel attention weight optimization. The new feature map is concat-operated with the other group of feature maps, and then channel shuffle is performed to obtain a feature map containing rich information; 2) Use GhostConv to replace the ordinary convolution layer; 3) Use SIOU as the loss function for boundary regression.

Images