An industrial pointer instrument reading recognition system based on deep learning

Abstract

The present invention belongs to the field of computer vision image processing technology, specifically, an industrial pointer meter reading recognition system based on deep learning, including: a data set construction and division module, used to construct and divide the industrial pointer meter image data set; an industrial pointer meter reading extraction network module, used to extract the semantic segmentation image of the pointer and scale of the industrial pointer meter; an industrial pointer meter reading recognition module, used to read the semantic segmentation image of the pointer and scale using the Hough circle transform and angle method. The present invention is based on the multi-residual small target feature extraction secondary submodule in the industrial instrument finger extraction network based on deep learning, which can enable the segmentation extraction model to obtain richer feature information in the shallow network, and improve the effect of extracting small target edge information when extracting small targets such as pointers and scales.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision image processing, and in particular to an industrial pointer instrument reading recognition system based on deep learning. Background Art

[0002] Pointer instruments mainly consist of two parts: the instrument and the pointer. They have been widely used in many fields such as power systems and manufacturing due to their simple structure, low design cost, strong anti-interference ability and high reliability. However, the traditional method of collecting readings from pointer instruments usually relies on manual inspection, which is not only inefficient and inaccurate, but also requires a lot of manpower. Especially in extreme environments with complex factors such as high temperature, high risk and strong radiation, the task of reading pointer instruments is more difficult. There are the following specific problems:

[0003] 1. The size of the pointer and scale in the industrial pointer instrument will change due to different shooting distances. The existing methods are not effective when dealing with small targets such as pointers and scales, resulting in unsatisfactory results when extracting the pointer and scale in the pointer instrument.

[0004] 2. In extreme environments, such as radiation, high temperature, and high pressure, industrial pointer instruments will become blurred, resulting in the existing methods being unable to accurately capture the multi-scale information of pointer instruments.

[0005] 3. Traditional deep learning networks find it difficult to effectively extract the detailed feature information of pointers and scales, which may result in omissions or errors when extracting pointers and scales.

[0006] Therefore, in order to solve the above problems, an industrial pointer meter reading recognition system based on deep learning is invented. The machine vision technology of deep learning is used to replace manual reading of pointer meters, which can realize the automatic collection of meter readings and greatly improve efficiency and accuracy. Therefore, an industrial pointer meter reading recognition system based on deep learning is invented. Summary of the invention

[0007] In view of the above-mentioned and / or existing problems in an existing industrial pointer instrument reading recognition system based on deep learning, the present invention is proposed.

[0008] Therefore, the purpose of the present invention is to provide an industrial pointer instrument reading recognition system based on deep learning, which can solve the above-mentioned existing problems.

[0009] To solve the above technical problems, according to one aspect of the present invention, the present invention provides the following technical solutions:

[0010] An industrial pointer instrument reading recognition system based on deep learning, comprising:

[0011] The dataset construction and division module is used to construct and divide the industrial pointer instrument image dataset;

[0012] Industrial pointer instrument reading extraction network module, used to extract the semantic segmentation image of the pointer and scale of the industrial pointer instrument;

[0013] Industrial pointer instrument reading recognition module, used to read the semantic segmentation image of pointer and scale using Hough circle transform and angle method;

[0014] The data set construction and division module is connected to the industrial pointer instrument reading extraction network module, and the industrial pointer instrument reading extraction network module is connected to the industrial pointer instrument reading identification module.

[0015] As a preferred solution of the industrial pointer instrument reading recognition system based on deep learning described in the present invention, the specific process of the data set construction and division module is as follows:

[0016] Process 1: Construct an industrial pointer instrument image dataset: First, use a camera to collect industrial pointer instrument image data to form an industrial pointer instrument classification dataset; then adjust each image in the industrial pointer instrument classification dataset to a uniform size, and annotate the pointers and scales in each image of the industrial pointer instrument classification dataset to generate an industrial pointer instrument segmentation dataset; then convert the industrial pointer instrument segmentation dataset into a semantic segmentation dataset, and convert the RGB color label of each image in the semantic segmentation dataset into a corresponding single-channel digital label; finally, combine the industrial pointer instrument single-channel digital label with the industrial pointer instrument classification dataset to form an industrial pointer instrument image dataset;

[0017] Process 2: Divide the industrial pointer instrument image dataset: divide the industrial pointer instrument image dataset constructed by S1 into a training set, a validation set, and a test set according to a certain ratio.

[0018] As a preferred solution of the industrial pointer instrument reading recognition system based on deep learning described in the present invention, the industrial pointer instrument reading extraction network module is composed of a reading extraction network first-level sub-module, a model training and verification first-level sub-module and a pointer and scale extraction first-level sub-module.

[0019] As a preferred solution of the industrial pointer instrument reading recognition system based on deep learning described in the present invention, the first-level submodule of the reading extraction network is composed of a second-level submodule for extracting multiple residual small target features, a second-level submodule for extracting multiple types of void convolution features, and a second-level submodule for extracting pointer instrument detail feature information.

[0020] As a preferred solution of the industrial pointer meter reading recognition system based on deep learning described in the present invention, the specific process of the industrial pointer meter reading extraction network module is as follows:

[0021] S1: Use the first-level submodule of the reading extraction network to extract features from the industrial pointer instrument image dataset constructed by the dataset construction and partitioning module, and generate semantic segmentation images of the pointer and scale;

[0022] S2: Use the model training and verification first-level submodule to train and verify the industrial instrument fingerprint extraction network;

[0023] S3: In the pointer and scale extraction first-level sub-module, the industrial instrument finger mark extraction network is applied to extract the semantic segmentation image of the pointer and scale.

[0024] As a preferred solution of the industrial pointer meter reading recognition system based on deep learning described in the present invention, the specific steps of S1 are as follows:

[0025] S11: Use the multi-residual small target feature extraction secondary submodule to extract the shallow and deep features of the original pointer and scale;

[0026] S12: Use multiple types of dilated convolution feature extraction submodules to extract multi-scale deep industrial pointer instrument image features;

[0027] S13: Use the pointer instrument detail feature information extraction secondary submodule to extract the detail feature information of industrial pointer instruments.

[0028] As a preferred solution of the industrial pointer meter reading recognition system based on deep learning described in the present invention, the specific steps of S11 are as follows:

[0029] S111: The size of the original pointer instrument image is 512×512 pixels, and the number of channels is 3. After the original pointer instrument image is input into the multi-residual small target feature extraction secondary submodule, a convolution operation is first performed by a convolution kernel with a size of 1×1 and a number of channels of 32 to obtain a shallow pointer instrument feature map F1. The size of F1 is 256×256 pixels, and the number of channels is 32. After that, F1 is input into the feature fusion convolution block;

[0030] S112: input the shallow pointer instrument feature map F1 into the feature fusion convolution block, firstly perform convolution operation by a convolution kernel of size 3×3 and padding 1, then process it through the ReLU activation function, then perform convolution operation by a convolution kernel of size 3×3 and padding 1, then concatenate it with F1 in the channel dimension, and finally use a convolution kernel of size 1×1 and padding 0 to adjust the number of channels, so as to obtain the shallow pointer instrument feature map F2;

[0031] S113: Perform spatial feature compression on the shallow pointer instrument feature map F2, that is, use average pooling and maximum pooling in parallel to process F2 in the spatial dimension to obtain a one-dimensional vector of length C, and then perform one-dimensional convolution C1D on the one-dimensional vector. t After processing, the weight w of each channel is obtained through the Sigmoid activation function. Finally, the weight w is multiplied by F2 to obtain the deep pointer instrument feature map F3. The process is shown in the following formula (1):

[0032] F3=w⊙F2=σ(C1D t (F 3×3 ([AvgPool(F2);MaxPool(F2)]))⊙F2 (1)

[0033] Among them, AvgPool represents average pooling, MaxPool represents maximum pooling, and F 3×3 () indicates a convolution operation with a convolution kernel size of 3×3, C1D t represents a one-dimensional convolution with a kernel size of t, σ represents the Sigmoid activation function, ⊙ represents the element-wise product, and C1D t The role of is to generate channel weights, and t is adaptively determined according to the mapping of the number of channels x of the output feature:

[0034]

[0035] Where []odd represents the nearest odd number of the parameter, ω, β and γ are hyperparameters;

[0036] Finally, the deep pointer instrument feature map F3 is input into the multi-class hole convolution feature extraction secondary submodule. The size of F3 is 128×128 pixels and the number of channels is 64.

[0037] As a preferred solution of the industrial pointer instrument reading recognition system based on deep learning described in the present invention, the specific steps of S12 are as follows:

[0038] S121: The deep pointer instrument feature map F3 is input into the multi-type dilated convolution feature extraction secondary submodule. F3 is first processed in parallel by four dilated convolutions, and the dilated rates of the four dilated convolutions are 1, 3, 5, and 7, respectively. Then, F3 is subjected to dilated convolution operations to obtain multi-scale depth feature maps R1, R2, R3, and R4, respectively. Finally, R1, R2, R3, R4, and F3 are concatenated in the channel dimension, where the sizes of R1, R2, R3, and R4 are all 128×128 pixels, and the number of channels is 64.

[0039] S122: After concatenating R1, R2, R3, R4 and F3 in the channel dimension, the result is a multi-scale deep feature map F4, the size of F4 is 128×128 pixels, the number of channels is 32, and after F4 undergoes 1×1 convolution, the number of channels of F4 is compressed to 64. Finally, after 3×3 convolution, the result of the secondary sub-module of multi-type hole convolution feature extraction is obtained, that is, the multi-scale deep pointer instrument feature map F5, the size of F5 is 256×256 pixels, the number of channels is 64, and F5 is input into the secondary sub-module of pointer instrument detail feature information extraction.

[0040] As a preferred solution of the industrial pointer meter reading recognition system based on deep learning described in the present invention, the specific steps of S13 are as follows:

[0041] S131: input F5 and F2 together to the pointer instrument detail feature information extraction secondary submodule, in which there are two processing routes, L1 and L2: in L1, F5 is processed by a 1×1 ordinary convolution and a Sigmoid activation function to obtain a detail feature information extraction feature map F6, the size of F6 is 256×256 pixels, and the number of channels is 64; in L2, F5 is processed by a 3×3 size, a hole rate of 2 hole convolution and a Sigmoid activation function to obtain a detail feature information extraction feature map F7, the size of F7 is 256×256 pixels, and the number of channels is 64;

[0042] S132: Connect the shallow pointer instrument feature map F2, the detail feature information extraction feature map F6, the detail feature information extraction feature map F7 and the multi-scale deep pointer instrument feature map F5 and splice them in the channel dimension to obtain the detail feature information extraction feature map F8, the size of F8 is 256×256 pixels, the number of channels is 256, and then F8 is input into the dual-line parallel attention three-level sub-module TPAM;

[0043] S133: In the two-line parallel attention three-level sub-module TPAM, the detail feature information extraction feature map F8 is bilinearly reshaped, the size is adjusted to 512×512 pixels, and the number of channels is adjusted to 256, so as to obtain the detail feature information extraction feature map F9;

[0044] S134: After bilinear reshaping, average pooling and maximum pooling are performed in parallel, followed by double 1×1 convolution operations, addition, and finally Sigmoid activation function operations to obtain detail feature information extraction feature map F10. The size of F10 is 512×512 pixels and the number of channels is 256. The formula for obtaining F10 from F9 is as follows:

[0045] F10=σ(B 1 (Avg(F9))+B 2 (Max(F9))) (3)

[0046] Among them, Avg() represents average pooling, Max() represents maximum pooling, and B 1 () and B 2 () represents the convolution combination operation of two convolution kernels of size 1×1, σ represents the Sigmoid activation function;

[0047] S135: performing a convolution operation on the detail feature information extraction feature map F10, wherein the convolution kernel size of the convolution operation is 1×1, and obtaining a semantic segmentation image of the pointer and the scale, wherein the semantic segmentation image size of the pointer and the scale is 512×512 pixels, and the number of channels is 1.

[0048] As a preferred solution of the industrial pointer instrument reading recognition system based on deep learning described in the present invention, the specific process of the industrial pointer instrument reading recognition module is as follows:

[0049] Step 1: Use Hough circle transform to calculate the center and radius of the circle in the semantic segmentation image of the pointer and scale;

[0050] Step 2: Get the deflection angle of the pointer;

[0051] Step 3: Calculate the relationship between the deflection angle of the pointer and the angle of the minimum scale line;

[0052] Step 4: Combine the relationship between the range and the angle to calculate the specific reading of the industrial pointer instrument.

[0053] Compared with existing technologies:

[0054] 1. The multi-residual small target feature extraction secondary sub-module in the industrial instrument finger print extraction network based on deep learning in the present invention can enable the segmentation and extraction model to obtain richer feature information in the shallow network, and improve the effect of extracting small target edge information when extracting small targets such as pointers and scales.

[0055] 2. The present invention is based on a multi-type hole convolution feature extraction secondary sub-module in the industrial instrument finger mark extraction network, which can splice four multi-scale deep feature maps with hole rates of 1, 3, 5 and 7 respectively in the channel dimension, thereby improving the network's ability to obtain multi-scale information, and can simultaneously obtain the global information of large-scale pointer instruments and the local detail information of small-scale pointers and scales.

[0056] 3. The present invention is based on the pointer instrument detail feature information extraction secondary submodule in the industrial instrument finger mark extraction network, which can effectively extract the detail feature information of the pointer and scale, and reduce the occurrence of missed detection and wrong detection when extracting the pointer and scale in the pointer instrument. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a flow chart of the overall technical solution of the system of the present invention;

[0058] Figure 2 It is a flow chart of the network module for extracting readings of industrial pointer type instruments of the present invention;

[0059] Figure 3 Extracting a network structure diagram for the industrial instrument finger of the present invention;

[0060] Figure 4 It is an execution flow chart of the industrial instrument finger extraction network of the present invention;

[0061] Figure 5 It is a secondary submodule diagram for extracting multi-residual small target features of the present invention;

[0062] Figure 6 This is the structure diagram of the feature fusion convolution block of the present invention;

[0063] Figure 7 It is a secondary submodule diagram for extracting multiple types of hole convolution features of the present invention;

[0064] Figure 8 It is a secondary submodule diagram for extracting detailed feature information of a pointer instrument according to the present invention;

[0065] Fig. 9 It is a diagram of the dual-line parallel attention three-level submodule TPAM of the present invention;

[0066] Fig.10 This is the angle identification diagram of the semantic segmentation image of the pointer and scale of the present invention. DETAILED DESCRIPTION

[0067] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0068] The present invention provides an industrial pointer instrument reading recognition system based on deep learning, please refer to Figure 1-Figure 10 , including a data set construction and division module, which is used to construct and divide the industrial pointer instrument image data set; an industrial pointer instrument reading extraction network module, which is used to extract the semantic segmentation image of the pointer and scale of the industrial pointer instrument; an industrial pointer instrument reading recognition module, which is used to read the semantic segmentation image of the pointer and scale using the Hough circle transform and angle method;

[0069] The data set construction and division module is connected with the industrial pointer instrument reading extraction network module, and the industrial pointer instrument reading extraction network module is connected with the industrial pointer instrument reading recognition module.

[0070] The specific process of data set construction and division module is as follows:

[0071] Process 1: Construct an industrial pointer instrument image dataset: First, use a camera to collect industrial pointer instrument image data to form an industrial pointer instrument classification dataset; then adjust each image in the industrial pointer instrument classification dataset to a uniform size, and annotate the pointers and scales in each image of the industrial pointer instrument classification dataset to generate an industrial pointer instrument segmentation dataset; then convert the industrial pointer instrument segmentation dataset into a semantic segmentation dataset, and convert the RGB color label of each image in the semantic segmentation dataset into a corresponding single-channel digital label; finally, combine the industrial pointer instrument single-channel digital label with the industrial pointer instrument classification dataset to form an industrial pointer instrument image dataset;

[0072] Process 2: Divide the industrial pointer instrument image dataset: divide the industrial pointer instrument image dataset constructed by S1 into a training set, a validation set, and a test set according to a certain ratio;

[0073] The data set construction and division module includes but is not limited to the following embodiments:

[0074] Use a camera to collect 2000 images of industrial pointer instruments to form an industrial pointer instrument classification dataset. Use Python scripts to resize the 2000 images in the industrial pointer instrument classification dataset to 512×512 pixels. Use Labelme to manually annotate each image in the industrial pointer instrument classification dataset, and use the "labelme2voc" script to convert the industrial pointer instrument classification dataset into a semantic segmentation dataset, and convert the 2000 RGB color labels in the semantic segmentation dataset into corresponding single-channel digital labels.

[0075] The final industrial pointer instrument image dataset includes 2,000 single-channel digital labels of pointer instruments and 2,000 industrial pointer instrument classification datasets taken from cameras;

[0076] The dataset is divided into training set, validation set and test set according to a certain ratio; the training set accounts for 70% of the total dataset, that is, 1,400 pointer instrument images; the validation set and test set each account for 15% of the total dataset, that is, 300 pointer instrument images respectively;

[0077] In the actual experiment of the present invention, the data set division ratio (training set: validation set: test set) is 70%: 15%: 15%;

[0078] Among them, Labelme annotation tool is used as the annotation tool;

[0079] A Python script is a set of instructions or codes written in the Python programming language, usually ending in a .py file; it is an automated program used to perform a specific task or series of operations;

[0080] labelme2voc is a script that converts Labelme annotation format to VOC (Pascal VOC) format. It is usually used to convert Labelme annotated image semantic segmentation data to Pascal VOC dataset format for training models in deep learning frameworks (such as TensorFlow, PyTorch);

[0081] The semantic segmentation label of the present invention is a binary classification semantic segmentation label, which is usually a single channel; the label value of each pixel can be one of two categories. For example, in image segmentation tasks, 0 and 1 are usually used to represent two categories (such as foreground and background); therefore, for binary classification semantic segmentation tasks, the label image only needs one channel to represent the category of the pixel.

[0082] The industrial pointer instrument reading extraction network module is composed of a first-level submodule of the reading extraction network, a first-level submodule of the model training and verification, and a first-level submodule of the pointer and scale extraction; the first-level submodule of the reading extraction network is composed of a second-level submodule of multi-residual small target feature extraction (used to extract the detailed feature information of industrial pointer instruments), a second-level submodule of multi-type hole convolution feature extraction (used to extract the shallow and deep features of the original pointer and scale), and a second-level submodule of the pointer instrument detailed feature information extraction (used to extract multi-scale deep industrial pointer instrument image features); the structure of the industrial instrument finger extraction network is as follows: Figure 3 As shown; In addition, the function of the industrial instrument fingerprint extraction network is to correctly extract the pointer and scale in the industrial instrument when there is complex background interference in the industrial pointer instrument picture, that is, to obtain the semantic segmentation image of the pointer and scale, so as to facilitate the subsequent reading recognition of the industrial instrument; Among them, the execution flow chart of the industrial instrument fingerprint extraction network is shown in Figure 4 As shown;

[0083] The specific process of the industrial pointer instrument reading extraction network module is as follows:

[0084] S1: Use the first-level submodule of the reading extraction network to extract features from the industrial pointer instrument image dataset constructed by the dataset construction and partitioning module, and generate semantic segmentation images of the pointer and scale;

[0085] The specific steps of S1 are as follows:

[0086] S11: Use the multi-residual small target feature extraction secondary submodule to extract the shallow features and deep features of the original pointer and scale; among them, the structure of the multi-residual small target feature extraction secondary submodule is as follows Figure 5 As shown;

[0087] The specific steps of S11 are as follows:

[0088] S111: The size of the original pointer instrument image is 512×512 pixels, and the number of channels is 3. After the original pointer instrument image is input into the multi-residual small target feature extraction secondary submodule, a convolution operation is first performed by a convolution kernel of size 1×1 and number of channels 32 to obtain a shallow pointer instrument feature map F1. The size of F1 is 256×256 pixels, and the number of channels is 32. After that, F1 is input into the feature fusion convolution block; the structure of the feature fusion convolution block is as follows Figure 6 As shown;

[0089] S112: The shallow pointer instrument feature map F1 is input into the feature fusion convolution block, and firstly, a convolution operation is performed by a convolution kernel of size 3×3 and filled with 1, then it is processed by the ReLU activation function, and then a convolution operation is performed by a convolution kernel of size 3×3 and filled with 1, and then it is spliced ​​with F1 in the channel dimension (i.e. Figure 6 The concat operation in is performed, and finally the number of channels is adjusted using a convolution kernel of size 1×1 and filled with 0 to obtain the shallow pointer instrument feature map F2 (size 256×256 pixels, number of channels 32);

[0090] S113: Perform spatial feature compression on the shallow pointer instrument feature map F2, that is, use average pooling and maximum pooling in parallel to process F2 in the spatial dimension to obtain a one-dimensional vector of length C, and then perform one-dimensional convolution C1D on the one-dimensional vector. t After processing, the weight w of each channel is obtained through the Sigmoid activation function. Finally, the weight w is multiplied by F2 to obtain the deep pointer instrument feature map F3. The process is shown in the following formula (1):

[0091] F3=w⊙F2=σ(C1D t (F 3×3 ([AvgPool(F2);MaxPool(F2)]))⊙F2 (1)

[0092] Among them, AvgPool represents average pooling, MaxPool represents maximum pooling, and F 3×3 () indicates a convolution operation with a convolution kernel size of 3×3, C1D t represents a one-dimensional convolution with a kernel size of t, σ represents the Sigmoid activation function, ⊙ represents the element-wise product, and C1D t The role of is to generate channel weights, and t is adaptively determined according to the mapping of the number of channels c of the output feature:

[0093]

[0094] Where []odd represents the nearest odd number of the parameter, ω, β and γ are hyperparameters;

[0095] Finally, the deep pointer instrument feature map F3 is input into the multi-class dilated convolution feature extraction secondary submodule. The size of F3 is 128×128 pixels and the number of channels is 64.

[0096] S12: Use multiple types of dilated convolution feature extraction secondary submodules to extract multi-scale deep industrial pointer instrument image features; among them, multiple types of dilated convolution feature extraction secondary submodules such as Figure 7 As shown;

[0097] The specific steps of S12 are as follows:

[0098] S121: The deep pointer instrument feature map F3 is input into the multi-type dilated convolution feature extraction secondary submodule. F3 is first processed in parallel by four dilated convolutions, and the dilated rates of the four dilated convolutions are 1, 3, 5, and 7, respectively. Then, F3 is subjected to dilated convolution operations to obtain multi-scale depth feature maps R1, R2, R3, and R4, respectively. Finally, R1, R2, R3, R4, and F3 are concatenated in the channel dimension (i.e., Figure 7 Concat operation in ), where the sizes of R1, R2, R3, and R4 are all 128×128 pixels and the number of channels is 64;

[0099] S122: After concatenating R1, R2, R3, R4 and F3 in the channel dimension, the result is a multi-scale deep feature map F4, the size of which is 128×128 pixels and the number of channels is 32. After F4 is subjected to 1×1 convolution, the number of channels of F4 is compressed to 64. Finally, after 3×3 convolution, the result of the secondary submodule of multi-type hole convolution feature extraction is obtained, that is, the multi-scale deep pointer instrument feature map F5, the size of which is 256×256 pixels and the number of channels is 64, and F5 is input into the secondary submodule of pointer instrument detail feature information extraction. Among them, the ordinary convolution operation is to calculate the pixel value of the corresponding position of the output feature map by multiplying the convolution kernel with each pixel point of the input feature map and summing them. The hole convolution inserts additional intervals (or "holes") in the convolution kernel. These intervals can increase the effective size of the convolution kernel without increasing the number of parameters. The size of this interval is called the "hole rate" (Dilation rate). Rate), usually expressed as an integer, which determines the size of the interval in the convolution kernel;

[0100] S13: extracting the detailed feature information of industrial pointer instruments using the secondary submodule of extracting detailed feature information of pointer instruments; wherein the secondary submodule of extracting detailed feature information of pointer instruments is as follows: Figure 8 As shown;

[0101] The specific steps of S13 are as follows:

[0102] S131: input F5 and F2 together to the pointer instrument detail feature information extraction secondary submodule, in which there are two processing routes, L1 and L2: in L1, F5 is processed by a 1×1 ordinary convolution and a Sigmoid activation function to obtain a detail feature information extraction feature map F6, the size of F6 is 256×256 pixels, and the number of channels is 64; in L2, F5 is processed by a 3×3 size, a hole rate of 2 hole convolution and a Sigmoid activation function to obtain a detail feature information extraction feature map F7, the size of F7 is 256×256 pixels, and the number of channels is 64;

[0103] S132: Connect the shallow pointer instrument feature map F2, the detail feature information extraction feature map F6, the detail feature information extraction feature map F7 and the multi-scale deep pointer instrument feature map F5 and splice them in the channel dimension (i.e. Figure 8 Concat operation in the middle), and obtain the detail feature information extraction feature map F8, the size of F8 is 256×256 pixels, the number of channels is 256, and then F8 is input into the two-line parallel attention three-level submodule TPAM (two-line parallel attention module, Two-lineParallel Attention Module); the structure of the two-line parallel attention three-level submodule TPAM is as follows Fig. 9 As shown;

[0104] S133: In the two-line parallel attention three-level sub-module TPAM, the detail feature information extraction feature map F8 is bilinearly reshaped, the size is adjusted to 512×512 pixels, and the number of channels is adjusted to 256, so as to obtain the detail feature information extraction feature map F9;

[0105] S134: After bilinear reshaping, average pooling and maximum pooling are performed in parallel, followed by double 1×1 convolution operations, addition, and finally Sigmoid activation function operations to obtain detail feature information extraction feature map F10. The size of F10 is 512×512 pixels and the number of channels is 256. The formula for obtaining F10 from F9 is as follows:

[0106] F10=σ(B 1 (Avg(F9))+B 2 (Max(F9))) (3)

[0107] Among them, Avg() represents average pooling, Max() represents maximum pooling, and B 1 () and B 2 () represents the convolution combination operation of two convolution kernels of size 1×1, and B represents the Sigmoid activation function; wherein, bilinear reshaping is an existing technology, and its bilinear reshaping is to achieve image enlargement or reduction without distorting the image resolution through bilinear reshaping; this technology is widely used in digital image processing, computer vision and graphics, such as image scaling function in image editing software and texture mapping in game development;

[0108] S135: performing a convolution operation on the detail feature information extraction feature map F10, wherein the convolution kernel size of the convolution operation is 1×1, and obtaining a semantic segmentation image of the pointer and the scale, wherein the semantic segmentation image size of the pointer and the scale is 512×512 pixels, and the number of channels is 1;

[0109] Among them, S1 includes but is not limited to the following embodiments:

[0110] Input size is 512×512 pixels, the number of channels is 3 RGB original industrial pointer instrument feature map: the original pointer instrument image is input to the multi-residual small target feature extraction secondary submodule to perform shallow and deep feature extraction of the original pointer and scale. First, after the convolution operation with a convolution kernel size of 1×1, the shallow pointer instrument feature map F1 is obtained. The size of F1 is 256×256 pixels and the number of channels is 32. Then F1 is input into FFCB to obtain the shallow pointer instrument feature map F2. The size of F2 is 256×256 pixels and the number of channels is 32. The spatial feature of F2 is compressed and the one-dimensional vector is passed through the one-dimensional convolution C1D tDuring processing, the three hyperparameters used by t are adaptively determined: ω, β and γ are set to 2, 1 and 1 respectively, and the deep pointer instrument feature map F3 is obtained. The size of F3 is 128×128 pixels and the number of channels is 64; F3 is input to the multi-type hole convolution feature extraction secondary submodule to extract the multi-scale deep industrial pointer instrument image features. The deep pointer instrument feature map F3 is processed in parallel by four hole convolutions with hole rates of 1, 3, 5 and 7 respectively, and the multi-scale depth feature maps R1, R2, R3 and R4 are obtained. The multi-scale deep feature map R4 is obtained, and then R1, R2, R3, R4 and the deep pointer instrument feature map F3 are connected and concatenated in the channel dimension (Concat operation). The size of the four feature maps R1, R2, R3 and R4 are 128×128 pixels, the number of channels is 64, and the multi-scale deep feature map F4 is obtained after the Concat operation. The size of F4 is 128×128 pixels and the number of channels is 32. After 1×1 convolution, the number of channels of the concatenated feature map is compressed to 64, and the multi-scale deep pointer instrument feature map F5 is obtained. The size of F5 is 256× 256 pixels, the number of channels is 64, and after a 3×3 convolution operation, the size becomes the same as F2, and a multi-scale deep pointer instrument feature map F5 is obtained. The size of F5 is 256×256 pixels, the number of channels is 64, and F5 is input to the pointer instrument detail feature information extraction secondary submodule; the multi-scale deep pointer instrument feature map F5 is input to the pointer instrument detail feature information enhancement secondary submodule to enhance the detail feature information of the industrial pointer instrument, and the detail feature information extraction feature map F6 is obtained in L1, the size of F6 is 256×256 pixels, and the number of channels is 64; In L2, the feature map F7 for extracting detail feature information is obtained. The size of F7 is 256×256 pixels and the number of channels is 64. F5, F6, F7 and F2 are concat-operated to obtain the feature map F8 for extracting detail feature information. Then, the feature map F8 is input into the two-line parallel attention module TPAM to obtain the semantic segmentation image of the pointer and scale. The size of the semantic segmentation image of the pointer and scale is 512×512 pixels and the number of channels is 1. The final result of the industrial instrument finger print extraction network is obtained (the background in the original industrial pointer instrument image is removed, and only the image of the pointer and scale is retained);

[0111] S2: Use the model training and verification first-level submodule to train and verify the industrial instrument fingerprint extraction network. Specifically, based on the industrial instrument fingerprint extraction network model constructed in S1, train and update the parameters of each layer; first initialize all neural network parameters and set model-related hyperparameters, including training rounds, batch size, weight decay coefficient, learning rate, and total number of iterations;

[0112] Wherein, S2 includes but is not limited to the following embodiments:

[0113] The industrial instrument fingerprint extraction network is trained, and the whole training round is 100 epochs; the weight decay coefficient is set to 0.05, the learning rate is set to 0.0000001, the batch size is set to 4, the number of iterations in the first 50 epochs is 217, the number of iterations in the next 50 epochs is 292, and the total number of iterations is 25450. Mosaic data enhancement is enabled in each round, and the resolution of the input image is uniformly adjusted to 512×512 format; after initializing the parameters, the industrial pointer instrument training set and industrial pointer instrument verification set data processed by Step 1 are divided into multiple batches, and a batch of industrial pointer instruments is added each time. The training set data is input into the industrial instrument finger print extraction network for training to obtain the training loss value loss of the batch; after completing one round of training for all batches of data in the entire pointer instrument training set (in the actual training process, there may be multiple rounds), the pointer instrument verification set is input into the industrial instrument finger print extraction network in batches to obtain the corresponding batch loss value batch_loss; during training and verification, the industrial instrument finger print extraction network will automatically learn and adjust parameters according to each loss and batch_loss situation; when the training process has been carried out for one or more rounds until the batch_loss value converges, the training of the industrial instrument finger print extraction network is completed;

[0114] S3: In the pointer and scale extraction first-level submodule, the industrial instrument finger mark extraction network is used to extract the semantic segmentation image of the pointer and scale. Specifically, after the industrial instrument finger mark extraction network model is trained, the industrial instrument finger mark extraction network is used to automatically extract the pointer and scale in the industrial pointer instrument image, and the semantic segmentation image of the pointer and scale is extracted;

[0115] Among them, S3 includes but is not limited to the following embodiments:

[0116] In practical applications, the remote sensing image classification dataset collected by the camera is input, and then the remote sensing image classification dataset is constructed and input into the industrial instrument finger scale extraction network, and the extraction result of the industrial pointer instrument is output, that is, the semantic segmentation image of the pointer and scale (that is, the background interference is removed and only the pointer and scale are retained).

[0117] The specific process of the industrial pointer instrument reading recognition module is as follows:

[0118] Step 1: Use Hough circle transform to calculate the center and radius of the circle in the semantic segmentation image of the pointer and scale;

[0119] Step 2: Get the deflection angle of the pointer;

[0120] Step 3: Calculate the relationship between the deflection angle of the pointer and the angle of the minimum scale line;

[0121] Step 4: Combine the relationship between the range and the angle to calculate the specific reading of the industrial pointer instrument;

[0122] Among them, Hough circle transform and angle method are both existing technologies; Hough circle transform is a popular method for detecting circles; by searching for circles that meet specified parameters in an image, using methods such as Canny edge detection, Hough circle transform is applied to detect circles in an image; the algorithm outputs the center coordinates and radius of each detected circle;

[0123] In addition, the industrial pointer instrument reading recognition module includes but is not limited to the following embodiments:

[0124] like Fig.10 As shown in the semantic segmentation image of the pointer and the scale, P is the pointer line, min is the minimum scale line, max is the maximum scale line, the angle between min and P is β, and the angle between max and P is α. The calculation formula of the pointer instrument reading R is:

[0125]

[0126] Assume that the coordinates of the center of the circle obtained by using Hough circle transform are (x 0 ,y 0 ), the radius of the instrument dial is r, and the intersection points of P, min, and max with the scale boundary are (x P ,y P )、(x min ,y min ) and (x max ,y max ),but: use The pointer instrument reading R can be obtained;

[0127] Substitute specific values: In the industrial pointer instrument for measuring mass, the coordinates of the center of the circle are (0, 0), the radius of the instrument dial is 1, the intersection points of P, min and max with the scale boundary are (0, 1), (-1, -1) and (1, -1), min = 0kg, max = 180kg, and we get β = 135°, α = 135°, and finally get the pointer instrument reading;

[0128] That is, the corresponding industrial pointer instrument reading is 90kg.

[0129] Although the present invention has been described above with reference to the embodiments, various modifications may be made thereto and parts thereof may be replaced by equivalents without departing from the scope of the present invention. In particular, as long as there is no structural conflict, the various features in the embodiments disclosed in the present invention may be used in combination with each other in any manner, and the fact that these combinations are not exhaustively described in this specification is only for the sake of omitting space and saving resources. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. An industrial pointer instrument reading recognition system based on deep learning, characterized in that: include: The dataset construction and division module is used to construct and divide the industrial pointer instrument image dataset; Industrial pointer instrument reading extraction network module, used to extract the semantic segmentation image of the pointer and scale of the industrial pointer instrument; Industrial pointer instrument reading recognition module, used to read the semantic segmentation image of pointer and scale using Hough circle transform and angle method; The data set construction and division module is connected to the industrial pointer instrument reading extraction network module, and the industrial pointer instrument reading extraction network module is connected to the industrial pointer instrument reading identification module; The industrial pointer instrument reading extraction network module is composed of a reading extraction network first-level submodule, a model training and verification first-level submodule, and a pointer and scale extraction first-level submodule; The first-level submodule of the reading extraction network is composed of a second-level submodule for extracting multi-residual small target features, a second-level submodule for extracting multi-type hole convolution features, and a second-level submodule for extracting pointer instrument detail feature information; The specific process of the industrial pointer meter reading extraction network module is as follows: S1: Use the first-level submodule of the reading extraction network to extract features from the industrial pointer instrument image dataset constructed by the dataset construction and partitioning module, and generate semantic segmentation images of the pointer and scale; S2: Use the model training and verification first-level submodule to train and verify the industrial instrument fingerprint extraction network; S3: In the pointer and scale extraction first-level submodule, the industrial instrument finger mark extraction network is used to extract the semantic segmentation image of the pointer and scale; The specific steps of S1 are as follows: S11: Use the multi-residual small target feature extraction secondary submodule to extract the shallow and deep features of the original pointer and scale; S12: Use multiple types of dilated convolution feature extraction submodules to extract multi-scale deep industrial pointer instrument image features; S13: extracting detailed feature information of industrial pointer instruments using the secondary submodule for extracting detailed feature information of pointer instruments; The specific steps of S13 are as follows: S131: input the multi-scale deep pointer instrument feature map F5 and the shallow pointer instrument feature map F2 into the pointer instrument detail feature information extraction secondary submodule. There are two processing routes, L1 and L2, in the pointer instrument detail feature information extraction module: in L1, the multi-scale deep pointer instrument feature map F5 is processed by a 1×1 ordinary convolution and a Sigmoid activation function to obtain a detail feature information extraction feature map F6, the size of the detail feature information extraction feature map F6 is 256×256 pixels, and the number of channels is 64; in L2, the multi-scale deep pointer instrument feature map F5 is processed by a 3×3 size, a hole rate of 2 hole convolution and a Sigmoid activation function to obtain a detail feature information extraction feature map F7, the size of the detail feature information extraction feature map F7 is 256×256 pixels, and the number of channels is 64; S132: Connect the shallow pointer instrument feature map F2, the detail feature information extraction feature map F6, the detail feature information extraction feature map F7 and the multi-scale deep pointer instrument feature map F5 and splice them in the channel dimension to obtain the detail feature information extraction feature map F8, the size of the detail feature information extraction feature map F8 is 256×256 pixels, the number of channels is 256, and then the detail feature information extraction feature map F8 is input into the dual-line parallel attention three-level sub-module TPAM; S133: In the two-line parallel attention three-level sub-module TPAM, the detail feature information extraction feature map F8 is bilinearly reshaped, the size is adjusted to 512×512 pixels, and the number of channels is adjusted to 256, so as to obtain the detail feature information extraction feature map F9; S134: After bilinear reshaping, average pooling and maximum pooling are performed in parallel, followed by double 1×1 convolution operations, which are then added, and finally a Sigmoid activation function operation is performed to obtain a detail feature information extraction feature map F10. The size of the detail feature information extraction feature map F10 is 512×512 pixels, and the number of channels is 256. The formula for obtaining the detail feature information extraction feature map F10 from the detail feature information extraction feature map F9 is as follows: F10=σ(B1(Avg(F9))+B2(Max(F9))) (3) Among them, Avg() represents average pooling, Max() represents maximum pooling, B1() and B2() represent the convolution combination operation of two convolution kernels of size 1×1, and σ represents the Sigmoid activation function; S135: performing a convolution operation on the detail feature information extraction feature map F10, wherein the convolution kernel size of the convolution operation is 1×1, and obtaining a semantic segmentation image of the pointer and the scale, wherein the semantic segmentation image size of the pointer and the scale is 512×512 pixels, and the number of channels is 1.

2. According to the deep learning-based industrial pointer instrument reading recognition system of claim 1, it is characterized in that: The specific process of the dataset construction and partitioning module is as follows: Process 1: Construct an industrial pointer instrument image dataset: First, use a camera to collect industrial pointer instrument image data to form an industrial pointer instrument classification dataset; then adjust each image in the industrial pointer instrument classification dataset to a uniform size, and annotate the pointers and scales in each image of the industrial pointer instrument classification dataset to generate an industrial pointer instrument segmentation dataset; then convert the industrial pointer instrument segmentation dataset into a semantic segmentation dataset, and convert the RGB color label of each image in the semantic segmentation dataset into a corresponding single-channel digital label; finally, combine the industrial pointer instrument single-channel digital label with the industrial pointer instrument classification dataset to form an industrial pointer instrument image dataset; Process 2: Divide the industrial pointer instrument image dataset: divide the industrial pointer instrument image dataset constructed by S1 into a training set, a validation set, and a test set according to a certain ratio.

3. The industrial pointer instrument reading recognition system based on deep learning according to claim 1 is characterized in that: The specific steps of S11 are as follows: S111: The size of the original pointer instrument image is 512×512 pixels, and the number of channels is 3. After the original pointer instrument image is input into the multi-residual small target feature extraction secondary submodule, a convolution operation is first performed by a convolution kernel with a size of 1×1 and a number of channels of 32 to obtain a shallow pointer instrument feature map F1. The size of the shallow pointer instrument feature map F1 is 256×256 pixels, and the number of channels is 32. After that, the shallow pointer instrument feature map F1 is input into the feature fusion convolution block; S112: input the shallow pointer instrument feature map F1 into the feature fusion convolution block, firstly perform a convolution operation by a convolution kernel of size 3×3 and padding 1, then process it through the ReLU activation function, then perform a convolution operation by a convolution kernel of size 3×3 and padding 1, then concatenate it with the shallow pointer instrument feature map F1 in the channel dimension, and finally use a convolution kernel of size 1×1 and padding 0 to adjust the number of channels, so as to obtain the shallow pointer instrument feature map F2; S113: perform spatial feature compression on the shallow pointer instrument feature map F2, that is, use average pooling and maximum pooling in parallel to process the shallow pointer instrument feature map F2 in the spatial dimension to obtain a one-dimensional vector of length C, and then perform one-dimensional convolution C1D on the one-dimensional vector t After processing, the weight w of each channel is obtained through the Sigmoid activation function. Finally, the weight w is multiplied by the shallow pointer instrument feature map F2 to obtain the deep pointer instrument feature map F3. The process is shown in the following formula (1): <h2 style=";text-align:left;direction:ltr">F3 = w⊙F2 = σ(C1D<h2 style=";text-align:left;direction:ltr"> t <h2 style=";text-align:left;direction:ltr"> (F<h2 style=";text-align:left;direction:ltr"> 3×3 <h2 style=";text-align:left;direction:ltr"> ([AvgPool(F2);MaxPool(F2)]))⊙F2 (1) Among them, AvgPool represents average pooling, MaxPool represents maximum pooling, and F 3×3 () indicates a convolution operation with a convolution kernel size of 3×3, C1D t represents a one-dimensional convolution with a kernel size of t, σ represents the Sigmoid activation function, ⊙ represents the element-wise product, and C1D t The role of is to generate channel weights, and t is adaptively determined according to the mapping of the number of channels c of the output feature: Where []odd represents the nearest odd number of the parameter, ω, β and γ are hyperparameters; Finally, the deep pointer instrument feature map F3 is input into the multi-class hole convolution feature extraction secondary submodule. The size of the deep pointer instrument feature map F3 is 128×128 pixels and the number of channels is 64.

4. The industrial pointer instrument reading recognition system based on deep learning according to claim 1 is characterized in that: The specific steps of S12 are as follows: S121: input the deep pointer instrument feature map F3 into the multi-type hole convolution feature extraction secondary submodule, the deep pointer instrument feature map F3 is first processed in parallel by four hole convolutions, and the hole rates of the four hole convolutions are 1, 3, 5 and 7 respectively, then the deep pointer instrument feature map F3 is respectively subjected to hole convolution operations to obtain multi-scale depth feature maps R1, multi-scale depth feature maps R2, multi-scale depth feature maps R3 and multi-scale depth feature maps R4, finally the multi-scale depth feature maps R1, multi-scale depth feature maps R2, multi-scale depth feature maps R3, multi-scale depth feature maps R4 and the deep pointer instrument feature map F3 are spliced ​​in the channel dimension, wherein the sizes of the multi-scale depth feature maps R1, multi-scale depth feature maps R2, multi-scale depth feature maps R3 and multi-scale depth feature maps R4 are all 128×128 pixels, and the number of channels is 64; S122: After splicing the multi-scale depth feature map R1, the multi-scale depth feature map R2, the multi-scale depth feature map R3, the multi-scale depth feature map R4 and the deep pointer instrument feature map F3 in the channel dimension, the result is the multi-scale depth feature map F4, the size of the multi-scale depth feature map F4 is 128×128 pixels, and the number of channels is 32. After the multi-scale depth feature map F4 is subjected to 1×1 convolution, the number of channels of the multi-scale depth feature map F4 is compressed to 64. Finally, after 3×3 convolution, the result of the multi-type hole convolution feature extraction secondary sub-module is obtained, that is, the multi-scale deep pointer instrument feature map F5. The size of the multi-scale deep pointer instrument feature map F5 is 256×256 pixels, and the number of channels is 64. The multi-scale deep pointer instrument feature map F5 is input into the pointer instrument detail feature information extraction secondary sub-module.

5. The industrial pointer instrument reading recognition system based on deep learning according to claim 1 is characterized in that: The specific process of the industrial pointer meter reading recognition module is as follows: Step 1: Use Hough circle transform to calculate the center and radius of the circle in the semantic segmentation image of the pointer and scale; Step 2: Get the deflection angle of the pointer; Step 3: Calculate the relationship between the deflection angle of the pointer and the angle of the minimum scale line; Step 4: Combine the relationship between the range and the angle to calculate the specific reading of the industrial pointer instrument.

Images