Automatic pointer type instrument identification method based on deep learning

Abstract

The invention discloses an automatic pointer instrument identification method based on deep learning. The automatic pointer type instrument identification method based on deep learning comprises the following steps: acquiring a pointer type instrument picture, and performing picture segmentation processing on the acquired pointer type instrument picture to obtain an instrument dial region picture; performing data augmentation processing on the pointer type instrument picture and the instrument dial region picture to generate pointer type instrument image data; in combination with the generated image data of the pointer type instrument, a pointer type instrument identification model based on Mask-RCNN-swin trassformer is constructed; and inputting the obtained pointer type instrument picture into a pointer type instrument identification model, and outputting an identification result of the pointer type instrument picture by the pointer type instrument identification model. According to the automatic pointer type instrument identification method based on deep learning, the effect of improving the identification rate of the pointer type instrument identification method is achieved, and the problem that in the prior art, the identification rate of a current pointer type instrument identification method is low is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic pointer instrument recognition, and in particular to an automatic pointer instrument recognition method based on deep learning. Background Art

[0002] The petroleum and petrochemical industry is an important pillar industry of the national economy. The safety and efficiency of its production process are of key significance to the country's energy supply and economic stability. The use of artificial intelligence algorithms to build an automated system for acquiring pointer instrument data can significantly improve the efficiency and accuracy of pointer instrument data acquisition, which will greatly improve the overall operating efficiency of the petroleum and petrochemical industry.

[0003] Currently, pointer instruments are still widely used in large-scale petroleum and petrochemical plants. Due to cost and early design considerations, most pointer instruments do not have communication interfaces. Relevant personnel need to manually obtain the data of pointer instruments regularly or at any time. Manually obtaining chart values ​​consumes a lot of manpower and financial resources. In addition, large-scale petroleum and petrochemical plants usually have complex equipment layouts, diverse process flows and harsh working conditions, which leads to the problem of low recognition rate of current pointer instrument recognition methods. Summary of the invention

[0004] The embodiments of the present application solve the problem of low recognition rate of current pointer instrument recognition methods in the prior art by providing an automated pointer instrument recognition method based on deep learning, thereby improving the recognition rate of the pointer instrument recognition method.

[0005] The embodiment of the present application provides an automatic pointer instrument recognition method based on deep learning, comprising the following steps: collecting pointer instrument pictures, performing image segmentation processing on the collected pointer instrument pictures to obtain instrument dial area pictures; performing data augmentation processing on the pointer instrument pictures and the instrument dial area pictures to generate pointer instrument image data; combining the generated pointer instrument image data, constructing a pointer instrument recognition model based on Mask-RCNN-swin trasnsformer; inputting the acquired pointer instrument pictures into the pointer instrument recognition model, and the pointer instrument recognition model outputs the recognition result of the pointer instrument pictures.

[0006] Furthermore, the image segmentation processing is performed on the collected pointer instrument image, specifically including: obtaining a preset dial template, sliding the dial template as a sliding window on the pointer instrument image, and calculating the matching degree; cropping the image of the area with the highest matching degree to obtain the instrument dial area image.

[0007] Furthermore, the data augmentation processing of the pointer instrument image and the instrument dial area image includes: performing geometric transformation and pixel transformation on the pointer instrument image and the instrument dial area image.

[0008] Furthermore, the geometric transformation mainly includes: rotating the rectangular pointer instrument image by 180°, and rotating the rectangular instrument dial area image by 90°, 180°, and 270° to obtain first image data; scaling the pointer instrument image outward, and cropping the scaled pointer instrument image to a size consistent with the first image data; scaling the pointer instrument image inward, and then scaling it outward to a size consistent with the first image data.

[0009] Furthermore, the pointer instrument recognition model based on Mask-RCNN-swin trasnsformer includes a feature extraction module, a feature fusion module, a classification regression network, and a pointer position recognition module, specifically including: the feature extraction module, using the feature extraction network to extract features of different dimensions of the input pointer instrument image data to obtain an instrument feature map; the feature fusion module, used to perform feature fusion on the extracted instrument feature map; the classification regression network, based on a fully convolutional neural network, generates a binary mask for the located features and performs instance segmentation; the pointer position recognition module, using perspective transformation, PCA pointer straight line fitting, and angle calculation to realize pointer reading.

[0010] Furthermore, the feature extraction module performs a deconvolution operation on the obtained instrument feature map, and the operation method is as follows: the deconvolution operation is performed by reducing the instrument feature map output by the 5-layer convolution pooling layer, and the instrument feature map is enlarged to the same size as the first image data during output, so as to obtain a more accurate instrument feature map.

[0011] Furthermore, the feature fusion occurs in the lateral connection process of the feature pyramid network based on multi-dimensional feature fusion, specifically: the feature is restored to the same scale as the low-dimensional feature by upsampling the high-dimensional feature map once; the low-dimensional feature map changes the number of feature channels while ensuring that the scale does not change through a 1×1 convolution operation; the low-dimensional feature map is simply fused with the high-dimensional feature map.

[0012] Furthermore, the classification and regression network has an operation method including: converting each region proposal box into a feature map of the same size in a pooling layer; inputting the obtained feature map into two fully connected layers, and inputting the offset of the anchor box corresponding to each region proposal box; the classification layer outputs the probability that each proposal box belongs to each category; and using a non-maximum suppression algorithm to select the feature map with the highest confidence as the final detection result.

[0013] Furthermore, the recognition result of the pointer instrument picture specifically includes: if it is a pointer instrument picture, the model will recognize the pointer position and output it; if the input is a non-pointer instrument picture, the system will directly end and return a corresponding prompt.

[0014] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0015] 1. By constructing an automatic identification system for pointer instruments in large petroleum and petrochemical plants, the recognition rate of pointer instrument identification methods is improved, effectively solving the problem of low recognition rate of current pointer instrument identification methods in the prior art.

[0016] 2. The algorithm innovatively uses Mask-RCNN as the baseline framework and the Swin transformer as the backbone network. By pre-training on a large amount of multi-dimensional pointer instrument image data, a pointer instrument image recognition model with high recognition accuracy and strong versatility is obtained.

[0017] 3. Through the existing data augmentation technology, a data augmentation technology suitable for pointer instrument images in the petroleum and petrochemical industry is proposed, so as to generate pointer instrument images with occlusion, fogging, rust and stains close to the production environment, thereby solving the problem of insufficient image data of pointer instruments in the petroleum and petrochemical industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A method flow chart of an automatic pointer instrument recognition method based on deep learning provided in an embodiment of the present application;

[0019] Figure 2 An algorithm flow chart of an automated pointer instrument recognition method based on deep learning provided in an embodiment of the present application;

[0020] Figure 3 An algorithm schematic diagram of an automatic pointer instrument recognition method based on deep learning provided in an embodiment of the present application; DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0023] Collecting pointer instrument pictures, performing image segmentation processing on the collected pointer instrument pictures, and obtaining instrument dial area pictures;

[0024] Performing data augmentation processing on the pointer instrument image and the instrument dial area image to generate pointer instrument image data;

[0025] Combined with the generated pointer instrument image data, a pointer instrument recognition model based on Mask-RCNN-swin trasnsformer is constructed;

[0026] The acquired pointer instrument image is input into the pointer instrument recognition model, and the pointer instrument recognition model outputs the recognition result of the pointer instrument image.

[0027] The pointer instrument picture is acquired by an image acquisition device.

[0028] The image segmentation process of the collected pointer instrument image specifically includes:

[0029] Get a preset dial template, slide the dial template as a sliding window on the pointer instrument picture, and calculate the matching degree;

[0030] The image of the area with the highest matching degree is cropped to obtain the instrument dial area image.

[0031] The image data of pointer instruments in large petroleum and petrochemical plants in different environments and weather conditions are relatively scarce. In order to improve the accuracy of the pointer instrument recognition model, as much pointer instrument image data as possible in different environments and weather conditions is needed. Therefore, this paper constructs a data expansion and augmentation technology for the petroleum and petrochemical industry.

[0032] The data expansion and augmentation technology generates a large amount of high-reliability, high-definition pointer image data by performing mixed simulation and super-resolution reconstruction on the existing image data, and simulating operations such as fogging and rusting on the existing data. Finally, the newly generated data is added together with the original data to the training of the pointer instrument recognition pre-training model to fully activate the pre-trained network.

[0033] Traditional image data enhancement solutions are mainly based on geometric transformation and pixel transformation of image data. Through the above transformations, the original data samples are expanded into image data with higher quality and more samples. The data augmentation processing includes geometric transformation and pixel transformation.

[0034] The geometric transformation mainly includes: in the process of processing the data of the pointer instrument, the first picture data is obtained by rotating the rectangular pointer instrument image by 180° and rotating the rectangular instrument dial area image by 90°, 180°, and 270°; by scaling the pointer instrument image inward and outward, the outward scaled image is cropped to the same size as the first picture data, and the inward scaled image is rotated and then scaled outward to the same size as the first picture data.

[0035] The main principle of pixel transformation is to use image noise, image blur, and image fusion information to augment the original image information, where Gaussian noise can be used to add noise blocks to the original image information;

[0036] The pixel transformation specifically includes:

[0037] The obtained first image data is blurred by combining Gaussian blur, and pointer instrument images of different categories are mixed by SMOTE, and the mixed first image data is cropped to obtain pointer instrument image data.

[0038] In order to perform efficient modeling, the algorithm introduces the calculation of self-attention in non-overlapping local windows, and divides the image into windows evenly in a non-overlapping manner. Specifically, assuming that each non-overlapping local window contains N = M × M patchtokens, the computational complexity of the image window module MSA based on N = h × m patchtokens and the non-overlapping local window WMSA module are:

[0039] Ω(MSA)=4×h×w×C 2 +2×(h×w) 2 ×C

[0040] Ω(MSA)=4×h×w×C 2 +2×M 2 ×h×w×C

[0041] Where: MSA has quadratic complexity with respect to the number of patchtokens h×w, M is the number of pixels in the feature map, and WMSA has linear complexity when the number of pixels in the feature map is fixed.

[0042] The first module uses the following regular window partitioning strategy:

[0043] The 8×8 feature map is evenly divided into 2×2 windows of size 4×4, where M = 4. Then, the regularly divided window is circularly shifted to the upper left by a preset number of pixels.

[0044] It should be noted that the preset number is half of the number of pixels in the feature map.

[0045] Using the shift window division method, the calculation formula of SwinTransformer is as follows:

[0046]

[0047] Where: represents the output feature of the (S)WMSA module of the i-th block, z i Represents the output features of the i-th MLP module.

[0048] Mask-RCNN was proposed by Kaiming He and others from the Facebook Artificial Intelligence Group at the 2017 ICCV International Conference. It uses a two-stage instance segmentation deep neural network to achieve multi-task modes such as object classification, object detection, semantic segmentation, and instance segmentation for images.

[0049] Currently, many scholars have applied Mask-RCNN to the pointer instrument reading acquisition system. Among them, Fang Yixiao used Mask-RCNN to detect the key points of the pointer and scale of the pointer instrument image to realize pointer reading recognition, and He Peilin used PrRoIPooling instead of RoiAlign to improve the instrument recognition accuracy of Mask-RCNN.

[0050] The implementation method of Mask-RCNN is to add a fully convolutional network branch on the basis of Faster-RCNN to generate the corresponding mask, use the mask to obtain the pixel area occupied by each target in the image, and then accurately segment the target through the contour of each target pixel area.

[0051] The pointer instrument recognition model based on Mask-RCNN includes a feature extraction module, a classification regression network, an instance segmentation network, and a pointer position recognition module;

[0052] The feature extraction module uses a feature extraction network to extract features of different dimensions from the input pointer instrument image data;

[0053] The feature fusion module performs feature fusion on the extracted features;

[0054] The instance segmentation network generates a binary mask for the located features based on a fully convolutional neural network and performs instance segmentation;

[0055] The pointer position recognition module realizes pointer reading by using perspective transformation, PCA pointer straight line fitting and angle calculation.

[0056] The feature extraction network is a feature pyramid network based on multi-dimensional feature fusion. The feature pyramid network performs a three-level feature extraction process, including bottom-up feature extraction, top-down feature fusion, and left-to-right horizontal connection.

[0057] The feature fusion occurs in the lateral connection process of the feature pyramid network based on multi-dimensional feature fusion, specifically:

[0058] The high-dimensional feature map is upsampled once to restore the feature to the same scale as the low-dimensional feature; the low-dimensional feature map is convolved by a 1×1 operation while changing the number of feature channels while ensuring that the scale remains unchanged; the low-dimensional feature map is simply fused with the high-dimensional feature map. The above operations are completed in sequence to fuse the feature maps of each dimension and use the fusion results for independent prediction.

[0059] The instance segmentation network consists of two fully connected layers. In order to adapt the input vector length of the fully connected layer, the RPN needs to be adjusted to a fixed size. The specific operation is as follows:

[0060] First, use the pooling layer to convert each region proposal box into a feature map of the same size;

[0061] The obtained feature map is input into two fully connected layers for classification and regression prediction of the target.

[0062] It should be noted that the regression layer outputs the offset of the anchor box corresponding to each suggestion box, and the classification layer outputs the probability that each suggestion box belongs to each category. Finally, the non-maximum suppression algorithm is used to select the target box with the highest confidence as the final detection result.

[0063] The principle of the fully convolutional network is to represent the fully connected layers in the convolutional neural network with convolutional layers. In ordinary convolutional neural networks, layers 1 to 5 are convolutional layers, layers 6 and 7 are one-dimensional vectors of length 4096, and the 8th layer is a one-dimensional vector of length 1000 corresponding to 1000 category classification probabilities. Layers 6, 7, and 8 are all fully connected layers in the convolutional neural network, and the fully convolutional network converts the fully connected layers in the convolutional neural network into convolutional layers. The size of the convolutional layer is represented by the number of channels, width, and height as (4096, 1, 1), (4096, 1, 1), and (1000, 1, 1), respectively.

[0064] The final output of the convolutional neural network is the probability that the image belongs to each category, obtained by normalizing the 1000-dimensional vector using softmax; and the final output of the full convolutional network is to upsample the last convolution layer through a deconvolution operation, restore the feature map to the original image size, and then predict each image pixel.

[0065] After multiple convolution and pooling operations, the fully convolutional network will make the feature map of the image more and more blurred, and the image resolution will become lower and lower. In order to solve this problem, the fully convolutional network uses deconvolution operations to implement upsampling technology. The specific operations are as follows:

[0066] The five convolutions and pooling operations reduce the image by 2, 4, 8, 16, and 32 times. The deconvolution operation upsamples the 5th, 4th, 3rd, 2nd, and 1st feature maps, respectively, and enlarges the feature maps to the size of the original image feature maps, thereby obtaining a more accurate image pixel segmentation result.

[0067] The recognition results of the pointer instrument image include:

[0068] If it is a pointer instrument picture, the model will identify the pointer position and output it. If the input is a non-pointer instrument picture, the system will end directly and return the corresponding prompt.

[0069] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0070] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0071] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0072] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0073] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0074] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An automatic pointer instrument recognition method based on deep learning, characterized in that: The following steps are involved: Collecting pointer instrument pictures, performing image segmentation processing on the collected pointer instrument pictures, and obtaining instrument dial area pictures; Performing data augmentation processing on the pointer instrument image and the instrument dial area image to generate pointer instrument image data; Combined with the generated pointer instrument image data, a pointer instrument recognition model based on Mask-RCNN-swin trasnsformer is constructed; The acquired pointer instrument image is input into the pointer instrument recognition model, and the pointer instrument recognition model outputs the recognition result of the pointer instrument image.

2. The method for automatic pointer instrument recognition based on deep learning as claimed in claim 1, characterized in that: The step of performing image segmentation processing on the collected pointer instrument image specifically includes: Get a preset dial template, slide the dial template as a sliding window on the pointer instrument picture, and calculate the matching degree; The image of the area with the highest matching degree is cropped to obtain the instrument dial area image.

3. The method for automatic pointer instrument recognition based on deep learning as claimed in claim 1, characterized in that: The data augmentation processing of the pointer-type instrument picture and the instrument dial area picture includes: performing geometric transformation and pixel transformation on the pointer-type instrument picture and the instrument dial area picture.

4. The method for automatic pointer instrument recognition based on deep learning as claimed in claim 1, characterized in that: The geometric transformation mainly includes: The rectangular pointer instrument image is rotated 180°, and the rectangular instrument dial area image is rotated 90°, 180°, and 270° to obtain first image data; Scaling the pointer-type instrument image outward, and cropping the scaled-out pointer-type instrument image to a size consistent with the first image data; After the pointer-type instrument image is scaled inward, it is scaled outward to be consistent with the size of the first image data.

5. The method for automatic pointer instrument recognition based on deep learning as claimed in claim 1, characterized in that: The calculation formula of the Swin Transformer is as follows: Where: represents the output feature of the (S)WMSA module of the i-th block, z i Represents the output features of the i-th MLP module.

6. The method for automatic pointer instrument recognition based on deep learning as claimed in claim 1, characterized in that: The pointer instrument recognition model based on Mask-RCNN-swin trasnsformer includes a feature extraction module, a feature fusion module, a classification regression network, and a pointer position recognition module, specifically including: The feature extraction module uses a feature extraction network to extract features of different dimensions from the input pointer instrument image data to obtain an instrument feature map; The feature fusion module is used to perform feature fusion on the extracted instrument feature graph; The classification regression network generates a binary mask for the located features based on a fully convolutional neural network and performs instance segmentation; The pointer position recognition module realizes pointer reading by using perspective transformation, PCA pointer straight line fitting and angle calculation.

7. The method for automatic pointer instrument recognition based on deep learning as claimed in claim 1, characterized in that: The feature extraction module performs a deconvolution operation on the obtained instrument feature map, and the operation method is as follows: The deconvolution operation reduces the instrument feature map output by the 5-layer convolution pooling layer, and enlarges the instrument feature map to the same size as the first image data during output, thereby obtaining a more accurate instrument feature map.

8. The method for automatic pointer instrument recognition based on deep learning as claimed in claim 1, characterized in that: The feature fusion occurs in the lateral connection process of the feature pyramid network based on multi-dimensional feature fusion, specifically: The high-dimensional feature map is upsampled once to restore the features to the same scale as the low-dimensional features; the low-dimensional feature map is changed through a 1×1 convolution operation while the number of feature channels is changed while the scale remains unchanged; the low-dimensional feature map is simply fused with the high-dimensional feature map.

9. The method for automatic pointer instrument recognition based on deep learning as claimed in claim 1, characterized in that: The classification regression network, the operation method includes: In the pooling layer, each region proposal box is converted into a feature map of the same size; The obtained feature map is input into two fully connected layers, and the offset of each region proposal box corresponding to the anchor box is input; The classification layer outputs the probability that each suggestion box belongs to each category; The non-maximum suppression algorithm is used to select the feature map with the highest confidence as the final detection result.

10. The method for automatic pointer instrument recognition based on deep learning as claimed in claim 1, characterized in that: The recognition result of the pointer instrument image specifically includes: If it is a pointer instrument picture, the model will identify the pointer position and output it. If the input is a non-pointer instrument picture, the system will end directly and return the corresponding prompt.

Images