Method and apparatus for reading a pointer-type instrument based on machine vision

Abstract

The application discloses a pointer type instrument reading method and device based on machine vision. An instrument image of a target instrument is collected, the target instrument being a pointer type instrument to be identified for reading; an instrument dial and a dial center are located from the instrument image by using a detection framework based on machine vision; each element in the instrument dial is segmented by using an anti-tilt non-uniform scale instrument dial segmentation scheme; and automatic reading of the instrument dial is performed by using at least one of a character value method and an angle method. The application solves the technical problem of low efficiency of reading an instrument by using manual reading in the related art.

Description

Technical Field

[0001] This application relates to the field of electrical automation, and more specifically, to a method and apparatus for reading pointer-type instruments based on machine vision. Background Technology

[0002] Power plants are the engines of the power grid, and the readings of various instruments within them are crucial for real-time monitoring of power plant operation data. Instruments in power plants are mainly divided into two categories: pointer-type instruments and digital display instruments. Although digital display instruments are widely used in industrial automation, pointer-type instruments still play an important role in industrial production due to their simple structure, anti-interference capabilities, and durability. However, pointer-type instruments mostly lack communication interfaces, requiring manual reading, which is inefficient and prone to errors. With the development of artificial intelligence, it has become possible to automatically read instruments using machine vision technology; however, there is still no mature and reliable technical solution for this purpose.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a machine vision-based method and apparatus for reading pointer-type instruments, which at least solves the technical problem of low efficiency in reading instruments manually in related technologies.

[0005] According to one aspect of the embodiments of this application, a machine vision-based method for reading pointer-type instruments is provided, comprising:

[0006] Step S1, Detection framework for automatic instrument reading: Considering the influence of factors such as the production environment and lighting conditions of the pointer-type instrument, resulting in low image quality, an image data preprocessing scheme including color gamut transformation, geometric transformation, contrast enhancement, and saturation enhancement is adopted to provide clear and easily processed image data for subsequent detection, localization, and segmentation algorithms. After preprocessing the acquired instrument image data, a Faster R-CNN object detection network based on deep learning is used to detect and locate the dial, and then the Hough circle detection is used to locate the center coordinates of the dial.

[0007] Step S2, Anti-tilt non-uniform scale instrument segmentation algorithm: After visual preprocessing of the instrument image and localization of the dial and center, an anti-tilt non-uniform scale instrument dial segmentation algorithm is designed. Hough line detection corrects the tilt of the detected dial image. Based on this, a high-resolution hollow feature pyramid and a deformable convolution decoder are designed respectively. Low-level detail information and context information are fused, and the receptive field of the decoder is changed to identify the deformation caused by the tilt correction. Finally, the angle method is introduced to combine the numerical characters and pointer angle to realize the automatic reading of non-uniform scale instruments.

[0008] Step S3, Automatic Instrument Reading Using a Combination of Character Values ​​and Angle Methods: Preprocessing and instrument dial segmentation are performed on the images of pointer instruments (including those with uniform and non-uniform scales) acquired from the power plant. This determines the pointer angle, the corresponding scale character values, and the instrument range, enabling automatic reading of single / multi-pointer uniform or non-uniform scale instruments. The instrument range and scale character values ​​are crucial for the automatic reading of non-uniform scale pointer instruments.

[0009] Optionally, the technical features of the detection framework for automatic instrument reading in step S1 are implemented as follows:

[0010] Step S11, Data Preprocessing: The instrument detection algorithm uses horizontal flipping to preprocess and enhance the dataset, ignoring the influence of dataset labeling errors. However, due to the influence of production environment, lighting, etc., the image quality is low, which seriously affects the detection accuracy. Therefore, this application directly removes horizontal flipping and adds data preprocessing such as color gamut transformation, geometric transformation, and contrast enhancement. That is, on the original training samples, the above preprocessing methods are randomly combined to improve the quality of instrument images, thereby improving the robustness of detection.

[0011] Step S12, Detection framework for automatic reading of pointer instruments: Since the dial segmentation algorithm requires a clear image containing all the information of the dial, a detection and positioning framework for automatic reading of pointer instruments is introduced to realize the position detection of the object dial. At the same time, features are extracted for pointer instruments with uniform and non-uniform scales, and the extracted features are shared with the subsequent segmentation network for dial segmentation. In addition, Hough circle detection is used to extract the center of the dial, that is, the pole of polar coordinates, to realize subsequent automatic reading.

[0012] Optionally, the technical features of the instrument segmentation algorithm for anti-tilt non-uniform scale in step S2 are implemented as follows:

[0013] Step S21, dial tilt correction based on improved Hough transform: by utilizing the duality of points and lines, the lines in the instrument image space are transformed into aggregation points in the parameter space, thereby detecting whether a curve with a given property exists in the given image. The number of collinear points in the instrument image space is detected by detecting the number of sine curves passing through a certain point in the parameter space. All curves passing through the point are counted in the array, and finally the value with the largest number in the array is found. This value is the tilt angle of the dial.

[0014] Step S22: Based on the improved deepLabv3+ dial segmentation framework, deepLabV3+ replaces the max pooling layer with depthwise separable convolution in the Xecption feature extraction network, which reduces the number of parameters and computation. However, because its depthwise separable convolution lacks contextual information, it has poor ability to perceive the deformation of the dial image. Therefore, deformable convolution is introduced on the ResNet50 feature extraction network to realize the perception of contextual information. This feature network is shared from the aforementioned dial detection and localization framework.

[0015] Step S23, High-resolution hollow feature pyramid pooling: In deepLabV3+, hollow spatial pyramid pooling is used for the 16× downsampled image. The high downsampling ratio will cause the dial features to be lost and cannot be recovered. Therefore, an improved high-resolution hollow feature pyramid pooling is proposed, which uniformly performs pyramid pooling on the 8× downsampled features, which improves the resolution of the encoder features and facilitates subsequent decoding.

[0016] Step S24, Deformable Convolution-Based Decoder: Since different shooting angles cause deformation, increasing the difficulty of dashboard segmentation, ResNet50 first extracts features, then HR-ASPP processes the features to obtain 8× upsampled features F. 8× To obtain a clear segmented dial image, the features need to be upsampled, and then the features F are further refined. 8× Perform two deformable convolution upsampling operations to obtain the region context pixel aggregation F. 4× and F 8× Features, and finally F 4×4 and F 8×8 Upsampled to F 2×2 The resolution is combined to achieve the fusion of local and contextual features, and convolutional upsampling is performed to achieve the segmentation result output. This decoder with deformable convolution has stronger resistance to deformation.

[0017] Optionally, the technical feature of automatic instrument reading using the combined character numerical value and angle method in step S3 is implemented as follows:

[0018] Step S31, Recognition of instrument range and scale character values ​​based on multilayer perceptron: In order to achieve automatic reading, it is necessary to use the scale indication and range indication in the dial. This application segments the characters in the image of the pointer instrument, selects the digital images as the dataset, and trains the multilayer perceptron model. The trained multilayer perceptron can accurately read the key characters in the instrument, such as the range and scale indication.

[0019] Step S32, reading conversion of pointer instrument based on angle method: This application uses the angle method to read pointer instrument. The center of the pointer area pixel is established according to the center of the circle, which is the pointer center line. The angles α and β between the pointer center line and the adjacent two scale values ​​are calculated through the linear relationship of angles to obtain the actual value of the pointer.

[0020] Step S33, Automatic reading of the instrument by combining character value recognition and angle method: For non-uniform pointer instruments, conventional angle method recognition cannot achieve reading of the display value. It is necessary to combine the instrument range and scale character value to achieve automatic reading.

[0021] According to another aspect of the embodiments of this application, a pointer-type instrument reading device based on machine vision is also provided, comprising: an acquisition unit for acquiring an instrument image of a target instrument, wherein the target instrument is a pointer-type instrument for which a reading is to be identified; a positioning unit for locating the instrument dial and the dial center from the instrument image using a machine vision-based detection framework; a segmentation unit for segmenting each element in the instrument dial using an instrument dial segmentation scheme with anti-tilt non-uniform scale; and a reading unit for automatically reading the instrument dial using at least one of character numerical method and angle method.

[0022] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor performs the above-described method through the computer program.

[0023] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of any of the embodiments of the methods described above.

[0024] The present invention employs a method of acquiring an image of a target instrument, wherein the target instrument is a pointer-type instrument for which readings are to be identified. A machine vision-based detection framework is used to locate the instrument dial and its center from the image. An anti-tilt, non-uniform scale instrument dial segmentation scheme is used to segment the elements within the instrument dial. At least one of the character numerical method and the angle method is used for automatic reading of the instrument dial. By employing artificial intelligence and an anti-tilt solution tailored to the harsh environment in which the instrument is located, the inefficiency of manual instrument reading in related technologies can be solved.

[0025] In addition to the objectives, features, and advantages described above, the present invention has other objectives, features, and advantages. The invention will now be described in further detail with reference to the figures. Attached Figure Description

[0026] The accompanying drawings, which form part of this specification, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0027] Figure 1 This is a flowchart of an optional machine vision-based pointer-type instrument reading method according to an embodiment of this application;

[0028] Figure 2 This is a flowchart of an optional machine vision-based pointer-type instrument reading method according to an embodiment of this application;

[0029] Figure 3 This is a schematic diagram of an optional instrument dial positioning detection result according to an embodiment of this application;

[0030] Figure 4 This is a schematic diagram of an optional instrument dial center positioning result according to an embodiment of this application;

[0031] Figure 5 This is a schematic diagram of an optional pointer segmentation result according to an embodiment of this application;

[0032] Figure 6 This is a schematic diagram illustrating an optional scale character area detection and recognition according to an embodiment of this application;

[0033] Figure 7 This is a schematic diagram of an optional angle method for automatic instrument reading according to an embodiment of this application;

[0034] Figure 8 This is a schematic diagram of an optional machine vision-based pointer-type instrument reading device according to an embodiment of this application. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0036] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] When automatically reading pointer-type instruments using machine vision, this can be achieved through statistical feature-based instrument reading recognition algorithms and convolutional neural network-based recognition algorithms. This solution aims to integrate the advantages of these two types of algorithms to further improve recognition performance. In pointer and dial positioning, the Hough transform algorithm is used to detect the pointer, combined with the center projection method to position it, thus improving the accuracy of instrument readings. Alternatively, the dial area can be located using the color histogram distribution features and brightness gradient features of the dial, and the pointer can be positioned using an improved Hough transform algorithm. Furthermore, support vector machines can be used to train image HOG (Histogram of Oriented Gradient, HOG) features to detect the dial, and then the pointer position can be detected by combining the image grayscale information with the Hough transform.

[0038] When identifying instrument readings based on statistical features, methods such as the Blob connected component method and the Hough transform can be used. However, considering that instrument images are susceptible to noise, resulting in discontinuous pointer connected components, and that the Hough line detection algorithm has high space and time complexity, and that the detected line is not the pointer center line but the pointer edge, this leads to reduced reading accuracy. Furthermore, instrument readings typically require pre-calibrated range information obtained from a template library. To address these issues, this application proposes a machine vision-based scheme for reading pointer-type instruments. This scheme segments and locates the dial, scale values, and pointer position, acquiring information such as the instrument scale values, dial center, and pointer center line. This method can accurately locate the pointer center line in complex backgrounds and, by referring to the instrument range information, can read the readings of non-uniform scale pointer-type instruments. The dial can be detected using the Faster R-CNN object detection algorithm based on convolutional neural networks, the pointer can be located using connected component analysis, and a deep learning-based segmentation algorithm can be used to segment the dial, scale values, and pointer position. This application combines the advantages of two types of algorithms, introducing an improved segmentation algorithm based on deep learning, deepLabV3+, to segment the dial area, and using perspective Hough transform to correct the instrument.

[0039] One of the innovations of this application is the proposal of an anti-tilt non-uniform scale meter reading algorithm based on the improved segmentation network deepLabv3+. For the preprocessed dial image, a two-stage object detection method is used for dial and circle center localization detection. This primarily utilizes a two-stage Faster R-CNN to locate the dial region and Hough circle detection to locate the circle center. To improve the segmentation accuracy of the dial pointers, scales, and scale characters, an improved deepLabV3+ dial segmentation framework is proposed. First, dial tilt correction is performed based on an improved Hough transform: the duality of points and lines is used to correct the tilt of the dial. Then, a high-resolution dilated spatial convolutional pooling pyramid (HR-ASPP) and a deformable de-decoder are fused, extracting features with richer details and contextual information, and improving the classifier's resistance to deformation. This makes the segmented dial area more precise. Finally, by combining the dial character values ​​and polar coordinate angles, the readings under different angle scale markings are converted to achieve pointer readings for non-uniform scales. Therefore, this application can be applied to scenarios with both uniform and non-uniform pointer scales on multiple instruments, exhibiting good accuracy and applicability.

[0040] According to one aspect of the embodiments of this application, a method embodiment for reading pointer-type instruments based on machine vision is provided. The machine vision-based pointer-type instrument reading method of this application embodiment can be executed by a server or a terminal. Specifically, the execution of the machine vision-based pointer-type instrument reading method of this application embodiment by a terminal can also be executed by a client installed on it. Figure 1 This is a flowchart of an optional machine vision-based pointer-type instrument reading method according to an embodiment of this application, such as... Figure 1 As shown, the method may include the following steps:

[0041] Step S102: Acquire the instrument image of the target instrument, which is a pointer-type instrument with the reading to be identified.

[0042] Step S104: Using a machine vision-based detection framework, locate the instrument panel and the center of the panel from the instrument image.

[0043] Optionally, before locating the instrument dial and dial center from the instrument image, the original sample is preprocessed to obtain an enhanced sample. The preprocessing includes one or a combination of randomly selected one of color gamut transformation, geometric transformation, contrast enhancement, and saturation enhancement. The preprocessing is used to eliminate the influence of the production environment and lighting on the image. The random selection is used to improve the robustness of detection. The original sample is an image obtained by image acquisition of the instrument. The enhanced sample is marked with the dial position and dial center. The enhanced sample is used for training to obtain the detection frame.

[0044] Then, the detection framework is used to locate the position of the instrument dial in the instrument image, and features are extracted for pointer instruments with uniform or non-uniform scales. The extracted features are shared with subsequent networks to facilitate dial segmentation. The Hough circle detection method is used to extract the center of the dial.

[0045] Step S106: The instrument dial is divided into elements using an anti-tilt non-uniform scale segmentation scheme.

[0046] Optionally, an anti-tilt non-uniform scale instrument dial segmentation scheme is adopted to segment the elements in the instrument dial, including:

[0047] 1) Tilting correction of the instrument panel based on improved Hough transform: edge detection based on Sobel operator, tilt angle calculation based on Hough transform, image rotation based on imrotate function, image restoration based on Hough transform, setting the gray value of all pixels on the straight line to 0, and correcting the tilt angle of the straight line.

[0048] 2) The instrument panel is segmented based on the improved deepLabv3+ model, wherein the improved deepLabv3+ model uses a ResNet50 feature extraction network, and deformable convolution is introduced on the ResNet50 feature extraction network to realize the perception of contextual information.

[0049] 3) Perform high-resolution hollow feature pyramid pooling on the instrument panel in the improved deepLabv3+ model: replace the 16× downsampling layer of the deepLabv3+ model with an 8× downsampling layer, and perform high-resolution hollow feature pyramid pooling on the sampled features of the 8× downsampling layer to improve the resolution of the encoder features and facilitate subsequent decoding.

[0050] 4) Using a deformable convolution decoder based on the improved deepLabv3+ model to segment the elements in the instrument panel: Features in the instrument panel are extracted using a ResNet network, and the features extracted by the ResNet network are upsampled using an HR-ASPP network. The upsampled features are then subjected to two deformable convolution upsampling operations to obtain region context pixel aggregation features. These region context pixel aggregation features are upsampled according to a specified resolution and then combined to achieve the fusion of local and context features, thus outputting the segmentation result.

[0051] Step S108: Automatically read the instrument dial using at least one of the character numerical method and the angle method.

[0052] Optionally, the automatic reading of the instrument dial is performed using at least one of the character numerical method and the angle method, including at least one of the following:

[0053] 1) The image samples are segmented into characters using the character value method to obtain character images. The image samples are obtained by image acquisition from the instrument. The original model of the multilayer perceptron is trained using the character images to obtain a multilayer perceptron for recognizing keywords in the instrument. The instrument range and scale character values ​​on the instrument dial are recognized by the multilayer perceptron to complete the automatic reading.

[0054] 2) Use the angle method to convert the readings on the instrument dial to complete automatic reading: Determine the angles α and β between the center line of the pointer on the instrument dial and the values ​​of two adjacent scales, and calculate the actual value pointed to by the pointer through the linear relationship of the angles to complete automatic reading;

[0055] 3) For non-uniform pointer-type instruments, the instrument dial is automatically read using a combination of character numerical method and angle method.

[0056] Through the above steps, an image of the target instrument is acquired, wherein the target instrument is a pointer-type instrument whose reading is to be identified; using a machine vision-based detection framework, the instrument dial and its center are located from the instrument image; an anti-tilt non-uniform scale instrument dial segmentation scheme is adopted to segment each element in the instrument dial; at least one of the character numerical method and the angle method is used to automatically read the instrument dial. By employing artificial intelligence and adopting an anti-tilt solution to address the poor environment in which the instrument is located, the technical problem of low efficiency in manually reading instruments in related technologies can be solved.

[0057] As an optional implementation method, the basic process of this application embodiment is as follows: Figure 2As shown, firstly, based on the detection framework of automatic instrument reading, the instrument dial and center are initially positioned; secondly, the instrument segmentation algorithm with anti-tilt non-uniform scale is constructed by sharing the features of the detection algorithm to realize the segmentation of the pointer; then, the instrument automatic reading is performed by combining the character numerical value and the angle method.

[0058] Step S1: Establish the detection framework for automatic instrument reading.

[0059] When establishing the detection framework, considering the influence of factors such as the production environment and lighting conditions of the pointer instrument, resulting in low image quality, an image data preprocessing scheme involving color gamut transformation, geometric transformation, and contrast and saturation enhancement was adopted to provide clear and easily processed image data for subsequent detection, localization, and segmentation algorithms. After preprocessing the acquired instrument image data, a Faster R-CNN object detection network with ResNet50 (where 50 refers to 50 layers, and ResNet is divided into 5 stages: Stage 0 to Stage 4. Stage 0 has a relatively simple structure and can be considered as preprocessing of the input; the latter four stages are composed of Bottlenecks with similar structures; Stage 1 contains 3 Bottlenecks, and the remaining 3 stages contain 4, 6, and 3 Bottlenecks respectively) as the feature extraction module was used to detect and locate the dial. Figure 3 As shown, the Hough circle is then used to determine the center coordinates of the positioning dial, as follows: Figure 4 As shown.

[0060] Faster R-CNN consists of the following parts: 1) Dataset image input; 2) Basic networks such as convolutional layers (CNN) to extract features and obtain feature maps; 3) RPN layer, which uses a 3×3 slide window to traverse the feature map extracted by the convolutional layers. During the traversal, 9 anchors are generated at the center of each window according to the rate and scale (1:2, 1:1, 2:1). Then, a fully connected layer is used to perform binary classification and preliminary bounding box regression on each anchor, finally outputting 300 relatively accurate ROIs; 4) The feature map after the convolutional layers is fixed by ROI pooling to fix the input dimension of the fully connected layer; 5) The ROIs output by the RPN are then mapped onto the feature map of ROI pooling for bounding box regression and classification.

[0061] Step S11, Data Preprocessing: Related instrument detection algorithms use horizontal flipping to preprocess and enhance the dataset, ignoring the impact of dataset labeling errors. However, the low image quality, influenced by factors such as the production environment and lighting, severely affects detection accuracy. Therefore, this application directly removes horizontal flipping and adds data preprocessing techniques such as color gamut transformation, geometric transformation, and contrast enhancement. Specifically, it randomly combines these preprocessing methods on the original training samples to improve the quality of the instrument images, thereby enhancing detection robustness.

[0062] Step S12, Detection framework for automatic reading of pointer-type instruments: Since the dial segmentation algorithm requires a clear image containing all information of the dial, a detection and positioning framework for automatic reading of pointer-type instruments is introduced to realize the position detection of the object dial, such as... Figure 5 As shown, features are extracted from both uniformly and non-uniformly graduated pointer-type instruments, and these extracted features are shared with the subsequent segmentation network for dial segmentation. Simultaneously, Hough circle detection is used to extract the center of the dial, i.e., the pole of polar coordinates, to enable subsequent automatic reading.

[0063] The proposed automatic instrument reading detection framework enables the dial detector and the instrument segmentation network to share backbone features, improving detection and segmentation efficiency while simultaneously performing multiple tasks. The framework consists of training and testing phases. During the training phase, the backbone features of the instrument dial detector are extracted, and two regressions significantly improve training efficiency. Furthermore, this framework is universally applicable, allowing the addition of other target detectors and easily expanding to other detection categories, such as the detection of dial character regions.

[0064] Step S2, design of instrument segmentation algorithm for anti-tilting non-uniform scale.

[0065] After visual preprocessing of the instrument image and localization of the dial and center, an anti-tilt non-uniform scale instrument dial segmentation algorithm is designed. Hough line detection corrects the tilt of the detected dial image. Based on this, a high-resolution hollow feature pyramid and a deformable convolution decoder are designed to fuse low-level detail and contextual information, while simultaneously changing the receptive field of the decoder to identify the deformation caused by the tilt correction. Finally, an angle method is introduced to combine the numerical characters and pointer angles to achieve automatic reading of non-uniform scale instruments.

[0066] Step S21, dial tilt correction based on improved Hough transform: Utilizing the duality of points and lines, lines in the instrument image space are transformed into cluster points in the parameter space, thereby detecting whether a curve with a given property exists in a given image. The number of collinear points in the instrument image space is detected by counting the number of sine curves passing through a certain point in the parameter space. The counts of all curves passing through a point are accumulated in an array; finally, the value with the largest count in the array is found, and this value is the dial tilt angle.

[0067] The dial tilt correction process based on the improved Hough transform is as follows: 1) Sobel operator edge detection; 2) Hough transform to calculate the tilt angle; 3) Image rotation correction; 4) Hough transform to detect all straight lines; 5) Set the gray value of all pixels on the straight line to 0, and correct the tilt angle of the straight line.

[0068] The most crucial aspect is pointer detection. While related technologies employ simple line detection, this application improves pointer detection accuracy through two detection steps. The Hough transform uses any point (a, b) in the shape to be detected as a reference point. Then, from each point on the edge of this arbitrary shape, the tangent direction and the offset vector r to the reference point (a, b), as well as the angle between r and the x-axis, are calculated. The position of the reference point can be calculated using equations (1) and (2).

[0069]

[0070]

[0071] For a straight line segment, only its endpoints (a1, b1) and (a2, b2) need to be detected. This improved algorithm corrects the instrument tilt by combining the angle between the line segment and the x-axis, and then obtains the corrected pointer position through a straight line detection.

[0072] Step S22: Based on the improved deepLabv3+ dial segmentation framework, deepLabv3+ replaces the max pooling layer with depthwise separable convolutions in the Xecption feature extraction network, reducing the number of parameters and computational cost. However, because its depthwise separable convolutions lack contextual information, their ability to perceive deformation in dial images is poor. Therefore, deformable convolutions are introduced on top of the ResNet50 feature extraction network to achieve contextual information perception. This feature network is shared with the aforementioned dial detection and localization framework.

[0073] To address the issues of poor deformation perception and low segmentation precision in dial segmentation algorithms, this application proposes a high-resolution dilated feature pyramid and a decoder module based on deformable convolution on deepLabV3+.

[0074] Step S23, High-Resolution Hollow Feature Pyramid Pooling: In deepLabV3+, hollow spatial pyramid pooling is used for images downsampled by 16× (equivalent to downsampling 16 pixels into 1 pixel). The high downsampling ratio leads to the loss of dial features, making recovery impossible. Therefore, an improved high-resolution hollow feature pyramid pooling is proposed, uniformly performing pyramid pooling on 8× downsampled features (equivalent to downsampling 8 pixels into 1 pixel), improving the resolution of encoder features and facilitating subsequent decoding.

[0075] Step S24, Deformable Convolution-Based Decoder: Since different shooting angles cause deformation, increasing the difficulty of dashboard segmentation, ResNet first extracts features, then HR-ASPP processes the features to obtain 8× upsampled (equivalent to upsampling 1 pixel to 8 pixels) features F. 8× To obtain a clear segmented dial image, the features need to be upsampled. Based on this, the features F... 8× Perform two deformable convolution upsampling operations to obtain the region context pixel aggregation F. 4× and F 8× Features. Finally, F 4×4 and F 8×8 Upsampled to F 2×2 The resolution is optimized by combining local and contextual features and performing convolutional upsampling to output the segmentation results. This fusion-based deformable convolution decoder has stronger resistance to deformation. The improved deepLabv3+ dial segmentation framework has achieved good results in both accuracy and real-time performance, meeting the actual needs of industrial production, as shown in Table 1 below.

[0076] Table 1

[0077] Model / Algorithm backbone network accuracy FPS Connected component analysis / 84.6％ 25 DeepLabv3+ ResNet50 94.2％ 21 HR-DeepLabv3+ ResNet50 97.3％ 37

[0078] Step S3: Automatic reading of the instrument using the combined character numerical value and angle method.

[0079] Step S31, Instrument range and scale character numerical recognition based on multilayer perceptron: To achieve automatic reading, it is necessary to use the scale indication and range indication on the dial. This application segments the characters of 500 (or more) images containing pointer-type instruments, selects 2000 (or more) digit images as a dataset, and trains the multilayer perceptron model. The trained multilayer perceptron can accurately read key characters in the instrument, such as range and scale indication, for example... Figure 6 As shown.

[0080] Step S32, reading conversion of pointer instrument based on angle method: This application uses the angle method to read pointer instruments. Based on the center of the circle, a plane rectangular coordinate system is established. The center of the pointer area pixel is the pointer centerline. The angles α and β between the pointer centerline and two adjacent scale values ​​are calculated through the linear relationship of angles to obtain the actual pointer value.

[0081] Step S33, Automatic Reading of Instruments Combining Character Value Recognition and Angle Method: For non-uniform pointer instruments, conventional angle method recognition cannot achieve reading; therefore, automatic reading is achieved by combining the instrument's range and scale character values. The point farthest from the center of the pointer area pixel is designated as point B, and line segment OB is the pointer's center line; the nearest counter-clockwise scale value of line segment OB is designated as d, and the centroid of the connected region corresponding to scale value d is designated as point A; the nearest clockwise scale value of line segment OB is designated as e, and the centroid of the connected region corresponding to scale value e is designated as point C; the angle between OA and OB is designated as α, and the angle between OA and OC is designated as β. For example... Figure 7 As shown, the reading result of the pointer instrument is as shown in formula (3):

[0082]

[0083] By applying the technical solution of this invention, the segmentation algorithm can be improved. Two improvement strategies are proposed to address the low precision, poor anti-tilt performance, and low pointer scale segmentation accuracy of related segmentation algorithms. High Resolution Atrous Spatial Pyramid pooling (HR-ASPP) and a deformable convolutional decoder (de-decoder) are proposed to effectively extract key information such as pointers, scales, and values ​​from the dial. Finally, by combining the dial numerical characters recognized by a Multilayer Perceptron (MLP) with the segmented pointer polar coordinate angles, automatic reading of non-uniform scale pointer instruments is achieved. This ensures accurate and efficient reading of pointer instruments.

[0084] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0085] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0086] According to another aspect of the embodiments of this application, a machine vision-based pointer instrument reading device is also provided for implementing the above-described machine vision-based pointer instrument reading method. Figure 8 This is a schematic diagram of an optional machine vision-based pointer-type instrument reading device according to an embodiment of this application, such as... Figure 8 As shown, the device may include:

[0087] The acquisition unit 81 is used to acquire the instrument image of the target instrument, wherein the target instrument is a pointer-type instrument whose reading is to be identified; the positioning unit 83 is used to locate the instrument dial and the dial center from the instrument image using a machine vision-based detection framework; the segmentation unit 85 is used to segment each element in the instrument dial using an anti-tilt non-uniform scale instrument dial segmentation scheme; and the reading unit 87 is used to automatically read the instrument dial using at least one of the character numerical method and the angle method.

[0088] Optionally, the localization unit is further configured to: preprocess the original sample to obtain an enhanced sample before locating the instrument dial and dial center from the instrument image; the preprocessing includes one or a combination of randomly selected one of color gamut transformation, geometric transformation, contrast enhancement, and saturation enhancement; the preprocessing is used to eliminate the influence of the production environment and lighting on the image; the random selection is used to improve the robustness of detection; the original sample is an image obtained by image acquisition of the instrument; the enhanced sample is marked with the dial position and dial center; the enhanced sample is used for training to obtain the detection framework. Then, the detection framework is used to locate the position of the instrument dial in the instrument image, and features are extracted for pointer instruments with uniform or non-uniform scales; the extracted features are shared with subsequent networks to facilitate dial segmentation; the Hough circle detection method is used to extract the dial center.

[0089] Optionally, the segmentation unit is also used for: 1) tilt correction of the instrument panel based on the improved Hough transform: edge detection based on the Sobel operator, tilt angle calculation based on the Hough transform, image rotation based on the imrotate function, image restoration based on the Hough transform, setting the grayscale value of all pixels on the straight line to 0, and correcting the tilt angle of the straight line; 2) segmenting the instrument panel based on the improved deepLabv3+ model, wherein the improved deepLabv3+ model uses a ResNet50 feature extraction network, and deformable convolution is introduced into the ResNet50 feature extraction network to realize the perception of contextual information; 3) performing high-resolution hollow feature pyramid pooling on the instrument panel based on the improved deepLabv3+ model: the deepLabv3+ model The 16× downsampling layer is replaced with an 8× downsampling layer. High-resolution dilated feature pyramid pooling is performed on the sampled features of the 8× downsampling layer to improve the resolution of the encoder features and facilitate subsequent decoding. 4) The elements in the instrument panel are segmented using a deformable convolution decoder based on the improved deepLabv3+ model: Features in the instrument panel are extracted using a ResNet network. The features extracted by the ResNet network are upsampled using an HR-ASPP network. The upsampled features are then subjected to two deformable convolution upsampling operations to obtain the region context pixel aggregation features. The region context pixel aggregation features are upsampled according to a specified resolution and then combined to achieve the fusion of local and context features and output the segmentation results.

[0090] Optionally, the reading unit is also used to: 1) segment the image sample using the character value method to obtain a character image, wherein the image sample is a sample obtained by image acquisition of the instrument; train the original model of the multilayer perceptron using the character image to obtain a multilayer perceptron for recognizing keywords in the instrument; and use the multilayer perceptron to recognize the instrument range and scale character values ​​on the instrument dial to complete automatic reading; 2) convert the reading of the instrument dial using the angle method to complete automatic reading: determine the angles α and β between the center line of the pointer on the instrument dial and two adjacent scale values, calculate the actual value pointed to by the pointer through the linear relationship of the angles, and complete automatic reading; 3) for non-uniform pointer instruments, combine the character value method and the angle method to automatically read the instrument dial.

[0091] For the preprocessed dial image, a two-stage object detection method is used to locate and detect the dial and its center. To improve the segmentation accuracy of the dial pointer, scale, and scale characters, an improved deepLabV3+ dial segmentation framework is proposed to correct the dial's tilt. High-resolution high-resolution hollow feature pyramid pooling (HR-ASPP) and a deformable decoder (Deform-Decoder) are fused to segment the dial more accurately. Finally, the dial character values ​​and polar coordinate angles are combined to analyze the pointer readings for non-uniform scales. This invention can be applied to scenarios with both uniform and non-uniform pointer scales on multiple instruments, demonstrating good accuracy and applicability.

[0092] The above scheme involves acquiring an image of a target instrument, which is a pointer-type instrument whose readings are to be identified; using a machine vision-based detection framework, locating the instrument dial and its center from the image; employing an anti-tilt, non-uniform scale instrument dial segmentation scheme to segment the elements within the instrument dial; and using at least one of the character numerical method and the angle method to automatically read the instrument dial. By employing artificial intelligence and an anti-tilt solution tailored to the harsh environment in which the instrument is located, the inefficiency of manual instrument reading in related technologies can be solved.

[0093] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0094] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0095] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0096] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0097] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0098] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0099] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0100] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0101] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A machine vision based method of reading a pointer type instrument, characterized by, The method comprises the following steps: Collecting an instrument image of a target instrument, wherein the target instrument is a pointer instrument with a reading to be recognized; Using a machine vision-based detection framework to locate an instrument dial and a dial center from the instrument image; Segmenting each element in the instrument dial using an anti-tilt non-uniform scale instrument dial segmentation scheme; Automatically reading the instrument dial using at least one of a character numerical method and an angle method; Segmenting each element in the instrument dial using an anti-tilt non-uniform scale instrument dial segmentation scheme, comprising: Correcting the tilt of the instrument dial based on an improved Hough transform: edge detection based on a Sobel operator, tilt angle calculation based on a Hough transform, image rotation based on an imrotate function, image restoration based on a Hough transform, setting all pixel gray values on a straight line to 0, and correcting the tilt angle of the straight line; Segmenting the instrument dial based on an improved deepLabv3+ model, wherein a ResNet50 feature extraction network is used in the improved deepLabv3+ model, and a deformable convolution is introduced on the ResNet50 feature extraction network to realize context information perception; Performing high-resolution hollow feature pyramid pooling on the instrument dial in the improved deepLabv3+ model: replacing the 16x downsampling layer of the deepLabv3+ model with an 8x downsampling layer, and performing high-resolution hollow feature pyramid pooling on the sampling features of the 8x downsampling layer to improve the resolution of the encoder features and facilitate subsequent decoding; Segmenting each element in the instrument dial using a decoder with deformable convolution in the improved deepLabv3+ model: extracting features in the instrument dial using a ResNet network, performing up-sampling processing on the features extracted by the ResNet network using an HR-ASPP network, obtaining up-sampled features, performing deformable convolution up-sampling twice on the up-sampled features obtained by the HR-ASPP network, obtaining regional context pixel aggregation features, up-sampling the regional context pixel aggregation features according to a specified resolution, and combining them to realize the fusion of local and context features and output the segmentation result.

2. The method of claim 1, wherein, Before using the machine vision-based detection framework to locate the instrument dial and the dial center from the instrument image, the method further comprises: Preprocessing the original sample to obtain an enhanced sample, wherein the preprocessing includes randomly selecting one or a combination of multiple randomly selected ones from color gamut transformation, geometric transformation, contrast enhancement, and saturation enhancement, and the preprocessing is used to eliminate the influence of the production site environment and light on the image, and the random selection is used to improve the robustness of the detection, the original sample is an image obtained by image acquisition of an instrument, and the enhanced sample is labeled with a dial position and a dial center; Training using the enhanced sample to obtain the detection framework.

3. The method according to claim 1 or 2, characterized in that, Using the machine vision-based detection framework to locate the instrument dial and the dial center from the instrument image comprises: The detection framework is used to locate a position of a dial plate of the instrument image and to extract features of a uniformly scaled or non-uniformly scaled pointer instrument, wherein the extracted features are shared to a subsequent network so as to perform dial plate segmentation by the subsequent network; A Hough circle detection method is used to extract the dial plate center of the dial plate.

4. The method of claim 1, wherein, Automatic reading of the instrument dial plate is performed by using at least one of a character value method and an angle method, including at least one of: The character value method is used to identify an instrument range and scale character value of the instrument dial plate by using a multi-layer perception machine, so as to complete automatic reading; The angle method is used to convert a reading of the instrument dial plate, so as to complete automatic reading; For a non-uniform pointer instrument, the character value method and the angle method are combined to perform automatic reading of the instrument dial plate.

5. The method of claim 4, wherein, The angle method is used to convert a reading of the instrument dial plate, so as to complete automatic reading, including: An angle α and an angle β between a pointer center line and adjacent two scale values in the instrument dial plate are determined; An actual value pointed by the pointer is calculated by using a linear relationship of the angle, so as to complete automatic reading.

6. The method of claim 4, wherein, Before the character value method is used to identify an instrument range and scale character value of the instrument dial plate by using a multi-layer perception machine, so as to complete automatic reading, the method further includes: A character segmentation is performed on an image sample to obtain a character image, wherein the image sample is a sample obtained by image acquisition of an instrument; The character image is used to train an original model of the multi-layer perception machine, so as to obtain the multi-layer perception machine used for identifying a key word in the instrument.

7. A machine vision based pointer type instrument reading device, characterized by The method includes: An acquisition unit is configured to acquire an instrument image of a target instrument, wherein the target instrument is a pointer instrument whose reading is to be identified; A positioning unit is configured to locate a dial plate and a dial plate center from the instrument image by using a machine vision-based detection framework. The segmentation unit is used to segment each element in the instrument dial by using an anti-inclination non-uniform scale instrument dial segmentation scheme, and the anti-inclination non-uniform scale instrument dial segmentation scheme includes: inclination correction of the instrument dial based on improved Hough transformation; edge detection based on a sobel operator, inclination angle calculation based on Hough transformation, image rotation based on an imrotate function, image restoration based on Hough transformation, setting all pixel gray values on a straight line to 0, and correcting the inclination angle of the straight line; segmentation of the instrument dial based on an improved deepLabv3+ model, wherein a ResNet50 feature extraction network is used in the improved deepLabv3+ model, and a deformable convolution is introduced into the ResNet50 feature extraction network to realize perception of context information; high-resolution hollow feature pyramid pooling of the instrument dial based on the improved deepLabv3+ model: replacing a 16x downsampling layer of the deepLabv3+ model with an 8x downsampling layer, and performing high-resolution hollow feature pyramid pooling on the sampling features of the 8x downsampling layer to improve the resolution of the encoder features and facilitate subsequent decoding; segmentation of each element in the instrument dial by using a decoder of the improved deepLabv3+ model: extracting features in the instrument dial by using a ResNet network, performing up-sampling processing on the features extracted by the ResNet network by using an HR-ASPP network, obtaining up-sampled features, performing twice deformable convolution up-sampling on the up-sampled features obtained by the HR-ASPP network, obtaining regional context pixel aggregation features, up-sampling the regional context pixel aggregation features according to a specified resolution, and combining them to realize fusion of local and context features and output of the segmentation result. The reading unit is used to automatically read the instrument dial by using at least one of a character numerical method and an angle method.

8. A computer readable storage medium, characterized in that, The storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 6 when running.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the method described in any one of claims 1 to 6 by using the computer program.

Images