An Automatic Reading Method for Pointer Instruments Based on Semantic Segmentation

Abstract

This application discloses an automatic reading method for pointer-type instruments based on semantic segmentation. The method includes: acquiring the original image of the instrument; constructing a DeepLabV3 semantic segmentation model, inputting the original image into the trained DeepLabV3 semantic segmentation model to obtain segmented pointer pixel regions and scale pixel regions; calculating the angle of the pointer and the angle of each scale, and determining which two adjacent scales the pointer falls on based on the pointer angle and the angle of each scale to obtain the pointer reading. The pointer and scale angle calculation method provided by this invention adopts a directional approach, making the angle calculation unique and highly accurate.

Description

Technical Field

[0001] This application relates to the field of intelligent inspection technology, and more specifically, to a method for automatic reading of pointer-type instruments based on semantic segmentation. Background Technology

[0002] With the development of unmanned substations, research on automatic identification and reading systems for pointer-type instruments is essential. Pointer-type instruments, with their advantages of low cost and simple structure, are widely used in many fields such as power systems, transportation, and aerospace. However, due to the lack of a digital interface, manual reading is required. In complex industrial environments, manual reading is inefficient and poses safety hazards. Therefore, developing an automatic identification and reading system for pointer-type instruments is crucial.

[0003] Traditional image processing algorithms currently suffer from poor robustness, exhibiting low recognition rates when faced with issues such as uneven brightness, image blur, occlusion, and varying distances. Furthermore, existing angle and distance methods are ineffective for reading non-uniform scale instruments. To overcome these shortcomings, this patent proposes an automatic reading method for pointer-type instruments based on DeepLabV3 semantic segmentation. This method provides accurate and stable readings in various substations and environments, and is applicable to both uniform and non-uniform scale instruments. Summary of the Invention

[0004] To address at least one deficiency or improvement requirement of the existing technology, the present invention provides an automatic reading method for pointer-type instruments based on semantic segmentation. By employing a directional method, the angle calculation is unique and has high accuracy.

[0005] To achieve the above objectives, according to a first aspect of the present invention, a method for automatic reading of pointer-type instruments based on semantic segmentation is provided, the method comprising:

[0006] Acquire the raw image of the instrument;

[0007] Construct a DeepLabV3 semantic segmentation model by inputting the original image into the trained DeepLabV3 semantic segmentation model to obtain the segmented pointer pixel region and tick pixel region;

[0008] Calculate the angle of the pointer and the angle of each scale mark separately. Based on the angle of the pointer and the angle of each scale mark, determine which two adjacent scale marks the pointer falls between, and obtain the pointer reading.

[0009] Furthermore, in the above-mentioned automatic reading method for pointer-type instruments based on semantic segmentation, the DeepLabV3 semantic segmentation model includes a deep convolutional network, and the upsampling channel of the deep convolutional network includes four convolutional kernels and pooling layers, one of which has a size of 1×1 and the remaining convolutional kernels have a size of 3×3.

[0010] The original image is subjected to four layers of convolution and pooling operations. The feature maps obtained after the convolution operation and the feature maps obtained after the pooling operation are merged, and then an upsampled feature map is obtained by performing a 1×1 convolution kernel operation.

[0011] Furthermore, in the above-mentioned automatic reading method for pointer-type instruments based on semantic segmentation, the downsampling channel of the deep convolutional network includes a convolutional kernel of size 1×1;

[0012] The downsampled feature map, which has undergone a 1×1 convolution kernel operation, is concatenated with the upsampled feature map, and then subjected to a 3×3 convolution kernel operation and upsampling to obtain the semantic segmentation image.

[0013] Furthermore, in the above-mentioned automatic reading method for pointer-type instruments based on semantic segmentation, the category of each pixel in the semantically segmented image is determined to obtain the category corresponding to each pixel.

[0014] Furthermore, in the above-mentioned automatic reading method for pointer-type instruments based on semantic segmentation, the training process of the DeepLabV3 semantic segmentation model is as follows:

[0015] Obtain several original images of the instruments as training sample images;

[0016] The sample image is input into the DeepLabV3 semantic segmentation model to obtain the predicted classification result of the sample image;

[0017] The predicted classification results and the true labels are iteratively trained based on the target loss function to obtain the trained DeepLabV3 semantic segmentation model.

[0018] Furthermore, in the above-mentioned automatic reading method for pointer-type instruments based on semantic segmentation, the calculation of the pointer angle specifically includes:

[0019] Obtain the pointer pixel region image and establish a Cartesian coordinate system;

[0020] When the pointer is recognized as a rectangle, take the midpoints of the two shorter sides of the rectangle and denote them as point P and point Q, where point P is the endpoint closer to the pointer tip and point Q is the endpoint closer to the pointer tail.

[0021] Obtain the coordinates of points P and Q, calculate the angle from point Q to point P based on the coordinates of points P and Q, and use the angle from point Q to point P as the final angle of the pointer.

[0022] Furthermore, in the above-mentioned automatic reading method for pointer-type instruments based on semantic segmentation, the calculation of the pointer angle further includes:

[0023] When the pointer image is recognized as a triangle, take the midpoint of the shorter side of the triangle and denote it as point Q. The endpoint opposite the shorter side is called point P. Point P is the endpoint closer to the pointer tip, and point Q is the endpoint closer to the pointer tail.

[0024] Obtain the coordinates of points P and Q, calculate the angle from point Q to point P based on the coordinates of points P and Q, and use the angle from point Q to point P as the final angle of the pointer.

[0025] Furthermore, in the above-mentioned automatic reading method for pointer-type instruments based on semantic segmentation, the calculation of the pointer angle further includes:

[0026] When the pointer is recognized as a line segment, take the two endpoints of the line segment and denote them as point P and point Q, where point P is the endpoint closer to the pointer tip and point Q is the endpoint closer to the pointer tail.

[0027] Obtain the coordinates of points P and Q, calculate the angle from point Q to point P based on the coordinates of points P and Q, and use the angle from point Q to point P as the final angle of the pointer.

[0028] Furthermore, in the above-mentioned automatic reading method for pointer-type instruments based on semantic segmentation, the angle of each scale mark further includes:

[0029] Obtain the pointer pixel region image and establish a Cartesian coordinate system;

[0030] When the scale is recognized as a line segment, take the two endpoints of the line segment and record them as point P and point Q, where point P is the endpoint far from the center of the image and point Q is the endpoint close to the center of the image.

[0031] Obtain the coordinates of points P and Q, calculate the angle from point Q to point P based on the coordinates of points P and Q, and use the angle from point Q to point P as the scale angle.

[0032] Furthermore, the above-mentioned automatic reading method for pointer-type instruments based on semantic segmentation, wherein determining which two adjacent scales the pointer falls on based on the angle of the pointer and the angle of each scale, and obtaining the pointer reading, specifically includes:

[0033] Determine which two adjacent marks the pointer falls between based on the angle of the pointer and the angle of each mark;

[0034] Calculate the angle difference between the pointer and the previous moment, and the angle difference between two adjacent scale divisions. Based on these two angle differences, calculate the pointer reading, as shown in the following formula:

[0035]

[0036] Where, α iα represents the angle of the previous scale division. i+1 s represents the angle of the next tick mark. i Let s be the angle of the pointer at the previous moment. i+1 The pointer is pointing to the current angle.

[0037] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0038] (1) The present invention provides an automatic reading method for pointer instruments based on semantic segmentation. By calculating the angle of the pointer and the angle of each scale respectively, the pointer is determined to fall between two adjacent scales based on the angle of the pointer and the angle of each scale, and the pointer reading is obtained. The directional method makes the angle calculation unique and has high accuracy.

[0039] (2) The present invention provides an automatic reading method for pointer instruments based on semantic segmentation, which converts the angle between the pointer and the scale into a reading by means of interval angle method. It is not subject to other constraints and is applicable to both uniform and non-uniform scale instruments.

[0040] (3) The present invention provides an automatic reading method for pointer instruments based on semantic segmentation. It uses the DeepLabV3 semantic segmentation model trained with a large dataset to predict weights, which can extract the feature regions of pointer and scale more stably and has higher robustness. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 A flowchart illustrating an automatic reading method for pointer-type instruments based on semantic segmentation, provided in an embodiment of this application;

[0043] Figure 2 This is a structural diagram of the DeepLabV3 semantic segmentation network model provided in the embodiments of this application;

[0044] Figure 3 This is a semantic segmentation effect diagram provided for an embodiment of this application;

[0045] Figure 4 A flowchart of a pointer reading method provided in an embodiment of this application;

[0046] Figure 5 The diagram shows the effect of judging the pointer scale angle provided in the embodiment of this application. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0048] The terms "first," "second," "third," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0049] This application provides a method for automatic reading of pointer-type instruments based on semantic segmentation. Figure 1 This is a flowchart illustrating an automatic reading method for pointer-type instruments based on semantic segmentation, provided as an embodiment of this application. Please refer to... Figure 1 The method includes the following steps:

[0050] (1) Obtain the original image of the instrument;

[0051] (2) Construct a DeepLabV3 semantic segmentation model. Input the original image into the trained DeepLabV3 semantic segmentation model to obtain the segmented pointer pixel region and tick pixel region.

[0052] The DeepLabV3 model structure constructed in this application adds dilated convolution operations to the SPP layer, which makes the network's feature extraction have a larger receptive field. After a series of downsampling operations, it is fused with the feature layer in the feature extraction network to obtain the prediction layer. After upsampling to the same size as the input image, the category of each pixel is judged. The pointer pixel region and the tick pixel region can be obtained by determining the category corresponding to each pixel.

[0053] Specifically, Figure 2This is a structural diagram of the DeepLabV3 semantic segmentation network model provided in this application embodiment. The DeepLabV3 semantic segmentation model includes a deep convolutional network. The upsampling channel of the deep convolutional network includes four convolutional kernels and pooling layers. One of the convolutional kernels has a size of 1×1, and the remaining convolutional kernels have a size of 3×3. The original image is subjected to four convolutional operations and pooling operations respectively. The feature map obtained after the convolution operation is merged with the feature map obtained after the pooling operation, and then the upsampled feature map is obtained by performing a 1×1 convolutional kernel operation.

[0054] The downsampling channel of a deep convolutional network includes a 1×1 convolutional kernel; the downsampling feature map after the 1×1 convolutional kernel operation is concatenated with the upsampling feature map, a 3×3 convolutional kernel operation is performed, and an upsampling operation is performed to obtain the semantic segmentation image. Figure 3 The semantic segmentation effect diagram provided in the embodiment of this application determines the category of each pixel in the semantic segmentation image, namely the pointer pixel region and the scale pixel region.

[0055] The training process of the DeepLabV3 semantic segmentation model is as follows: Several original images of instruments are acquired as training sample images. These training sample images include at least pointer-type instruments such as ammeters, voltmeters, and barometers, and further include both uniformly graduated and non-uniformly graduated instruments. The sample images are input into the DeepLabV3 semantic segmentation model to obtain the predicted classification results. The predicted classification results and ground truth labels (needle pixel regions and scale pixel regions) are then iteratively trained based on the target loss function to obtain the trained DeepLabV3 semantic segmentation model.

[0056] (3) Figure 4 The flowchart of the pointer reading method provided in this application embodiment calculates the pointer angle and the angle of each scale, determines which two adjacent scales the pointer falls on based on the pointer angle and the angle of each scale, and obtains the pointer reading.

[0057] The pointer pixel region and the scale pixel region have been segmented. Each segmented region can form a connected component, and the scale has multiple connected components. By calculating the angle of each connected component, we can obtain the rotation angle of the pointer and the angle of the scale.

[0058] Specifically, the pointer pixel region image is acquired, and a Cartesian coordinate system is established;

[0059] Figure 5The diagram shows the effect of judging the pointer scale angle provided in the embodiment of this application. When the pointer is recognized as a rectangle, the midpoints of the two shorter sides of the rectangle are taken and recorded as point P and point Q, respectively. Point P is the endpoint closer to the pointer tip, and point Q is the endpoint closer to the pointer tail.

[0060] Obtain the coordinates of points P and Q, and calculate the angle from point Q to point P based on these coordinates. This angle ranges from 0° to 360°. A diagonal line may have two angles, meaning the angle from point P to point Q and the angle from point Q to point P are different. Therefore, it is stipulated here that the angle from point Q to point P is used as the final angle of the pointer.

[0061] When the pointer image is recognized as a triangle, take the midpoint of the shorter side of the triangle and denote it as point Q. The endpoint opposite the shorter side is called point P. Point P is the endpoint closer to the pointer tip, and point Q is the endpoint closer to the pointer tail.

[0062] Obtain the coordinates of points P and Q. Calculate the angle from point Q to point P based on these coordinates; this angle ranges from 0° to 360°. Use this angle from point Q to point P as the final angle of the pointer.

[0063] When the pointer is recognized as a line segment, take the two endpoints of the line segment and denote them as point P and point Q, where point P is the endpoint closer to the pointer tip and point Q is the endpoint closer to the pointer tail.

[0064] Obtain the coordinates of points P and Q. Calculate the angle from point Q to point P based on these coordinates; this angle ranges from 0° to 360°. Use this angle from point Q to point P as the final angle of the pointer.

[0065] Furthermore, the pointer pixel region image is acquired, and a Cartesian coordinate system is established;

[0066] When the scale is recognized as a line segment (since the scale lines are relatively small, they are generally recognized as line segments), take the two endpoints of the line segment and record them as point P and point Q, where point P is the endpoint far from the center of the image and point Q is the endpoint close to the center of the image.

[0067] Obtain the coordinates of points P and Q. Calculate the angle from point Q to point P based on these coordinates. This angle ranges from 0° to 360°. Use the angle from point Q to point P as the scale angle.

[0068] The angle from point Q to point P is calculated using the following formula:

[0069]

[0070] The range of θ is (0°, 360°).

[0071] Furthermore, based on the interval angle method, the pointer is determined between two adjacent scales by comparing the angle of the pointer with the angle of each scale, and the pointer reading is obtained.

[0072] Specifically, the pointer is positioned between two adjacent scale marks based on its angle and the angle of each scale mark; the angle difference between the pointer and the previous moment and the angle difference between two adjacent scale marks are calculated, and the pointer reading is calculated based on these two angle differences, as shown in the following formula:

[0073]

[0074] Where, α i α represents the angle of the previous scale division. i+1 s represents the angle of the next tick mark. i Let s be the angle of the pointer at the previous moment. i+1 The pointer is pointing to the current angle.

[0075] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0076] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0077] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of embodiments of this disclosure upon considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described herein. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

[0078] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0079] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for automatically reading pointer-type instruments based on semantic segmentation, characterized in that, include: Acquire the raw image of the instrument; Construct a DeepLabV3 semantic segmentation model by inputting the original image into the trained DeepLabV3 semantic segmentation model to obtain the segmented pointer pixel region and tick pixel region; The calculation of the pointer angle specifically includes: acquiring the pointer pixel area image and establishing a Cartesian coordinate system; when the pointer is recognized as a rectangle, taking the midpoints of the two shorter sides of the rectangle, denoted as point P and point Q, where point P is the endpoint closer to the pointer tip and point Q is the endpoint closer to the pointer tail; acquiring the coordinates of point P and point Q, calculating the angle from point Q to point P based on the coordinates of point P and point Q, and using the angle from point Q to point P as the final angle of the pointer; Calculate the angle of each scale mark, including: acquiring the pointer pixel area image and establishing a Cartesian coordinate system; when the scale mark is recognized as a line segment, take the two endpoints of the line segment and record them as point P and point Q, where point P is the endpoint far from the image center and point Q is the endpoint close to the image center; acquire the coordinates of point P and point Q, calculate the angle from point Q to point P based on the coordinates of point P and point Q, and use the angle from point Q to point P as the angle of the scale mark; To determine which two adjacent scale marks the pointer falls between, the pointer reading is obtained by comparing the pointer's angle with the angle of each scale mark. Specifically, this involves: determining which two adjacent scale marks the pointer falls between; calculating the angle difference between the pointer and the previous time step, and the angle difference between two adjacent scale marks; and calculating the pointer reading based on these two angle differences, as shown in the following formula: in, α i The angle of the previous scale, α i+1 The angle is the next tick mark after the pointer. s i The angle of the pointer at the previous moment. s i+1 The pointer represents the angle at the current moment.

2. The automatic reading method for pointer-type instruments based on semantic segmentation as described in claim 1, wherein, The DeepLabV3 semantic segmentation model includes a deep convolutional network. The upsampling channel of the deep convolutional network includes four convolutional kernels and pooling layers. One of the convolutional kernels has a size of 1×1, and the remaining convolutional kernels have a size of 3×3. The original image is subjected to four layers of convolution and pooling operations. The feature maps obtained after the convolution operation and the feature maps obtained after the pooling operation are merged, and then an upsampled feature map is obtained by performing a 1×1 convolution kernel operation.

3. The automatic reading method for pointer-type instruments based on semantic segmentation as described in claim 2, wherein, The downsampling channel of the deep convolutional network includes a convolutional kernel of size 1×1; The downsampled feature map, which has undergone a 1×1 convolution kernel operation, is concatenated with the upsampled feature map, and then subjected to a 3×3 convolution kernel operation and upsampling to obtain the semantic segmentation image.

4. The automatic reading method for pointer-type instruments based on semantic segmentation as described in claim 3, wherein, The category of each pixel in the semantic segmentation image is determined to obtain the category corresponding to each pixel.

5. The automatic reading method for pointer-type instruments based on semantic segmentation as described in claim 1, wherein, The training process of the DeepLabV3 semantic segmentation model is as follows: Obtain several original images of the instruments as training sample images; The sample image is input into the DeepLabV3 semantic segmentation model to obtain the predicted classification result of the sample image; The predicted classification results and the true labels are iteratively trained based on the target loss function to obtain the trained DeepLabV3 semantic segmentation model.

6. The automatic reading method for pointer-type instruments based on semantic segmentation as described in claim 1, wherein, The angle of the calculation pointer also includes: When the pointer image is recognized as a triangle, take the midpoint of the shorter side of the triangle and denote it as point Q. The endpoint opposite the shorter side is called point P. Point P is the endpoint closer to the pointer tip, and point Q is the endpoint closer to the pointer tail. Obtain the coordinates of points P and Q, calculate the angle from point Q to point P based on the coordinates of points P and Q, and use the angle from point Q to point P as the final angle of the pointer.

7. The automatic reading method for pointer-type instruments based on semantic segmentation as described in claim 6, wherein, The angle of the calculation pointer also includes: When the pointer is recognized as a line segment, take the two endpoints of the line segment and denote them as point P and point Q, where point P is the endpoint closer to the pointer tip and point Q is the endpoint closer to the pointer tail. Obtain the coordinates of points P and Q, calculate the angle from point Q to point P based on the coordinates of points P and Q, and use the angle from point Q to point P as the final angle of the pointer.

Images