A pointer instrument reading recognition method based on multi-resolution deep feature learning

Abstract

The present invention discloses a method for identifying pointer instrument readings based on multi-resolution deep feature learning, comprising the following steps: constructing a multi-resolution deformable convolutional neural network to determine the network's backbone network and key point detection head; annotating the key points of the instrument representation of each instrument image in an instrument image set to create a data set; inputting the preprocessed data set into a multi-resolution deformable convolutional neural network for training to obtain a multi-resolution deformable convolutional neural network that can be used to predict the key points of the instrument representation; inputting video frames from an inspection robot into the trained network to output predictions of the instrument's key point coordinates; and obtaining the instrument reading using an angle method based on the key point coordinate information. The present invention identifies readings based on the instrument's key points. The simplification of the target and the focus on a limited spatial range not only help the network provide accurate instrument readings but also significantly reduce the complexity of the task.

Description

Technical Field

[0001] The present invention relates to the technical field of image data recognition, and in particular to a pointer instrument reading recognition method based on multi-resolution deep feature learning. Background Art

[0002] Since the beginning of the 21st century, how to quickly and cost-effectively implement digital, networked manufacturing using existing instruments and equipment has become a pressing and important challenge. Among these challenges, the automatic identification and reading of traditional pointer instruments has become a pressing issue in the manufacturing industry's upgrade. Traditional manual visual inspection methods suffer from inaccurate readings, low efficiency, and human error, limiting improvements in production efficiency and quality control. In this context, applying deep learning algorithms to inspection robots, enabling them to automatically identify the readings of pointer instruments, not only addresses the human errors and instability inherent in traditional manual visual inspection methods, improving recognition accuracy and reliability, but also significantly boosts production efficiency and quality control, providing a foundation for real-time monitoring and data analysis. Furthermore, integration with other intelligent devices and systems further promotes intelligent management of production lines, providing a new approach to achieving digital, networked upgrades and innovation in the manufacturing industry.

[0003] Early researchers designed solutions for pointer instrument reading recognition based on traditional machine vision algorithms. They employed template matching to precisely align the camera with the instrument panel, segmented the instrument dial and pointer, extracted features, and used a lookup table to identify the pointer instrument. While this approach was effective for pointer instrument recognition, it was susceptible to various uncontrollable factors, including complex backgrounds, wide variations in lighting, pointer instrument tilt, image blur, and scale variations, limiting its effectiveness to specific environments or fixed locations.

[0004] In recent years, thanks to the rapid development of convolutional neural networks, researchers have successively developed pointer instrument reading recognition schemes based on target detection and semantic segmentation. For example, the prior art publication number CN114037993B discloses a substation pointer instrument reading method, device, storage medium, and electronic device that predicts the precise area where the pointer instrument is located and extracts pointer features based on traditional machine vision algorithms to achieve pointer instrument recognition. Although the problem of instrument recognition in complex backgrounds has been solved, the features of the instrument dial and pointer cannot be effectively extracted. For example, the prior art publication number CN114037824A discloses a pointer instrument reading recognition method that predicts the binary mask of the instrument marking area, inserts a clustering model into training and inference, groups the mask pixels into a single instance, and finally uses a straight line fitting to parameterize the result. However, the pixel-level output of the straight line fitting is often redundant and noisy, which has a negative impact on the accuracy of the final result.

[0005] Furthermore, most of the aforementioned research methods rely solely on the rotational deflection angle of the pointer to determine the meter reading. However, since the inspection robot is a mobile platform, the deflection angle varies from image to image, resulting in a different tilt and deflection angle for each meter in the captured image. These varying tilt and deflection angles can significantly affect the extracted rotational deflection angle of the pointer, hindering the model's ability to accurately determine the meter reading. Summary of the Invention

[0006] 1. Technical problems to be solved:

[0007] To address the above technical issues, the present invention provides a method for pointer instrument reading recognition based on multi-resolution deep feature learning. This method formulates the pointer instrument reading recognition task from the perspective of key point estimation and designs a multi-resolution deformable convolutional model that directly represents each detection target in the pointer instrument using key points, achieving accurate readings of pointer instruments. This method can automatically complete pointer instrument reading recognition with high accuracy and robustness even in situations where a mobile inspection robot's complex working environment may result in blurred images, complex lighting conditions, or tilted shooting angles.

[0008] 2. Technical solution:

[0009] A pointer instrument reading recognition method based on multi-resolution deep feature learning is characterized by comprising the following steps:

[0010] Step 1: Build a multi-resolution deformable convolutional neural network and determine the network backbone and loss function;

[0011] Step 2: Collect instrument images including the original environment to obtain an instrument image set, annotate the key points of the instrument representation of each instrument image in the instrument image set to create a data set; and expand the data set through preprocessing including data enhancement;

[0012] Step 3: Input the preprocessed dataset into a multi-resolution deformable convolutional neural network for iterative training and testing to obtain a multi-resolution deformable convolutional neural network that can be used to predict instrument characterization key points;

[0013] Step 4: Use the inspection robot to obtain the environment video, input the video frame into the trained multi-resolution deformation convolutional neural network, and output the predicted results of the key point coordinates of the instrument contained in the video frame;

[0014] Step 5: Calculate the instrument reading value using the angle method based on the key point coordinate information of the instrument representation in the prediction results to obtain the instrument reading.

[0015] Furthermore, the multi-resolution deformable convolutional neural network in step 1 includes a spatial local feature enhancement module, a parallel multi-resolution sub-network, and a global perception module;

[0016] The spatial local feature enhancement module is located at the head of the parallel multi-resolution sub-network, and the image to be recognized is enhanced by the spatial local feature enhancement module and then input into the parallel multi-resolution sub-network;

[0017] The parallel multi-resolution subnetwork has a four-layer structure; the high-resolution branch of the first layer is downsampled 4 times, and the enhanced image is processed by one convolution, four inverted residual blocks and corresponding information fusion and then output to the tail end of the subnetwork; the branch of the second layer is downsampled 8 times, and the enhanced image is processed by one convolution, four inverted residual blocks and corresponding information fusion and then output to the tail end of the subnetwork; the branch of the third layer is downsampled 16 times, and branches are branched after the data fusion after the one convolution and one inverted residual block processing of the second layer, and the branch is sequentially processed by the global perception module, the three inverted residual block and its corresponding three-time information fusion and then output to the tail end of the subnetwork; the fourth layer is downsampled 32 times, and branches are branched after the data fusion after the global perception module and one inverted residual block processing of the third layer, and the branch is processed by the global perception module, the two inverted residual blocks and its corresponding information fusion and then output to the tail end of the subnetwork; wherein the resolution of the first, second, third and fourth layers of the network is reduced by half layer by layer;

[0018] Image processing in the multi-resolution deformable convolutional neural network is divided into four stages. In the first stage, the image enhanced by the spatial local feature enhancement module is passed through two convolutional layers with a convolution kernel size of 3×3 in parallel and output to the corresponding branches with a downsampling of 4 and 8 times, respectively. In the second stage, the outputs of the first and second branches are passed through the inverted residual block in sequence and then the two branches are fused and output. In the third stage, after the second stage outputs of the first and second branches, a branch with a downsampling of 16 is added through the global perception module. The three branches are passed through the inverted residual block in sequence and then the three branches are fused and output. In the fourth stage, after the outputs of the first, second, and third branches in the third stage, a branch with a downsampling of 32 is added through the global perception module. The four branches are passed through the inverted residual block in sequence and then fused. The fused branch passes through the inverted residual block again and is upsampled to converge the outputs of the four branches into one dimension. After passing through a convolution layer with a convolution kernel size of 1×1, the feature map with the same input resolution is output. Information fusion between branches in different layers in each of the second, third, and fourth stages is achieved through upsampling and downsampling.

[0019] Furthermore, the spatial local feature enhancement module is used to locally enhance the spatial features of the input image and filter out invalid information; the specific process of processing the input image is: the input image is subjected to 3×3 and 1×1 convolution to learn local spatial information and the input features are projected into a high-dimensional space, and then expanded into N non-overlapping patches. For each patch, a multi-head attention mechanism is executed; finally, the result of the multi-head attention mechanism is fused with the input image as the output of the module.

[0020] Furthermore, the global perception module can adaptively learn the deformation features in the input image, accelerate the convergence of its sub-network, and further improve the accuracy of model detection; the specific process of processing the input image is: the input image is sequentially passed through a depth-wise separable convolution layer with a convolution kernel size of 3×3 and a convolution layer with a convolution kernel size of 1×1 to obtain a feature map with a downsample of 2 times and doubled number of channels; then each channel is processed by a convolution layer with a convolution kernel size of 3×3 to obtain a feature map with a channel number 3 times the number of channels of the input image, and the map is input to the deformable convolution submodule; the offset and activation weight of each convolution kernel are calculated in the deformable convolution submodule; and the offset and activation weight of each convolution kernel are added to the convolution layer with a convolution kernel size of 3×3 to obtain the output of the deformable convolution submodule, and finally a convolution layer with a convolution kernel size of 1×1 is used to fuse the input features and the output features of the deformable convolution submodule as the output of the module.

[0021] Furthermore, the images in the instrument image set include 10,264 RGB images of pointer instruments of different models with a size of 1920×1080; the images include instruments and original environmental information; the original environmental information includes time information, light information, surrounding environment information, and shooting deflection angle information;

[0022] Step 2 specifically includes the following steps:

[0023] S21: Use Labelme to mark the pointer vertex, instrument center, and all scale line vertices in the instrument image to obtain the key point labels corresponding to the image; the key point labels contain the key point set corresponding to the image N represents the total number of key points in the tag, and M represents the total number of types of key points in the tag; represents the coordinates of the i-th key point of the j-th category, that is,

[0024] S22: Expand the dataset using preprocessing including data enhancement; the data enhancement is to perform random scaling, cropping, horizontal flipping, random rotation, and Gaussian blurring on the image including the label generated in step S21, thereby obtaining an expanded instrument image set.

[0025] Furthermore, step three specifically includes:

[0026] The label information in the dataset obtained in step 2 is used as the true value of the key point, and the two-dimensional Gaussian algorithm is used to generate the key point heat map As the input of the multi-resolution deformable convolutional neural network, where H is the height of the input image; W is the width of the input image; the heat map represents the probability that each pixel position in the image is the key point to be predicted. The closer the pixel is to the key point to be estimated, the closer the probability is to 1, and the farther the pixel is from the key point, the closer the probability is to 0; the value at the coordinate (x, y) in the heat map is:

[0027]

[0028] Where, and Represents the coordinates of the true value of each key point, with a standard deviation σ of 1 pixel;

[0029] During training, the multi-resolution deformable convolutional neural network outputs a heat map representing the key point positions of the predicted instrument, and compares it with the actual heat map to calculate the mean square error for backpropagation and optimize the network weight parameters; it determines whether the preset number of loop iterations is met. If the number of loop iterations reaches the set number, the iteration is stopped; otherwise, training continues.

[0030] Finally, a heat map of the key points of the instrument representation predicted by the network is obtained, and the coordinate information of the key points of the instrument needle is obtained based on the heat map.

[0031] Furthermore, the process of obtaining the meter reading in step 5 specifically includes:

[0032] From the prediction results, the scale values corresponding to the coordinates of the two scale line vertices adjacent to the pointer are obtained, which are expressed as in The pointer vertex coordinates are located at and This forms four groups of data areas, namely and Each group is assigned a different weight. The weights of the four groups of data are 0.4, 0.25, 0.25, and 0.1 respectively. The weighted pointer readings of the four groups of data are calculated as follows:

[0033]

[0034] In the above formula: S represents the scale corresponding to the instrument pointer; w i Represents the weight corresponding to the i-th group of data; α iγ represents the angle between the vertex of the scale line connecting the smaller scale values of the i-th group of data and the vertex of the scale line connecting the center of the instrument and the larger scale value; i It represents the angle between the pointer vertex of the i-th group of data, the center of the instrument and the vertex of the scale line with the smaller scale value.

[0035] 3.Beneficial effects:

[0036] (1) This method for identifying pointer instrument readings addresses the existing techniques that typically use target detection or semantic segmentation methods. Since the former uses bounding boxes as output, the latter's output is redundant and noisy, resulting in their inability to obtain accurate location information. This method, on the other hand, identifies readings based on the instrument's key points. This simplification of the target and focus on a limited spatial range not only helps the network provide accurate instrument readings but also significantly reduces the complexity of the task.

[0037] (2) This pointer instrument reading recognition method constructs a multi-resolution deformable convolutional neural network to predict the key point information of the image; the network adopts a multi-resolution parallel architecture design, maintains high-resolution representation throughout the process, connects multiple resolutions in parallel, repeatedly exchanges information in parallel multi-resolution sub-networks, and realizes high-resolution feature output through cross-resolution information interaction, realizes multi-scale fusion, and strengthens the position information in the network.

[0038] (3) A spatial local feature enhancement module and a global perception module are set up in the multi-resolution deformable convolutional neural network. Among them, the spatial local feature enhancement module is placed before the parallel multi-resolution sub-network to locally enhance the spatial features, filter out some invalid information, and indirectly improve the utilization rate of effective information in the subsequent sub-network. The global perception module is placed in the medium and low resolution of the parallel multi-resolution sub-network. This module is used to adaptively learn the deformation features of the input image, accelerate network convergence, and further improve the accuracy of model detection.

[0039] (4) In this solution, the angle method is used to calculate the instrument readings for the predicted key points. In view of the tilt and deflection of the instrument in the collected environmental image, the step of tilt correction of the instrument image is eliminated. This not only improves the system response time but also avoids the cumulative error caused by tilt correction, fundamentally solving the interference of the pointer instrument tilt. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is the overall structure diagram of the multi-resolution deformation convolutional neural network in the present invention;

[0041] Figure 2 Schematic diagram of the spatial local feature enhancement module in the present invention;

[0042] Figure 3Schematic diagram of the global perception module in the present invention;

[0043] Figure 4 This is a flow chart for realizing pointer instrument reading recognition in the present invention;

[0044] Figure 5 It is part of the instrument image set in a specific embodiment;

[0045] Figure 6 It is a schematic diagram of calculating the pointer angle in a specific embodiment. DETAILED DESCRIPTION

[0046] The present invention will be described in detail below with reference to the accompanying drawings.

[0047] Specific embodiment: Its process is as shown in the attached Figure 4 shown.

[0048] The instrument images including the original environment are collected and summarized to obtain the instrument image set. The data set of this embodiment contains 10264 RGB images of different models of pointer instruments with a size of 1920×1080, as shown in the attached figure. Figure 5 The picture shows a part of the instrument image set, and the situations involved in the picture include: different time, light, environment, shooting deflection angle, etc. The images in the data set are composed of image information and corresponding label information; the image information is different time, light, environment, shooting deflection angle, etc. It is necessary to distinguish the image information from the instrument information, and use the software Labelme to mark the key points of the pointer instrument and the instrument in the image to generate the key point label of the image; in this embodiment, the key points include the pointer vertex, the center of the instrument, and all the scale line vertices; thus, the marked label information is used as the true value of the key point, and the two-dimensional Gaussian algorithm is used to generate the key point heat map. As the input of the multi-resolution deformable convolutional neural network; the heat map represents the probability that each pixel position in the image is the key point to be estimated. The closer the pixel is to the key point to be estimated, the closer the probability is to 1, and the farther the pixel is from the key point, the closer the probability is to 0.

[0049] Construct a multi-resolution deformable convolutional neural network, the structure of which is shown in the attached figure. Figure 1 As shown in the figure, it consists of four branches, with a portion of each branch transmitting information through upsampling and downsampling, forming four stages. The design of the multi-resolution parallel architecture maintains high-resolution representation throughout the entire process. Simultaneously, by connecting multiple resolutions in parallel and repeatedly exchanging information within the parallel multi-resolution subnetworks, high-resolution feature output is achieved through cross-resolution information exchange, enabling multi-scale fusion.

[0050] The multi-resolution deformable convolutional neural network in this method includes a spatial local feature enhancement module, a parallel multi-resolution sub-network and a global perception module; the structure of the spatial local feature enhancement module is as shown in the attached figure. Figure 2 As shown in the attached Figure 3 shown.

[0051] The spatial local feature enhancement module is placed before the parallel multi-resolution sub-network to locally enhance spatial features, filter out some invalid information, and indirectly improve the utilization of valid information in subsequent sub-networks. The specific principle of image processing includes the following steps:

[0052] Step A1: For a given input image X∈R H×W×C ; Use 3×3 and 1×1 convolution to learn local spatial information and project the input features into high-dimensional space to obtain X L ∈R H×W×d ; Where H is the height of the input image, W is the width of the input image, C is the number of channels of the input image, X L It is the feature map obtained after 3×3 and 1×1 convolution learning.

[0053] Step A2: Place X L Expand into N non-overlapping patches: X U ∈R P×N×D Where P = wh, is the number of patches, h≦n, w≦n is the height and width of the patch; for each expanded patch, a multi-head attention mechanism is executed; X U , represents patch.

[0054] Taking the p-th patch as an example, the formula for its multi-head attention is:

[0055]

[0056]

[0057] Where, M represents the number of heads, D represents the number of channels, and P represents the input resolution. Represents the output representation of MultiHead.

[0058] Step A3: Fold the features processed by the multi-head attention mechanism in step A2 back to the original feature map size and fuse them with the input image using a 1x1 convolutional layer to optimize deeper layers in the architecture.

[0059] The global perception module is placed in the parallel multi-resolution sub-network at low and medium resolutions. This module is used to adaptively learn the deformation characteristics of the detected target, accelerate network convergence, and further improve the accuracy of model detection. The specific processing steps are as follows:

[0060] Step B1: Input the feature image X∈R of the module H×W×C Each position P0 on the image is subjected to a convolution operation with a kernel size of 3, a stride of 1, and a padding of 1:

[0061]

[0062] In the above formula, K represents the total number of sampling points, k is the number of sampling points, and P k represents the position of the kth convolution kernel in R, w k is the projection weight.

[0063] Thus, 9 positions are sampled from the input feature map X, and these 9 positions can be represented by the set R:

[0064]

[0065] In the formula, (-1,-1) represents the upper left corner of P0, (1,1) represents the lower right corner of P0, and the 9 points are arranged in a 3*3 square. Through the above convolution operation, an offset feature map Y with the same size as the input feature map X and 3 times the number of channels is obtained. H ×W×3c , where two channels correspond to the offset ΔP of each position P0 on the input feature map X in both the x-axis and y-axis directions, and the other channel corresponds to the activation weight Δm at each position P0 on the input feature map X k .

[0066] Step B2: Based on the calculation of the convolution operation in step B1, add the learned offset ΔP k and activation weight Δm k , so that the standard convolution layer with a 3×3 convolution kernel is extended to an irregular shape. The operation is specifically expressed as:

[0067]

[0068] Step B3: Use bilinear interpolation to calculate x(P0+P k +ΔP k ) integer value corresponding to the position on the input feature map X;

[0069] x(P0+P k +ΔP k )=∑ q G(q,p)·x(q)

[0070] Where q is the enumeration of all integer positions on the input feature map X; G(-,-) is a two-dimensional bilinear interpolation kernel, where:

[0071] G(q,p)=g(q x ,p x )·g(q y ,p y )

[0072] Where g(a,b)=max(0,1-|ab|);

[0073] Step B4: Concatenate the input features in step B1 and the module output features obtained in step B3, perform feature fusion through a 1x1 convolution, adjust the number of channels, and then output.

[0074] Some parameters and function settings of the multi-resolution deformable convolutional neural network are as follows:

[0075] (1) The activation function of each layer branch is:

[0076] ReLU6=min(6,max(0,x)).

[0077] (2) The loss function is:

[0078]

[0079] Where Y i represents the true value, Represents the predicted value, and n is the total number of samples.

[0080] (3) The model was iteratively trained using the Adam optimizer and an initial learning rate of 0.001. The training was repeated 40 times with a batch size of 16.

[0081] Step B5: Obtain the scale values corresponding to the coordinates of the two scale line vertices adjacent to the pointer from the prediction results, which are expressed as P l 2 ,P l 1 , in The pointer vertex coordinates are located at P l 1 and This forms four groups of data areas, namely and Each group is assigned a different weight. The weights of the four groups of data are 0.4, 0.25, 0.25, and 0.1 respectively. The weighted pointer readings of the four groups of data are calculated as follows:

[0082]

[0083] In the above formula: S represents the scale corresponding to the instrument pointer; wi represents the weight corresponding to the i-th group of data; Figure 6 As shown, α i γ represents the angle between the vertex of the scale line connecting the smaller scale values of the i-th group of data and the vertex of the scale line connecting the center of the instrument and the larger scale value; i It represents the angle between the pointer vertex of the i-th group of data, the center of the instrument and the vertex of the scale line with the smaller scale value.

[0084] Verification example:

[0085] This verification is to evaluate the key point judgment in the multi-resolution deformable convolutional neural network; the experimental environment is shown in Table 1.

[0086] Table 1 Experimental environment

[0087]

[0088] We use the object similarity (OKS) as the keypoint estimation evaluation metric, including AP (average precision), AP50, and AP75. AP is the average precision of all OKS thresholds, ranging from 0.5 to 0.95. AP50 represents the average precision when the OKS threshold is greater than 0.5, and AP75 represents the average precision when the OKS threshold is greater than 0.75.

[0089]

[0090] Where, d i is the Euclidean distance between the detected key point and the corresponding true value, vi is the visibility flag of the key point true value, s is the scale of the detected object, and k i is the decay constant of key point category i. Recall is used as the evaluation metric for instrument reading recognition. The true value of the annotation will be selected from the set of instrument scale values, and the instrument reading output by the algorithm will be rounded to the closest scale value.

[0091]

[0092] Where TP represents the number of samples whose readings recognized by the algorithm are the same as the true values marked by humans, that is, the number of samples correctly identified; FN represents the number of samples whose instrument readings recognized by the algorithm are different from the true values marked by humans, that is, the number of samples incorrectly identified.

[0093] The MRDNet of the present invention reaches the state-of-the-art level with 82.4% AP, 93.5% AP50, and 94.6% AP75, with great advantages in performance and speed. Our MRDNet achieves leading performance while significantly reducing the number of parameters.

[0094] Based on the above evaluation metrics, we conclude that our method for identifying pointer instrument readings based on multi-resolution deep feature learning achieves an industry-leading recall of 99.6%. This is primarily due to the fact that previous methods calculated readings solely based on the pointer's deflection angle, ignoring the importance of scale markings. This resulted in poor robustness and inability to accurately identify readings when the pointer instrument is tilted. This method, however, calculates readings based on the deflection angle between the pointer and adjacent scale markings, fundamentally addressing the interference caused by pointer instrument tilt.

[0095] Although the present invention has been disclosed above in terms of preferred embodiments, they are not intended to limit the present invention. Anyone skilled in the art can make various changes or modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection defined by the claims of this application.

Claims

1. A pointer instrument reading recognition method based on multi-resolution deep feature learning, characterized by: The following steps are involved: Step 1: Build a multi-resolution deformable convolutional neural network and determine the network backbone and loss function; Step 2: Collect instrument images including the original environment to obtain an instrument image set, annotate the key points of the instrument representation of each instrument image in the instrument image set to create a data set; and expand the data set through preprocessing including data enhancement; Step 3: Input the preprocessed dataset into a multi-resolution deformable convolutional neural network for iterative training and testing to obtain a multi-resolution deformable convolutional neural network that can be used to predict instrument characterization key points; Step 4: Use the inspection robot to obtain the environment video, input the video frame into the trained multi-resolution deformation convolutional neural network, and output the prediction results of the key point coordinates of the instrument contained in the video frame; Step 5: Calculate the instrument reading value using the angle method based on the key point coordinate information of the instrument representation in the prediction results to obtain the instrument reading; The multi-resolution deformable convolutional neural network in step 1 includes a spatial local feature enhancement module, a parallel multi-resolution sub-network, and a global perception module; The spatial local feature enhancement module is located at the head of the parallel multi-resolution sub-network, and the image to be recognized is enhanced by the spatial local feature enhancement module and then input into the parallel multi-resolution sub-network; The parallel multi-resolution subnetwork has a four-layer structure; the high-resolution branch of the first layer is downsampled 4 times, and the enhanced image is processed by one convolution, four inverted residual blocks and corresponding information fusion and then output to the tail end of the subnetwork; the branch of the second layer is downsampled 8 times, and the enhanced image is processed by one convolution, four inverted residual blocks and corresponding information fusion and then output to the tail end of the subnetwork; the branch of the third layer is downsampled 16 times, and the branch is branched after the data fusion after the one convolution and one inverted residual block processing of the second layer, and the branch is sequentially processed through the global perception module, the three inverted residual block and its corresponding three-time information fusion and then output to the tail end of the subnetwork; the fourth layer is downsampled 32 times, and the branch is branched after the data fusion after the global perception module and one inverted residual block processing of the third layer, and the branch is processed through the global perception module, the two inverted residual blocks and its corresponding information fusion and then output to the tail end of the subnetwork; wherein the resolution of the first, second, third and fourth layers of the network is reduced by half layer by layer.

2. The method for pointer instrument reading recognition based on multi-resolution deep feature learning according to claim 1, characterized in that: In step 1, the image processing in the multi-resolution deformable convolutional neural network is divided into four stages. In the first stage, the image enhanced by the spatial local feature enhancement module is passed through two convolutional layers with a convolution kernel size of 3×3 in parallel and output to the corresponding branches with a downsampling of 4 times and 8 times, respectively. In the second stage, the outputs of the first and second branches in the first stage are passed through the inverted residual block in sequence, and the two branches are fused and output; in the third stage, a branch with a 16-fold downsampling is added through the global perception module after the output of the second stage of the first and second branches, and the three branches are fused and output in sequence after passing through the inverted residual block; in the fourth stage, a branch with a 32-fold downsampling is added through the global perception module after the output of the first, second, and third branches in the third stage, and the four branches are fused after passing through the inverted residual block in sequence. The fused branches pass through the inverted residual block again and use upsampling to converge the outputs of the four branches into one dimension, and then pass through a convolution layer with a convolution kernel size of 1×1 to output a feature map with the same input resolution; the information fusion between branches of different layers in each stage of the second, third, and fourth stages is achieved through upsampling and downsampling.

3. The method for pointer instrument reading recognition based on multi-resolution deep feature learning according to claim 2, characterized in that: The spatial local feature enhancement module is used to locally enhance the spatial features of the input image and filter out invalid information. The specific process of processing the input image is as follows: the input image is subjected to 3×3 and 1×1 convolution to learn local spatial information and the input features are projected into a high-dimensional space, and then expanded into N non-overlapping patches. For each patch, a multi-head attention mechanism is then executed; finally, the result of the multi-head attention mechanism is fused with the input image as the output of the module.

4. The method for pointer instrument reading recognition based on multi-resolution deep feature learning according to claim 2, characterized in that: The global perception module can adaptively learn the deformation features in the input image, accelerate the convergence of its sub-network, and further improve the accuracy of model detection. The specific process of processing the input image is as follows: the input image is sequentially passed through a depthwise separable convolution layer with a convolution kernel size of 3×3 and a convolution layer with a convolution kernel size of 1×1 to obtain a feature map that is downsampled by 2 times and has twice the number of channels; each channel is then processed through a convolution layer with a convolution kernel size of 3×3 to obtain a feature map with a channel number that is 3 times the number of channels of the input image, and the feature map is input to the deformable convolution sub-module; the offset and activation weight of each convolution kernel are calculated in the deformable convolution sub-module; The offset and activation weight of each convolution kernel are added to the convolution layer with a convolution kernel size of 3×3 to obtain the output of the deformable convolution submodule. Finally, a convolution layer with a convolution kernel size of 1×1 is used to fuse the input features and the output features of the deformable convolution submodule as the output of the module.

5. The method for pointer instrument reading recognition based on multi-resolution deep feature learning according to claim 1, characterized in that: The images in the instrument image collection include 10,264 RGB images of different models of pointer instruments with a size of 1920×1080; the images include instruments and original environmental information; the original environmental information includes time information, light information, surrounding environment information, and shooting deflection angle information; Step 2 specifically includes the following steps: S21: Use Labelme to mark the pointer vertex, instrument center, and all scale line vertices in the instrument image to obtain the key point labels corresponding to the image; the key point labels contain the key point set corresponding to the image ; N represents the total number of key points in the tag, and M is the total number of types of key points in the tag; represents the coordinates of the i-th key point of the j-th category, that is, ; S22: Expand the dataset using preprocessing including data enhancement; the data enhancement is to perform random scaling, cropping, horizontal flipping, random rotation, and Gaussian blurring on the image including the label generated in step S21, thereby obtaining an expanded instrument image set.

6. The method for pointer instrument reading recognition based on multi-resolution deep feature learning according to claim 5, characterized in that: Step three specifically includes: The label information in the dataset obtained in step 2 is used as the true value of the key point, and the two-dimensional Gaussian algorithm is used to generate the key point heat map As the input of the multi-resolution deformable convolutional neural network, where H is the height of the input image; W is the width of the input image; the heat map represents the probability that each pixel position in the image is the key point to be predicted. The closer the pixel is to the key point to be estimated, the closer the probability is to 1, and the farther the pixel is from the key point, the closer the probability is to 0; the value at the coordinate (x, y) in the heat map is: ； Where, and Indicates the coordinates and standard deviation of the true value of each key point is 1 pixel; During training, the multi-resolution deformable convolutional neural network outputs a heat map representing the key point positions of the predicted instrument, and compares it with the actual heat map to calculate the mean square error to perform back propagation and optimize the network weight parameters; it determines whether the preset number of loop iterations is met. If the number of loop iterations reaches the set number, the iteration is stopped; otherwise, training continues; Finally, a heat map of the key points of the instrument representation predicted by the network is obtained, and the coordinate information of the key points of the instrument needle is obtained based on the heat map.

7. The method for pointer instrument reading recognition based on multi-resolution deep feature learning according to claim 6, characterized in that: The process of obtaining the meter reading in step 5 specifically includes: From the prediction results, the scale values corresponding to the coordinates of the two scale line vertices adjacent to the pointer are obtained, which are expressed as , , , ,in < < < , the pointer vertex coordinates are located at and Thus, four groups of data areas are formed, namely ( )、( , )、( , )and( , ); and assign different weights to each group. The weights corresponding to the four groups of data are 0.4, 0.25, 0.25 and 0.1 respectively; the weighted pointer readings of the four groups of data are calculated as follows: ； In the above formula: S represents the scale corresponding to the instrument pointer; w i Indicates the weight corresponding to the i-th group of data; It represents the angle between the vertex of the scale line connecting the smaller scale values of the i-th group of data and the vertex of the scale line connecting the center of the instrument and the larger scale value; It represents the angle between the pointer vertex of the i-th group of data, the center of the instrument and the vertex of the scale line with the smaller scale value.

Images