A digital instrument display end-to-end recognition method

Abstract

This invention discloses an end-to-end method for digitizer reading recognition, comprising: acquiring an original digitizer reading image; extracting multiple feature maps from the original digitizer reading image to form an initial feature pyramid; performing multi-scale and multi-level feature fusion on the initial feature pyramid to obtain a feature map with reading position information; performing image segmentation on the feature map with reading position information to generate a probability map and a threshold map; predicting the reading position based on the probability map and the threshold map using an adaptive differentiable binarization method to obtain an approximate binary map; using the approximate binary map as a binary mask to extract feature map patches of the reading region from the feature map with reading position information; and performing reading recognition on the feature map patches of the reading region to obtain the digitizer reading content. This invention can effectively reduce the time cost of each step, achieving high inference speed while maintaining competitive accuracy.

Description

Technical Field

[0001] This invention relates to the field of end-to-end scene text observation technology, specifically to an end-to-end method for digital instrument representation of data. Background Technology

[0002] Currently, text detection and text recognition are generally considered two independent sub-problems. Text detection typically takes images captured in natural scenes as input and outputs bounding boxes of text within the image; while text recognition takes a single text image as input and outputs the text content contained within the image. To accomplish the task of reading text symbols from an image, both methods need to be combined. However, combining these independent methods into a complete and unified scene text reading system inevitably introduces additional computational overhead and consumes more system resources. For example, when cropping the text bounding boxes from the original image using text detection methods into individual text images to be recognized, the cropped images are stored on disk and then read from disk as input for the text recognition step. Furthermore, the two sets of parameters and weights obtained after training for the text detection and text recognition models are independent, and prediction and inference are also called independently, which increases the complexity of the scene text reading system. Summary of the Invention

[0003] To address the aforementioned shortcomings in the prior art, this invention provides an end-to-end identification method for digital instrument readings.

[0004] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0005] A digital instrument display end-to-end identification method includes the following steps:

[0006] S1. Obtain the original digital instrument reading image;

[0007] S2. Extract multiple feature maps from the original digital instrument reading image to form an initial feature pyramid;

[0008] S3. Perform multi-scale and multi-level feature fusion on the initial feature pyramid to obtain a feature map with numerical location information;

[0009] S4. Perform image segmentation on the feature map containing the display position information to generate a probability map and a threshold map;

[0010] S5. Based on the probability map and threshold map, the display position is predicted by the adaptive differentiable binarization method to obtain an approximate binary map;

[0011] S6. Using an approximate binary map as a binary mask, extract feature map blocks of the display region from the feature map containing display position information;

[0012] S7. Perform reading recognition on the feature blocks of the reading area to obtain the reading content of the digital instrument.

[0013] Optionally, step S2 includes the following sub-steps:

[0014] S21. A lightweight deep convolutional neural network is used to extract multiple feature maps of different scales from the original digital instrument display images;

[0015] S22. The extracted feature maps of different scales are converted into feature maps with a uniform number of channels through convolution operation.

[0016] S23. Construct an initial feature pyramid based on the feature map of the unified number of channels.

[0017] Optionally, step S3 includes the following sub-steps:

[0018] S31. Perform a downward fusion operation on the feature maps of each group of adjacent layers in the initial feature pyramid to obtain the feature pyramid after downward fusion.

[0019] S32. Perform bottom-up path enhancement operation on the feature maps of each group of adjacent layers in the feature pyramid after downward fusion to obtain the feature pyramid after upward enhancement.

[0020] S33. Perform convolution operation on each feature map in the up-enhanced feature pyramid to obtain a feature map with the number of channels reduced to a set ratio, then upsample it to a set image size, and concatenate the processed feature maps to obtain a feature map with display position information.

[0021] Optionally, step S31 includes the following sub-steps:

[0022] S311. For each group of adjacent feature maps in the initial feature pyramid, perform double bilinear interpolation upsampling on the high-level feature map to obtain a feature map with the same size as the low-level feature map.

[0023] S312. Add the feature map obtained in step S311 to the low-level feature map element by element to obtain the first fused feature map;

[0024] S313. Perform a depthwise separable convolution operation on the first fused feature map to obtain a single-channel feature map with the same number of channels as the first fused feature map.

[0025] S314. Perform pointwise convolution operations on each single-channel feature map obtained in step S313, and weight and combine all single-channel feature maps in the depth direction to generate the second fused feature map.

[0026] Optionally, step S4 includes the following sub-steps:

[0027] S41. Construct a display region segmentation network model; the display region segmentation network model includes a cascaded first convolutional layer, a first batch of normalized layers, a first ReLU activation layer, a first deconvolutional layer, a second batch of normalized layers, a second ReLU activation layer, a second deconvolutional layer, a third batch of normalized layers, and a third ReLU activation layer;

[0028] S42. The feature map with display position information is processed using the first convolutional layer, the first batch of normalization layers, and the first ReLU activation layer to obtain the first segmentation feature map;

[0029] S43. The first segmentation feature map is processed using the first deconvolution layer, the second batch normalization layer, and the second ReLU activation layer to obtain the second segmentation feature map;

[0030] S44. The second segmentation feature map is processed using the second deconvolution layer, the third batch normalization layer, and the third ReLU activation layer to generate a probability map and a threshold map.

[0031] Optionally, step S5 specifically includes:

[0032] Based on the probability map and threshold map generated in step S4, the weight value of each pixel is calculated using an adaptive differentiable binarization function to obtain an approximate binary map.

[0033] Optionally, step S6 includes the following sub-steps:

[0034] S61. Using an approximate binary image as a binary mask, multiply the feature image with the display position information element by element with the binary mask to obtain the filtered feature image.

[0035] S62. Calculate the smallest bounding rectangle of the target display region contained in the filtered feature map;

[0036] S63. Crops out all feature map blocks within the smallest bounding rectangle from the filtered feature map;

[0037] S64. Adjust the cut feature blocks to a fixed size to obtain the feature blocks of the display area.

[0038] Optionally, step S7 includes the following sub-steps:

[0039] S71. Construct a display recognition network model; the display recognition network model includes a activator and a decoder, the activator includes a first embedding layer and a first multi-head attention layer connected to the first embedding layer, the decoder includes a second long short-term memory network layer and a second embedding layer, a second multi-head attention layer and a first long short-term memory network layer respectively connected to the second long short-term memory network layer.

[0040] S72. Use the first embedding layer of the initiator to convert the character sequence start identifier into a one-hot encoded vector;

[0041] S73. The first multi-head attention layer of the initiator extracts the feature vector of the start identifier of the character sequence based on the feature map of the display region and the one-hot encoded vector.

[0042] S74. Using the first long short-term memory network layer of the decoder, the hidden state vector of the initial time slice is obtained based on the feature vector of the character sequence start identifier and the state vector of the initial time slice.

[0043] S75. Using the second embedding layer, the second long short-term memory network layer, and the second multi-head attention layer of the decoder, the hidden state vector of the initial time slice, the character sequence start marker, and the feature map patch of the display region, the character sequence end marker is obtained by iteratively performing loops.

[0044] The present invention has the following beneficial effects:

[0045] The end-to-end digital instrument reading recognition method provided by this invention, through the above design, can effectively reduce the time overhead of each step, thereby better realizing an instrument reading recognition system that meets real-time recognition requirements. Thanks to the aforementioned pipeline design, the end-to-end digital instrument reading recognition method achieves high inference speed while maintaining competitive accuracy. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating an end-to-end digital instrument reading recognition method according to an embodiment of the present invention.

[0047] Figure 2 This is a schematic diagram of the overall architecture of a digital instrument display end-to-end identification method according to an embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of the structure of the multi-level feature fusion module in an embodiment of the present invention;

[0049] Figure 4 This is a flowchart illustrating the downward fusion operation in an embodiment of the present invention;

[0050] Figure 5 This is a schematic diagram of the bottom-up path enhancement unit in an embodiment of the present invention;

[0051] Figure 6 This is a schematic diagram of the detection head structure of the display area detection module in an embodiment of the present invention;

[0052] Figure 7 This is a schematic diagram of the process for extracting feature blocks from a binary mask RoI in an embodiment of the present invention;

[0053] Figure 8 This is a schematic diagram of the display recognition module in an embodiment of the present invention;

[0054] Figure 9 This is a schematic diagram of the structure of the first multi-head attention layer in an embodiment of the present invention;

[0055] Figure 10 This is a schematic diagram of the structure of the second multi-head attention layer in an embodiment of the present invention;

[0056] Figure 11 This is a schematic diagram illustrating a label generation example in an embodiment of the present invention;

[0057] Figure 12 This is a simplified illustration of label generation in an embodiment of the present invention. Detailed Implementation

[0058] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0059] like Figures 1 to 9 As shown, this embodiment of the invention provides an end-to-end digital instrument representation identification method, including the following steps S1 to S7:

[0060] S1. Obtain the original digital instrument reading image;

[0061] In an optional embodiment of the present invention, the image captured by the digitizer is used as the original image. When training the end-to-end recognition network model, the original image to be recognized and its corresponding ground truth are used as model input, wherein the ground truth is labeled at the word level rather than the character level. During the training phase, the bounding boxes labeled in the ground truth are used to extract feature patches, maintaining the consistency of the recognition features.

[0062] S2. Extract multiple feature maps from the original digital instrument reading image to form an initial feature pyramid;

[0063] In an optional embodiment of the present invention, step S2 includes the following sub-steps:

[0064] S21. A lightweight deep convolutional neural network is used to extract multiple feature maps of different scales from the original digital instrument display images;

[0065] S22. The extracted feature maps of different scales are converted into feature maps with a uniform number of channels through convolution operation.

[0066] S23. Construct an initial feature pyramid based on the feature map of the unified number of channels.

[0067] Specifically, since end-to-end image recognition scenarios require efficient recognition, this embodiment uses lightweight deep convolutional neural networks (such as ResNet-18, Vgg-16, MobileNet, etc.) to extract feature maps from the original digital instrument reading images, thereby improving the recognition efficiency of digital instrument readings in end-to-end image recognition scenarios.

[0068] Taking the ResNet-18 network as an example, this embodiment inputs the original digital instrument display image with a size of H×W×3 into the ResNet-18 network. The conv2, conv3, conv4 and conv5 layers of the ResNet-18 network are used to generate four feature maps, and 1×1 convolutions are used to adjust them to a uniform number of channels. Their sizes are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the input image, respectively, with 512 channels in each case. Thus, the four extracted feature maps are combined into a 4-layer feature pyramid to obtain the initial feature pyramid.

[0069] S3. Perform multi-scale and multi-level feature fusion on the initial feature pyramid to obtain a feature map with numerical location information;

[0070] In an optional embodiment of the present invention, since the features generated by lightweight networks typically have only a small receptive field and weak representational power, they cannot effectively detect small instrument display targets and ambiguous targets; therefore, this embodiment employs a multi-level feature fusion module to perform multi-scale and multi-level fusion and enhancement of the initial feature pyramid. The fusion structure of the multi-level feature fusion module is superimposed and computationally efficient. It can be directly added after the lightweight network, allowing the generated feature map to contain more edge and positional information of the digit display than using only the lightweight network, making the features more expressive, thereby improving the recognition accuracy of the digit display.

[0071] Step S3 in this embodiment includes the following sub-steps:

[0072] S31. Perform a downward fusion operation on the feature maps of each group of adjacent layers in the initial feature pyramid to obtain the feature pyramid after downward fusion.

[0073] S32. Perform bottom-up path enhancement operation on the feature maps of each group of adjacent layers in the feature pyramid after downward fusion to obtain the feature pyramid after upward enhancement.

[0074] S33. Perform convolution operation on each feature map in the up-enhanced feature pyramid to obtain a feature map with the number of channels reduced to a set ratio, then upsample it to a set image size, and concatenate the processed feature maps to obtain a feature map with display position information.

[0075] Step S31 includes the following sub-steps:

[0076] S311. For each group of adjacent feature maps in the initial feature pyramid, perform double bilinear interpolation upsampling on the high-level feature map to obtain a feature map with the same size as the low-level feature map.

[0077] S312. Add the feature map obtained in step S311 to the low-level feature map element by element to obtain the first fused feature map;

[0078] S313. Perform a depthwise separable convolution operation on the first fused feature map to obtain a single-channel feature map with the same number of channels as the first fused feature map.

[0079] S314. Perform pointwise convolution operations on each single-channel feature map obtained in step S313, and weight and combine all single-channel feature maps in the depth direction to generate the second fused feature map.

[0080] Specifically, in this embodiment, the multi-level feature fusion module is mainly used to fuse and enhance the features extracted from the lightweight backbone network. Figure 3 As can be seen, the feature information fusion proceeds from left to right and from top to bottom, and can be viewed as a U-shaped structure overall. The fusion of the multi-level feature fusion module is divided into two stages: one is the downward fusion stage of the feature pyramid, and the other is the upward fusion stage using bottom-up path enhancement units.

[0081] In the down-fusion stage, the smaller feature maps at higher levels of the feature pyramid are first upsampled by a 2x bilinear interpolation to obtain feature maps of the same size as the larger feature maps at lower levels immediately adjacent to them. Then, the upsampled feature maps and the lower-level feature maps are added element-wise. Next, a 3×3 depthwise separable convolution is performed on the sum of the two, resulting in N. chn Zhang's single-channel feature map, N chnThis refers to the number of channels in the input feature map of the depthwise separable convolution. Because depthwise separable convolution only performs convolution operations on each channel of the input feature map independently, it doesn't effectively utilize information from different feature maps at the same spatial location. Therefore, pointwise convolution is then used, where the kernel size is 1×1 and the number of channels is N. chn The conventional convolution takes the feature maps output by the deep convolutional layer and combines them in the depth direction to generate new feature maps. Figure 4 The flowchart of the downward fusion operation is shown. The use of depthwise separable convolutions allows the multi-level feature fusion module to expand its receptive field. Furthermore, the number of parameters required by combining depthwise separable convolutions and pointwise convolutions is approximately one-third that of conventional convolutions, significantly reducing the model's parameter count and saving computational overhead.

[0082] The final step in the down-fusion operation is a batch normalization layer and a ReLU activation layer. Additionally, a fusion factor is used at each feature map fusion step to drive the lower layers of the feature pyramid to learn small targets, making the feature fusion suitable for detecting small text instances.

[0083] The higher layers of the Feature Pyramid Network (FPN) contain more large target features due to their smaller feature map size, while the lower layers have larger feature maps and can include more small target features. Since target characters in text detection tasks often appear as small objects in images, it is desirable to focus on learning small targets in the lower layers to improve the network's ability to detect tiny targets. Therefore, this paper incorporates a fusion factor into the top-down feature fusion process in the Feature Pyramid module. This factor controls the information passed from higher to lower layers, encouraging the lower layers to focus on learning small target features, making the feature fusion suitable for small target detection.

[0084] After adding the fusion factor σ, the adjacent layer P i and P i+1 The feature fusion process between them can be expressed by the following formula:

[0085]

[0086] In the formula, f chn (·) is a 1×1 convolution operation for channel matching; f up (·) is a 2x upsampling operation for size matching; Representative from P i+1 Layer to P i Fusion factor for layer feature fusion.

[0087] For the upward fusion stage, the multi-level feature fusion module introduces a bottom-up path enhancement unit to transmit the positional information of edge and shape features in the feature pyramid from low to high.

[0088] The lower layers of the feature pyramid contain more edge and shape features, which is crucial for pixel-level segmentation. Therefore, this embodiment introduces a bottom-up path enhancement unit in the feature pyramid module, enabling the feature map F to contain more positional information of instrument readings, thereby improving network detection performance.

[0089] Bottom-up path-reinforced unit structure, such as Figure 5 As shown, each unit consists of a larger feature map N. i and smaller P i+1 After fusion and connection, a new feature map N is generated. i+1 Each feature map N i Both require downsampling using a 3×3 kernel and a stride of 2, and then P i+1 The feature maps obtained by downsampling are added element-wise to obtain N for feature map concatenation. i+1 Similar to top-down feature fusion, P2, located at the bottom layer, is identical to N2 because it has no lower-level feature maps to fuse with. During the feature fusion stage, the number of channels in the feature maps remains constant, and a ReLU activation layer is added after each convolutional layer.

[0090] This embodiment employs a multi-level feature fusion module, which can enhance multi-scale feature maps by fusing low-level and high-level information. In addition, the multi-level feature fusion module has the following other advantages:

[0091] 1) Bottom-up path enhancement and fusion factor were used to enhance the location and edge features of small targets such as small text in natural scenes.

[0092] 2) The multi-level feature fusion module is scalable and portable. It can be used in a stacked manner to achieve full fusion of feature maps, thereby expanding the receptive field of the feature maps.

[0093] 3) The multi-level feature fusion module is computationally efficient. This advantage benefits from the application of separable convolution (depthwise convolution + pointwise convolution), which allows the multi-level feature fusion module to have only a slight computational overhead. S4. Perform image segmentation on the feature map with indicator location information to generate probability maps and threshold maps;

[0094] In an optional embodiment of the present invention, step S4 includes the following sub-steps:

[0095] S41. Construct a display region segmentation network model; the display region segmentation network model includes a cascaded first convolutional layer, a first batch of normalized layers, a first ReLU activation layer, a first deconvolutional layer, a second batch of normalized layers, a second ReLU activation layer, a second deconvolutional layer, a third batch of normalized layers, and a third ReLU activation layer;

[0096] S42. The feature map with display position information is processed using the first convolutional layer, the first batch of normalization layers, and the first ReLU activation layer to obtain the first segmentation feature map;

[0097] S43. The first segmentation feature map is processed using the first deconvolution layer, the second batch normalization layer, and the second ReLU activation layer to obtain the second segmentation feature map;

[0098] S44. The second segmentation feature map is processed using the second deconvolution layer, the third batch normalization layer, and the third ReLU activation layer to generate a probability map and a threshold map.

[0099] Specifically, such as Figure 6 As shown, this embodiment utilizes a detection head included in the indicator region detection module to segment the feature map containing indicator position information, thereby generating a probability map and a threshold map. The detection head is a simple segmentation network whose function is to predict and generate the probability map P and the threshold map T. The detection head takes the feature map F containing indicator position information as input and contains only one regular convolutional layer with a kernel size of 3×3 (stride and padding are both 1), two 2×2 deconvolutional layers (stride of 2, padding of 0), and corresponding batch normalization layers and ReLU activation layers.

[0100] S5. Based on the probability map and threshold map, the display position is predicted by the adaptive differentiable binarization method to obtain an approximate binary map;

[0101] In an optional embodiment of the present invention, step S5 specifically includes:

[0102] Based on the probability map and threshold map generated in step S4, the weight value of each pixel is calculated using an adaptive differentiable binarization function to obtain an approximate binary map.

[0103] This embodiment designs a differentiable binarization function that adaptively approximates the standard binarization function during model training. The curvature, which was numerically fixed in the past in differentiable binarized functions, is transformed into a trainable parameter to better adapt to different datasets, thereby enhancing the model's generality and robustness. The calculation method is shown below:

[0104]

[0105] In the formula, x is the function input; tanh(·) is the hyperbolic tangent function; log(·) is the logarithmic function; δ is the similarity factor, δ∈(0,1). When δ is sufficiently small, the approximate function can approach the standard binary function. This means that a suitable δ can improve the model's detection accuracy. Therefore, in order to adaptively determine δ, this embodiment adds δ as an optimizable variable to the segmentation network in generating the approximate binary map. During the training process, this allows δ to be adaptively adjusted and the approximate function to evolve into a standard binary function during training.

[0106] In form, the adaptive training process can be viewed as a loss minimization problem of the segmentation network, as shown in the following equation, where x represents the network input and y represents the corresponding network output:

[0107]

[0108] Therefore, the gradient of δ during backpropagation can be calculated and automatically adjusted using L2 regularization constraints, as formally represented below:

[0109]

[0110] Therefore, in the differentiable binarization module, the differentiable binarization function can be transformed into the following expression as the differentiable binarization function used by RetinaDB:

[0111]

[0112] In the formula, The graph represents an approximate binary image; i and j represent the x and y coordinates of each pixel in the approximate binary image, probability image, and threshold image, respectively; tanh(·) is the hyperbolic tangent function; log(·) is the logarithmic function; δ is the similarity factor; P represents the probability image; and T represents the threshold image.

[0113] Specifically, after the detection head predicts the probability map P and the threshold map T, the height and width of P and T are the same as the input image. This embodiment uses an adaptive differentiable binarization function to take the probability map P and the threshold map T as input, calculating an approximate binary map where the weight value of each pixel can only be 0 or 1. In an approximate binary image, a pixel with a weight of 1 represents that the pixel belongs to the instrument reading region and is considered the foreground for detection; a pixel with a weight of 0 represents that the pixel does not belong to the instrument reading region and is considered the background for detection. Therefore, the approximate binary image can be used as a binary mask containing the instrument reading region for feature patch extraction.

[0114] S6. Using an approximate binary map as a binary mask, extract feature map blocks of the display region from the feature map containing display position information;

[0115] In an optional embodiment of the present invention, step S6 includes the following sub-steps:

[0116] S61. Using an approximate binary image as a binary mask, multiply the feature image with the display position information element by element with the binary mask to obtain the filtered feature image.

[0117] S62. Calculate the smallest bounding rectangle of the target display region contained in the filtered feature map;

[0118] S63. Crops out all feature map blocks within the smallest bounding rectangle from the filtered feature map;

[0119] S64. Adjust the cut feature blocks to a fixed size to obtain the feature blocks of the display area.

[0120] Specifically, in order to extract the feature map block containing the instrument reading region from the feature map F, this embodiment uses a binary mask RoI extractor, which uses the approximate binary map B^ generated by the reading region detection module as a binary mask to extract and cut out the reading region feature map block of fixed size and arbitrary shape.

[0121] The workflow of the binary mask RoI extractor includes the following steps:

[0122] (1) The filtered feature map is obtained by element-wise multiplying the feature map F with the binary mask to remove noisy features. Here, an approximate binary map is used as the mask, where the weight within the target display region is 1, and the weight outside the target display region is 0. It is worth noting that the size of the approximate binary map is the same as that of the feature map F.

[0123] (2) Calculate the minimum bounding rectangle of the target display region in the filtered feature map. The number of minimum bounding rectangles is equal to the number of target display regions.

[0124] (3) Cut out all feature blocks within the smallest bounding rectangle.

[0125] (4) After uniformly adjusting the cropped feature blocks to a fixed size, feature block F is obtained. roi F roi The specific dimensions can be flexibly set according to the accuracy and real-time requirements of the task.

[0126] A flowchart illustrating the process of extracting feature patches from a binary mask RoI is shown below. Figure 7 As shown in the figure, the red part of the binary mask represents the display region, with a pixel weight of 1; the blue part represents background noise, with a pixel weight of 0. After filtering out noise by element-wise multiplication of the binary mask and the feature map F, the resulting feature map retains only the feature weights corresponding to the display region, and the rest is treated as background (weight value of 0). The red rectangle is the calculated minimum bounding rectangle, with all four sides parallel to the horizontal and vertical coordinate axes.

[0127] The above process can be formally expressed as follows:

[0128]

[0129] In the formula, It is the i-th feature map patch extracted and cropped from the binary mask RoI, which is used for subsequent display recognition; This indicates that the feature patch size is adjusted to H×W using bilinear interpolation, where H and W are set to 16 pixels and 64 pixels respectively in this paper; f crop (·) indicates clipping the feature block within the smallest bounding rectangle; This represents the i-th smallest bounding rectangle in the filtered feature map. F is the input feature map of the indicator region detection module; * represents element-wise multiplication; It is an approximate binary image used as a binary mask.

[0130] The binary mask RoI extractor has two advantages: First, the binary mask of the target display region can eliminate noise features caused by the background or other non-target display regions, thereby accurately extracting features of display regions of arbitrary shapes. Second, this RoI extractor skips the spatial correction step, reducing the time cost of feature extraction.

[0131] While binary masked RoIs are somewhat similar to masked RoIs, there is a key difference. In masked RoIs, the mask is generated using a pixel aggregation method based on three components: instance vectors, text kernels, and text regions. Binary masked RoIs, however, use an approximate binary map of the same size as the feature map F to be cropped. Since this approximate binary map is itself a binary mask and is output by the region detection module, it can be directly used to filter noisy features in feature map F without any additional processing. Although binary masked RoIs cannot predict a soft mask at the text instance level for the region of interest (mask weights can be floating-point numbers) like RoI Masking, they can remove noise using the binary mask, which is easily generated by the region detection module. Furthermore, the noise weight in the binary mask is 0. Therefore, the binary masked RoI used in this embodiment can remove noisy features more thoroughly than RoIMasking.

[0132] S7. Perform reading recognition on the feature blocks of the reading area to obtain the reading content of the digital instrument.

[0133] In an optional embodiment of the present invention, step S7 includes the following sub-steps:

[0134] S71. Construct a display recognition network model; the display recognition network model includes a activator and a decoder, the activator includes a first embedding layer and a first multi-head attention layer connected to the first embedding layer, the decoder includes a second long short-term memory network layer and a second embedding layer, a second multi-head attention layer and a first long short-term memory network layer respectively connected to the second long short-term memory network layer.

[0135] S72. Use the first embedding layer of the initiator to convert the character sequence start identifier into a one-hot encoded vector;

[0136] S73. The first multi-head attention layer of the initiator extracts the feature vector of the start identifier of the character sequence based on the feature map of the display region and the one-hot encoded vector.

[0137] S74. Using the first long short-term memory network layer of the decoder, the hidden state vector of the initial time slice is obtained based on the feature vector of the character sequence start identifier and the state vector of the initial time slice.

[0138] S75. Using the second embedding layer, the second long short-term memory network layer, and the second multi-head attention layer of the decoder, the hidden state vector of the initial time slice, the character sequence start marker, and the feature map patch of the display region, the character sequence end marker is obtained by iteratively performing loops.

[0139] Specifically, this embodiment introduces a recognition head into the recognition module to achieve efficient recognition of digital instrument readings. The recognition head is a sequence-to-sequence model with a multi-head attention mechanism. It includes a starter and a decoder, such as... Figure 8 As shown.

[0140] Because the start of a character sequence in a display region of arbitrary shape is not necessarily located at the leftmost position, the goal of the starter is to find the Beginning of Sequence (BOS) position. The starter contains only one embedding layer for linear variations. and a multi-head attention layer As shown in the following formula:

[0141]

[0142] In the formula, f s This represents the feature vector output by the initiator; This indicates a multi-head attention layer. See implementation details Figure 9 ; Indicates the embedding layer; "BOS" indicates the start identifier character of the character sequence; F roi This represents the feature map output by the binary mask RoI extractor.

[0143] Embedded layer Convert the character used as the BOS identifier into a 128-dimensional one-hot encoded vector e. s Then the vector is compared with the flattened feature map F. roi Send them together into the bullish attention layer Obtain the eigenvector f of BOS s f s It is also the input of the decoder at the initial moment, such as Figure 8 As shown.

[0144] The decoder consists of only one embedding layer, two Long Short-Term Memory (LSTM) layers, and one multi-head attention layer. For details on composition and implementation, please see Figure 10 At time slice 0 (i.e., the initial time slice), the decoder will use the BOS feature vector f s The initial state of the LSTM (set to all zeros) is used as input. At time slice 1, the hidden state h0 of time slice 0 and the BOS identifier are input into the LSTM together, and the output identifier y1 of time slice 1 is predicted. Subsequently, the output identifier from the previous step is fed into the LSTM until the End of Sequence (EOS) symbol is predicted. During training, the decoder's input identifiers are the characters in the truth label sequence. The above steps can be described by the following formula:

[0145] h0 = LSTM(f s ,0)

[0146]

[0147]

[0148] Where h0 represents the hidden state of time slice 0; LSTM(·) represents the LSTM layer; f s represents the feature vector output by the initiator; 0 represents the all-zero initial state of the LSTM. h t This represents the hidden state at time slice t (t>0); The embedding layer represents the character identifiers. It's important to note that after all character identifiers (including BOS and EOS) undergo linear transformation and flattening at the embedding layer, they are represented using one-hot encoded vectors. t The character identifier output indicates the time slice t; ARGMAX(·) represents the argmax function; FC(·) represents a fully connected layer; The representation structure is different Another multi-head attention layer; F roi This represents the feature map output by the binary mask RoI extractor, such as... Figure 7 As shown.

[0149] The recognition head used in this embodiment can effectively fuse temporal (i.e., LSTM layer) features and visual (i.e., CNN layer) features. Furthermore, the computational cost of the multi-head attention module is less than that of the ordinary attention module used in SAR.

[0150] In an optional embodiment of the present invention, an end-to-end training process simultaneously supervises both the detection module and the recognition module, and the weight values ​​for detection and recognition are also updated uniformly. The trained model parameters and weights are only one set, which enables the integration of the detection model and the recognition model.

[0151] Because the generation of probabilistic maps, threshold maps, and approximate binary maps requires supervised training, it is necessary to convert the annotation information of text instances in the training set into ground truth labels for each of the three maps. This operation is commonly referred to as label generation. This embodiment proposes a label generation method for threshold maps, probabilistic maps, and approximate binary maps. Figure 11 As shown, this example diagram contains only one instance of numerical text enclosed by a polygon. The annotations of the text instance are visualized by red lines. By shrinking and expanding the labeled polygon W, a shrunken polygon W can be obtained, visualized by yellow and green lines. S and the extended polygon W d These are generated by shrinking and expanding the labeled polygons using the offset Δ, respectively. Figure 12 This is a schematic diagram of a quadrilateral label being generated.

[0152] The labeled polygon W for each text instance is composed of a set of edges E i It is composed of the following formula:

[0153]

[0154] In the formula, E i denoted by , v represents the number of edges that make up polygon W; v is the number of vertices; and i is the index of each edge of the polygon, ranging from 1 to v. The number of vertices v of the polygon depends on the annotation method of the dataset. For example, each polygon in the ICDAR 2015 dataset contains 4 vertices, while the labeled polygons in the digital instrumentation dataset created in this embodiment contain between 4 and 8 vertices. This embodiment uses the Vatti clipping algorithm to shrink the labeled polygon W to W0. s This is used to generate the foreground region. The shrinkage offset Δ can be expressed by the following formula:

[0155]

[0156] In the formula, A represents the area of ​​the labeled polygon, and L represents the perimeter of the labeled polygon. The shrinkage rate r is a hyperparameter that needs to be manually set, and this value often varies depending on the dataset. If the shrinkage rate is set too large, the label area will be too large; if it is set too small, the label area will be too small. Therefore, only by setting an appropriate value can the text detection performance be avoided. In the probabilistic map label, located at W... S The value corresponding to the middle pixel is set to 1, while other positions are set to 0. The approximate binary map labels are generated in the same way.

[0157] Table 1. Value ranges for the three types of labels

[0158]

[0159]

[0160] Similar to the generation of probabilistic map labels, the generation of threshold map labels requires first expanding the labeled polygon W to W' using the same offset Δ. d Then, the value of each pixel position in the threshold map label is calculated using W. s and W d The distance from each pixel in the interval (the boundary of the text instance) to the nearest edge of W is obtained. As... Figure 12 As shown, the labeled polygon W is visualized using a red line. Shrinking polygon W... s and the extended polygon W d They are represented by yellow and green lines, respectively. The value ranges of the labels for the probability plot, approximate binary plot, and threshold plot are shown in Table 1.

[0161] The total loss function used in this embodiment during model training It can be expressed as

[0162]

[0163] In the formula, The loss function represents the reading region detection module; This represents the loss function of the indicator recognition module.

[0164] The loss function of the indicator detection module It consists of three parts: the loss function of the probabilistic graph. Loss function of threshold map Loss function of approximate binary graph Represented as:

[0165]

[0166] In the formula, This represents the loss of the probabilistic graph; α is the weighting of the approximate binary graph, which is manually set to 1. This represents an approximate binary image; β is the weighting of the threshold image, which is empirically set to 5. This represents the loss of the threshold map.

[0167] For both probabilistic graphs and approximate binary graphs, the same binary cross-entropy function is used to calculate their respective losses, expressed as:

[0168]

[0169] In the formula, The predicted value representing the probability that "pixel i is a positive sample"; y i That is the predicted value The corresponding tag value; S p It is a sample set obtained from difficult negative sample sampling.

[0170] To address the imbalance between positive and negative samples, hard negative sample mining is employed in the binary cross-entropy loss function to sample negative samples. The hard negative sample mining strategy uses a bootstrap approach, progressively selecting only those negative samples with high practical value (difficult to detect). The specific process is as follows:

[0171] (1) First, use an initial negative sample training set to train the classifier. At this time, in order to balance the data, the negative samples used are only a subset of all negative samples.

[0172] (2) Use the classifier trained in (1) to classify the samples, and put the misclassified samples (i.e. hard samples) into the negative sample subset to form a set of hard negative samples.

[0173] (3) Use the new negative sample subset to continue training the new classifier.

[0174] (4) Repeat the above steps several times until the classifier performance no longer improves.

[0175] To create positive and negative training sets based on labeled data, image patches are randomly generated using labeled images. Patches with an IoU (Intersection over Union) greater than a threshold (set to 0.8 in this embodiment) are considered positive samples; otherwise, they are considered negative samples. However, negative samples generally far outnumber positive samples. To prevent the trained classifier from biasedly predicting negative examples, sample balance needs to be maintained. Therefore, the initial negative sample training set needs to be sampled from the negative sample set to form a negative sample subset. The ratio of positive to negative samples during sampling is 1:3. After the first round of training, the trained classifier is used to predict the remaining negative samples (i.e., the negative samples not added to the training set). After predicting an image patch, the classifier provides the probability of whether it is positive or negative. A threshold is set here (0.6 in this article). If the probability of predicting positive is greater than this threshold, the image patch is added to the negative sample training set. In this way, the positive sample training set remains unchanged, while the negative sample training set includes the initial portion as well as newly added incorrectly predicted samples. Finally, a new round of training is started using the newly formed training set until the classifier's performance no longer improves.

[0176] For the threshold map, L1 loss is used as the loss function. The L1 loss is used to calculate the loss over the expanded polygon W. d The sum of the L1 distances between the predicted values ​​and the label values ​​is expressed as:

[0177]

[0178] In the formula, n is the extended polygon W d The number of pixels in the array; i is the number of each pixel; The predicted value of the threshold map That is the corresponding tag value.

[0179] Loss function of the display recognition module The Cross Entropy Loss function is introduced, and its calculation method is as follows:

[0180]

[0181] In the formula, N represents the number of characters in the current transcribed sequence; K represents the number of characters in the tag dictionary; y i,k The truth value of the label for the i-th character in the transcribed sequence is k; p i,k This represents the probability that the i-th character is predicted to be k.

[0182] The overall pipeline of the digital instrument reading end-to-end recognition method in this embodiment of the invention, through the above design, can effectively reduce the time overhead of each step, thereby better realizing an instrument reading recognition system that meets real-time recognition requirements. Thanks to the above pipeline design, the digital instrument reading end-to-end recognition method can achieve high inference speed while maintaining competitive accuracy.

[0183] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0184] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0185] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0186] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

[0187] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A digital instrument display end-to-end recognition method, characterized in that, Includes the following steps: S1. Obtain the original digital instrument reading image; S2. Extract multiple feature maps from the original digital instrument reading image to form an initial feature pyramid; S3. Perform multi-scale and multi-level feature fusion on the initial feature pyramid to obtain a feature map with numerical location information; S4. Perform image segmentation on the feature map containing the display position information to generate a probability map and a threshold map; S5. Based on the probability map and threshold map, the indicator position is predicted using an adaptive differentiable binarization method to obtain an approximate binary map; wherein, a differentiable binarization function adaptively approximates the standard binarization function as the model is trained. The curvature, which was previously a fixed value in a differentiable binary function, is transformed into a trainable parameter. The calculation method is as follows: In the formula, Input for the function; It is the hyperbolic tangent function; It is a logarithmic function; It is a similarity factor. ,when If it is small enough, the approximate function can approximate the standard binary function; In form, the adaptive training process is treated as a loss minimization problem of the segmentation network, as shown in the following equation. Indicates network input. This indicates the corresponding network output: Therefore, it is possible to calculate The gradient during backpropagation, which is automatically adjusted using L2 regularization constraints, is formally represented as follows: Therefore, in the differentiable binarization module, the differentiable binarization function can be transformed into the following expression as the differentiable binarization function used by RetinaDB: In the formula, Represents an approximate binary graph; and These represent the horizontal and vertical coordinates of each pixel in the three images: the approximate binary image, the probability image, and the threshold image, respectively. It is the hyperbolic tangent function; It is a logarithmic function; It represents the similarity factor; P represents the probability graph; T represents the threshold graph; After the detection head predicts the probability map P and the threshold map T, the height and width of P and T are the same as the input image. An adaptive differentiable binarization function is used to calculate an approximate binary image where the weight value of each pixel can only be 0 or 1, by taking the probability map P and the threshold map T as input. In the approximate binary image, a pixel with a weight of 1 represents that the pixel belongs to the instrument reading region and is used as the foreground for detection; a pixel with a weight of 0 represents that the pixel does not belong to the instrument reading region and is used as the background for detection. The approximate binary image is used as a binary mask containing the instrument reading region for feature patch extraction. S6. Using an approximate binary map as a binary mask, extract feature map blocks of the display region from the feature map containing display position information; S7. Perform reading recognition on the feature blocks of the reading area to obtain the reading content of the digital instrument; including... S71. Construct a display recognition network model; the display recognition network model includes a activator and a decoder, the activator includes a first embedding layer and a first multi-head attention layer connected to the first embedding layer, the decoder includes a second long short-term memory network layer and a second embedding layer, a second multi-head attention layer and a first long short-term memory network layer respectively connected to the second long short-term memory network layer. S72. Use the first embedding layer of the initiator to convert the character sequence start identifier into a one-hot encoded vector; S73. The first multi-head attention layer of the initiator extracts the feature vector of the start identifier of the character sequence based on the feature map of the display region and the one-hot encoded vector. S74. Using the first long short-term memory network layer of the decoder, the hidden state vector of the initial time slice is obtained based on the feature vector of the character sequence start identifier and the state vector of the initial time slice. S75. Using the second embedding layer, the second long short-term memory network layer, and the second multi-head attention layer of the decoder, the hidden state vector of the initial time slice, the character sequence start marker, and the feature map patch of the display region, the character sequence end marker is obtained by iteratively performing loops.

2. The digital instrument display end-to-end identification method according to claim 1, characterized in that, Step S2 includes the following sub-steps: S21. A lightweight deep convolutional neural network is used to extract multiple feature maps of different scales from the original digital instrument display images; S22. The extracted feature maps of different scales are converted into feature maps with a uniform number of channels through convolution operation. S23. Construct an initial feature pyramid based on the feature map of the unified number of channels.

3. The digital instrument display end-to-end identification method according to claim 1, characterized in that, Step S3 includes the following sub-steps: S31. Perform a downward fusion operation on the feature maps of each group of adjacent layers in the initial feature pyramid to obtain the feature pyramid after downward fusion. S32. Perform bottom-up path enhancement operation on the feature maps of each group of adjacent layers in the feature pyramid after downward fusion to obtain the feature pyramid after upward enhancement. S33. Perform convolution operation on each feature map in the up-enhanced feature pyramid to obtain a feature map with the number of channels reduced to a set ratio, then upsample it to a set size, and concatenate the processed feature maps to obtain a feature map with display position information.

4. The digital instrument display end-to-end identification method according to claim 3, characterized in that, Step S31 includes the following sub-steps: S311. For each group of adjacent feature maps in the initial feature pyramid, perform double bilinear interpolation upsampling on the high-level feature map to obtain a feature map with the same size as the low-level feature map. S312. Add the feature map obtained in step S311 to the low-level feature map element by element to obtain the first fused feature map; S313. Perform a depthwise separable convolution operation on the first fused feature map to obtain a single-channel feature map with the same number of channels as the first fused feature map. S314. Perform pointwise convolution operations on each single-channel feature map obtained in step S313, and weight and combine all single-channel feature maps in the depth direction to generate the second fused feature map.

5. The digital instrument display end-to-end identification method according to claim 1, characterized in that, Step S4 includes the following sub-steps: S41. Construct a display region segmentation network model; the display region segmentation network model includes a cascaded first convolutional layer, a first batch of normalized layers, a first ReLU activation layer, a first deconvolutional layer, a second batch of normalized layers, a second ReLU activation layer, a second deconvolutional layer, a third batch of normalized layers, and a third ReLU activation layer; S42. The feature map with display position information is processed using the first convolutional layer, the first batch of normalization layers, and the first ReLU activation layer to obtain the first segmentation feature map; S43. The first segmentation feature map is processed using the first deconvolution layer, the second batch normalization layer, and the second ReLU activation layer to obtain the second segmentation feature map; S44. The second segmentation feature map is processed using the second deconvolution layer, the third batch normalization layer, and the third ReLU activation layer to generate a probability map and a threshold map.

6. The digital instrument display end-to-end identification method according to claim 1, characterized in that, Step S6 includes the following sub-steps: S61. Using an approximate binary image as a binary mask, multiply the feature image with the display position information element by element with the binary mask to obtain the filtered feature image. S62. Calculate the smallest bounding rectangle of the target display region contained in the filtered feature map; S63. Crops out all feature map blocks within the smallest bounding rectangle from the filtered feature map; S64. Adjust the cut feature blocks to a fixed size to obtain the feature blocks of the display area.

Images