An improved handwritten slab number recognition method based on attention mechanism

Abstract

The application belongs to the technical field of automatic control in the production process of steel enterprises, and proposes an improved handwritten slab number recognition method based on an attention mechanism, which pre-processes handwritten slab number image data in combination with a complex production scene to ensure the quality of model input pictures; extracts picture features by using a residual neural network module; extracts sequence features based on images by using a bidirectional long short-term memory network; and finally captures information flow within the sequence based on the attention mechanism, integrates the features of each character to form a text feature vector to predict the output sequence. Through field testing, the accuracy of the steel plate surface handwritten plate number recognition task reaches 93.49%, and the results show that the algorithm is feasible and effective, and meets the actual production requirements.

Description

Technical Field

[0001] This invention relates to the field of automatic control technology for steel production processes, and in particular to an improved handwritten slab number recognition method based on an attention mechanism. Background Technology

[0002] During the production process, steel plate manufacturers assign a unique string of characters to each steel plate as its slab number. This number reflects information such as the steel plate's manufacturing process, material, production stage, and intended use, playing a crucial role in steel plate quality traceability and inventory management.

[0003] Currently, many domestic steel companies still encode steel plates by handwriting slab numbers on the surface. Testing has shown that there is currently no effective intelligent recognition method for handwritten slab numbers. Taking the best CTC-based text recognition method in "Shi B, Bai X, Yao C. An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text Recognition[J].IEEE Transactions on Pattern Analysis & Machine Intelligence, 2016, 39(11):2298-2304." as an example, the standard CTC loss calculation for character recognition prediction assumes internal label conditional independence, uses Bayes' theorem to calculate the posterior distribution, and outputs the probability of a single character each time, resulting in only obtaining local information for prediction. Ultimately, the accuracy of the CTC-based recognition method cannot meet the recognition accuracy requirements of steel mills. Summary of the Invention

[0004] Therefore, it is necessary to propose an improved handwritten slab number recognition method based on an attention mechanism to address the above problems, which can accurately identify handwritten slab numbers from steel mills.

[0005] The technical solution of the present invention is as follows: An improved handwritten slab number recognition method based on an attention mechanism, comprising the following steps:

[0006] Step 1: Filter the video stream of the steel slab moving on the roller conveyor to obtain the handwritten slab number image;

[0007] Step 2: Preprocess the handwritten slab number image obtained in Step 1 to obtain the training set, validation set, and test set;

[0008] Step 3: Based on identity mapping, feature extraction is performed on the preprocessed handwritten slab number image to obtain a feature map group;

[0009] Step 4: Perform serialized feature calculation on the feature map group obtained in Step 3, output the sequence weight, and simultaneously map the feature map group to obtain the serialized feature vector, which serves as the sequence vector;

[0010] Step 5: Design a feature decoder based on the attention mechanism, use the feature decoder to calculate the joint probability to obtain the matching degree between the sequence vector and the content of the known dictionary, and output the predicted character sequence;

[0011] Step 6: Repeat steps 3 to 5 to iterate and obtain the prediction model. Use the prediction model for autonomous iterative annotation. Use the autonomous iterative annotation results to expand the training set and update the prediction model to obtain the updated prediction model.

[0012] Step 7: Input the sample to be identified, and obtain the actual output result based on the updated prediction model obtained in Step 6.

[0013] The specific method for step 1 is as follows:

[0014] Step 1.1: The video stream camera is installed at a horizontal position of 7000-8000mm and a height of 8000-9000mm from the roller conveyor, with a downward field of view of not less than 26000mm;

[0015] Step 1.2: Construct a filter based on the inter-frame difference method, and compare the differences between consecutive frames to obtain an image containing the handwritten slab number.

[0016] The specific method for step 2 is as follows:

[0017] Based on the probabilistic Hough transform, the straight line fragments, noise, and other coexisting non-linear structural parts of the handwritten slab number image are extracted to obtain the outlines of the lower edge and side edges of the steel plate, and the output is the first-processed image. The first-processed image is then corrected by perspective transformation to convert it into a top view and pixel size correspondence is performed to obtain the second-processed image. Then, the edges are extracted using the Gaussian derivative, and the foreground of the image is extracted based on maximizing the inter-class variance.

[0018] In step 3, feature maps are extracted and feature map groups (FS) are constructed based on the identity mapping. The specific method is as follows:

[0019] The training, validation, and test sets are divided in a 7:2:1 ratio. An identity mapping is performed using a residual neural network module with a cross-layer connection structure, which simultaneously extracts images from the training set to obtain multiple attributes of continuous text regions and construct different feature maps, including text offset (TDC), boundary offset (EDC), center deviation (CDC) value, and sequence region feature matrix (SRE). The residual neural network module with a cross-layer connection structure includes various convolutional modules, pooling modules, residual modules, network concatenation modules, and network parallel basic modules. Multiple feature maps are combined to generate a feature map group (FS), which is used to describe the central axis offset distance of pixels in the handwritten slab number image, the offset of horizontal and vertical boundaries from the center point in the foreground text, the deviation value between the text center point and the region center, and the local feature extraction of the sequence region.

[0020] Step 4 specifically involves:

[0021] The feature map group FS is serialized, and the complete feature map group is horizontally segmented to form a continuous sequence of preceding and following text. A bidirectional long short-term memory (BiLSTM) network layer is introduced to extract the continuous sequence information of the segmented preceding and following text, and multi-layer parameter sharing is performed to update the sequence weights W of the feature map group. FS The bidirectional long short-term memory network consists of two independent LSTM networks, one for forward processing of feature sequences. Another one performs reverse sequence calculations. The two LSTM layers share parameters and are connected to the same output layer. Since the standard CTC loss calculation method is abandoned, the feature map groups need to undergo uniform input length correction; all feature maps need to be corrected to the same length before feature sequence segmentation.

[0022] Step 5 designs a feature decoder based on an attention mechanism to calculate the continuous sequence of preceding and following text. The feature decoder consists of three parts and five sub-layers presented in the code in a class structure, respectively performing positional information decoding, semantic information decoding, and calculation of the correlation between preceding and following text. The calculation results are stored and transmitted using high-dimensional matrix properties. Through the above feature decoder, the textual correlation of two consecutive character features is expanded to the long-distance textual correlation of continuous text features throughout the entire text, so as to capture the correlation relationship of all foreground pixels. The joint probability P is calculated. att We obtain the output character probability, which is the numerical value of the match between the sequence vector and the dictionary content. Let the output at the current step t be y. t y t-1 The output for step (t-1) can be represented as:

[0023] y t =softmax(W o s t +b0) (1)

[0024] Where W0 and b0 are trainable parameters, s t The hidden state of the LSTM layer at time t:

[0025] s t =LSTM(y t-1 c t s t-1 (2)

[0026] Context vector c t The output from the previous stage is H = {h} i The weighted sum of |i=1,2,3,...,I} is calculated to obtain:

[0027]

[0028] Where, α ti The joint weights are calculated as follows:

[0029]

[0030] Parameter e ti =v T tanh(Ws t-1 +Vh i +b) reflects h i Matching degree with other elements, e ti The larger the value, the greater the influence of the element, and the greater the weight of the element; finally, the predicted output sequence Y = {y∈D|t = 1, 2, 3, ..., T} is obtained, where D is a dictionary containing all characters.

[0031] The beneficial effects of this invention are:

[0032] (1) The method proposed in this invention fills the gap in the current identification methods and solves the problem that handwritten slab numbers cannot be identified in the steel plant production process.

[0033] (2) The attention mechanism-based method proposed in this invention can ensure character integrity, eliminates the need to label the position information of each character, improves data labeling and model training efficiency, has high accuracy, and meets the actual production needs. Attached Figure Description

[0034] Figure 1 This is an example image of a handwritten slab number captured on-site in one embodiment;

[0035] Figure 2 Here is a flowchart of the handwritten slab number data processing in one embodiment;

[0036] Figure 3 This is a diagram illustrating the structure of a recognition algorithm used in an improved handwritten slab number recognition method based on an attention mechanism. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0038] This invention provides an improved handwritten slab number recognition method and system based on an attention mechanism. The handwritten slab number recognition method makes up for the lack of handwritten recognition function in various current slab number recognition methods.

[0039] The entity performing the identification includes, but is not limited to, at least one of electronic devices that can be configured to perform the methods provided in the embodiments of the present invention, such as servers, graphics workstations, and terminal devices.

[0040] Step 1: Obtain the handwritten slab number image to get the handwritten slab number data.

[0041] like Figure 1 As shown in the figure, this invention provides a field deployment hardware topology diagram for a handwritten text recognition method. The method includes: a video shooting device installed on the side of the roller conveyor to capture video of the slab's movement; the video signal is converted into an image signal by a video processing model; the image signal undergoes character positioning to locate the position of the slab number character in the entire slab image; and the area to be recognized is extracted by a text detection module. The video stream camera is mounted at a horizontal position of 7000-8000mm and a height of 8000-9000mm from the roller conveyor, with a top-down field of view of not less than 26000mm. Furthermore, a filter is constructed based on the inter-frame difference method, and the differences between consecutive frames are compared to obtain image data containing the handwritten slab number.

[0042] Step 2: Perform data preprocessing on the acquired images.

[0043] like Figure 2 As shown, the data preprocessing workflow includes real-time image acquisition, tilt correction, denoising, enhancement, and normalization. Image processing is performed based on the characteristics of the samples: First, to extract small target text from the image, the probabilistic Hough transform is used to extract the lower and side edges of the steel plate, leveraging its insensitivity to straight line fragments, noise, and other coexisting non-linear structures in the image. This coarsely locates the target string, and then the Canny edge detection algorithm is used to truncate the target text, resulting in a first-processed image. Next, to correct the distortion of the camera-captured image, perspective transformation is used for tilt correction, converting the tilted image into a top view and performing pixel size mapping. The model can be abstracted as follows:

[0044]

[0045] In the formula: [uvw] and [x′ y′ w′] are the images before and after the transformation, respectively. Let be the transformation matrix, where Representing a linear transformation, T2 = [a 13 a 23 ] T Used to generate image perspective transformation; T3 = [a 31 a 32 Used to translate images.

[0046] This invention introduces variance calculation to find the grayscale value with the largest variance, and then calculates correlation to divide the image into foreground and background parts. This design primarily aims to solve the problem of text content omission when the chalk writing and background color are similar, caused by the complexity of industrial environments and the unevenness of handwritten slab numbers, making it difficult to adapt to changes in lighting in different areas using a global threshold. This design method can be abstractly represented as:

[0047] G = W0 × (U0 - U) 2 +W1×(U1-U) 2 =W0×W1×(U0-U1) 2 2)

[0048] Where U = U0 × (W1 + W0). W0 represents the proportion of foreground pixels to total pixels, and U0 is its average gray level; W1 represents the proportion of foreground pixels to total pixels, and U1 is its average gray level.

[0049] Step 3: Extract feature maps based on the identity mapping idea and construct a feature description map group (FS).

[0050] Invalid features are mapped using a residual neural network module with a cross-layer connection structure. This module simultaneously extracts multiple attributes from continuous text regions, including text offset, boundary offset, center deviation, and sequence region feature matrix, denoted as TDC, EDC, CDC, and SRE, respectively. The network internally includes various basic modules such as convolution, pooling, residual, concatenated networks, and parallel networks. The input size is denoted as N*H*W*C, representing the number of images captured at one time, their length, width, and number of channels, respectively. Multiple feature maps are combined to generate a feature map set FS, which is used to describe the central axis offset distance of pixels in the handwritten slab number image, the offset of horizontal and vertical boundaries from the center point in the foreground text, the deviation between the text center point and the region center, and the local feature extraction of the sequence region.

[0051] Steps 4 and 5: To achieve a higher accuracy than the current best feature processing algorithm CTC, this invention uses an attention mechanism to jointly calculate the probability of the output character, thus predicting the handwritten slab number. The specific steps are as follows:

[0052] The obtained feature map group FS is serialized, and the complete feature map group is horizontally segmented to form a continuous sequence with preceding and following context. A BiLSTM layer is introduced to extract the preceding and following context information after segmentation, and multi-layer parameter sharing is performed to update the sequence weight W of the feature map group. FS This design allows input features to be directly passed to the output of this layer, added to the convolutional features, and together they form the output of this layer, effectively avoiding the gradient vanishing problem that easily occurs in input image feature extraction. Furthermore, using a bidirectional long short-term memory network to obtain some information from both the forward and backward directions can further improve accuracy. The bidirectional long short-term memory network consists of two independent LSTM networks, one for forward processing of the feature sequence. Another one performs reverse sequence calculations. The two LSTM layers share parameters and are connected to the same output layer.

[0053] Inspired by the attention mechanism, step 5 of this invention calculates the aforementioned feature sequence, expanding the text association of short paths to capture the association relationships of all foreground pixels along long paths. This is achieved by calculating the joint probability P. att The output character probability is obtained, which is the numerical value of the matching degree between the sequence vector and the dictionary content. Specifically, the long sequence learning capability of this design is utilized to automatically capture the information flow within the input sequence of this layer, integrating the features of each character to form a text feature vector. At step t, the decoder of this layer predicts the output as y. t :

[0054] y t =soffmax(W0s t +b0) 3)

[0055] Where W0 and b0 are trainable parameters, s t The decoder's LSTM hidden state at time t:

[0056] s t =LSTM(y t-1 c t s t-1 ) 4)

[0057] Context vector c t The output from the previous stage is H = {h} i The weighted sum of |i=1,2,3,...,I} is calculated to obtain:

[0058]

[0059] Where α ti The joint weights are calculated as follows:

[0060]

[0061] Parameter e ti =v T tanh(Ws t-1 +Vh i +b) reflects h i Matching degree with other elements, e ti The larger the value, the greater the influence of the element, and the greater its corresponding weight. The final predicted output sequence is Y = {y...} t ∈D|t=1,2,3,...,T}, where D is a dictionary containing all characters.

[0062] Step 6: Use the algorithm proposed in this invention to train the recognition model, and use the pre-trained model to perform initial labeling of the new data, expand the existing database, and realize automatic iterative optimization of the model.

[0063] In this embodiment, a database is built using existing data for model pre-training. New data is then identified using the pre-trained model. After manual review, the data is used for further model training. This not only enables automatic iterative optimization of the model, further improving recognition accuracy, but also expands the production database. The training parameters are set as follows:

[0064]

[0065] Because the initial database is small, the limited training data is insufficient for model convergence. Therefore, data augmentation is necessary, primarily involving stretching, distortion, and noise addition to enrich the model input. New annotations are handled by a predictive model instead of online OCR tools, automatically generating labels in a specific format which are then manually corrected. The corrected data is added to the database and used for model training to improve accuracy and robustness. This process significantly reduces the workload of manual annotation and saves time.

[0066] Finally, to verify that the improved handwritten slab number recognition method based on the attention mechanism proposed in this invention has a higher recognition accuracy than the current best CTC-based algorithm, the two model structures were tested on four public datasets and a handwritten slab number dataset, and the best results were achieved in both cases.

[0067]

[0068] Comparing the performance of the two algorithms on the handwritten slab number dataset, it can be found that the improved handwritten slab number recognition method based on the attention mechanism proposed in this invention achieves a 3.97% higher accuracy than the CTC algorithm. On the public dataset CT, the recognition accuracy is improved by 2.3%, indicating that the algorithm is better suited to recognizing curved text. This characteristic aligns with the features of handwritten slab number datasets in actual production, and it also performs better when faced with irregularly arranged handwritten text.

[0069] In summary, the improved handwritten slab number recognition method based on an attention mechanism proposed in this invention can achieve a high recognition accuracy. In complex production environments where the text on the steel plate surface is handwritten, the algorithm fills the gaps in current recognition methods and effectively solves the problem of difficulty in recognizing handwritten handwriting on steel plate surfaces during production.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to depart from the scope defined by the claims of the present invention.

Claims

1. An improved handwritten slab number recognition method based on an attention mechanism, characterized in that, Includes the following steps: Step 1: Filter the video stream of the steel slab moving on the roller conveyor to obtain the handwritten slab number image; Step 2: Preprocess the handwritten slab number image obtained in Step 1 to obtain the training set, validation set, and test set; Step 3: Based on identity mapping, feature extraction is performed on the preprocessed handwritten slab number image to obtain a feature map group; Step 4: Perform serialized feature calculation on the feature map group obtained in Step 3, output the sequence weight, and simultaneously map the feature map group to obtain the serialized feature vector, which serves as the sequence vector; Step 5: Design a feature decoder based on the attention mechanism, use the feature decoder to calculate the joint probability, obtain the matching degree between the sequence vector and the content of the known dictionary, and output the predicted character sequence; Step 6: Repeat steps 3 to 5 to iterate and obtain the prediction model. Use the prediction model for autonomous iterative annotation. Use the autonomous iterative annotation results to expand the training set and update the prediction model to obtain the updated prediction model. Step 7: Input the sample to be identified, and obtain the actual output result based on the updated prediction model obtained in Step 6; In step 3, feature maps are extracted and feature map groups (FS) are constructed based on the identity mapping. The specific method is as follows: The training, validation, and test sets are divided in a 7:2:1 ratio. An identity mapping is performed using a residual neural network module with a cross-layer connection structure, which simultaneously extracts images from the training set to obtain multiple attributes of continuous text regions and construct different feature maps, including text offset (TDC), boundary offset (EDC), center deviation (CDC) value, and sequence region feature matrix (SRE). These feature maps are combined to generate a feature map group (FS), which is used to describe the central axis offset distance of pixels in the handwritten slab number image, the offset of horizontal and vertical boundaries in the foreground text from the center point, the deviation value between the text center point and the region center, and the local feature extraction of the sequence region.

2. The improved handwritten slab number recognition method based on attention mechanism according to claim 1, characterized in that, The specific method for step 1 is as follows: Step 1.1: The video stream camera is installed at a horizontal position of 7000-8000mm and a height of 8000-9000mm from the roller conveyor, with a downward field of view of not less than 26000mm; Step 1.2: Construct a filter based on the inter-frame difference method, and compare the differences between consecutive frames to obtain an image containing the handwritten slab number.

3. The improved handwritten slab number recognition method based on attention mechanism according to claim 1, characterized in that, The specific method for step 2 is as follows: Based on the probabilistic Hough transform, the straight line fragments, noise, and other coexisting non-linear structural parts of the handwritten slab number image are extracted to obtain the outlines of the lower edge and side edges of the steel plate, and the output is the first-processed image. The first-processed image is then corrected by perspective transformation to convert it into a top view and pixel size correspondence is performed to obtain the second-processed image. Then, the edges are extracted using the Gaussian derivative, and the foreground of the image is extracted based on maximizing the inter-class variance.

4. The improved handwritten slab number recognition method based on attention mechanism according to claim 1, characterized in that, The residual neural network module with cross-layer connection structure includes various convolutional modules, pooling modules, residual modules, network terminal concatenation modules, and network parallel basic modules.

5. The improved handwritten slab number recognition method based on attention mechanism according to claim 1, characterized in that, Step 4 specifically involves: The feature map group FS is serialized, and the complete feature map group is horizontally segmented to form a continuous sequence of preceding and following text. A bidirectional long short-term memory (BiLSTM) network layer is introduced to extract the continuous sequence information of the segmented preceding and following text, and multi-layer parameter sharing is performed to update the sequence weights W of the feature map group. FS The bidirectional long short-term memory network consists of two independent LSTM networks, one for forward processing of feature sequences. Another one performs reverse sequence calculations. The two LSTM layers share parameters and are connected to the same output layer.

6. The improved handwritten slab number recognition method based on attention mechanism according to claim 5, characterized in that, Step 5 designs a feature decoder based on an attention mechanism to calculate the continuous sequence of preceding and following text. The feature decoder consists of three parts: positional information decoding, semantic information decoding, and preceding and following text correlation calculation. The calculation results are stored and transmitted using high-dimensional matrix properties. The feature decoder expands the textual correlation between two consecutive character features to long-distance textual correlation of continuous text features throughout the entire text, capturing the correlation relationships of all foreground pixels. The joint probability P is calculated... att Obtain the output character probability, which is the numerical value of the matching degree between the sequence vector and the dictionary content; let the output at the current step t be y. t y t-1 The output for step (t-1) is represented as: y t =softmax(W0s t +b0) (1) Where W0 and b0 are trainable parameters, s t The hidden state of the LSTM layer at time t: s t =LSTM(y t-1 ,c t ,s t-1 ) (2) Context vector c t The output from the previous stage is H = {h} i The weighted sum of |i=1,2,3,…,I} yields: Where, α ti The joint weights are calculated as follows: Parameter e ti =v T tanh(Ws t-1 +Vh i +b) Reaction h i Matching degree with other elements, e ti The larger the value, the greater the influence of the element, and the greater the weight of the element; ultimately, the predicted output sequence Y = {y} is obtained. t ∈D|t=1,2,3,…,T}, where D is a dictionary containing all characters.

Images