Method, device, storage medium and processor for recognizing characters of a steel seal

Abstract

Embodiments of the present application provide a method and device for identifying characters on a steel seal, a storage medium and a processor. The method comprises: obtaining a steel seal character image to be identified; inputting the steel seal character image into an image character positioning model, determining a rotation angle of a detection box for the steel seal character in the steel seal character image through an angle prediction branch of the image character positioning model, and cutting out a region image containing the steel seal character in the steel seal character image based on the rotation angle; obtaining the region image output by the image character positioning model as a target region image; and inputting the target region image into a character recognition module to output character information of the steel seal character in the target region image through the character recognition module. Through intelligent visual means, the steel seal characters engraved on the surface of a metal object can be automatically identified with high precision and at high speed, which can greatly reduce the labor cost and time cost, improve the management efficiency of products, and realize the optimized control of the production process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, and in particular to a steel stamp character recognition method and device, a mobile device, a storage medium and a processor. BACKGROUND

[0002] In the production and manufacturing process of large engineering machinery, steel stamp characters are engraved on the surface of products and metal parts thereof, so as to realize quantity statistics, vehicle assembly, quality supervision and tracking management of the products through one product one code. The steel stamp characters are usually engraved on the surface of metal objects by the on-site operator using a steel stamp marking device. If the operator is negligent and the engraved characters are wrong, it will seriously affect the transportation and delivery of the products, causing economic losses. At present, steel stamp character recognition is mainly realized through manual inspection. The quality inspection personnel sent by the enterprise read and write the steel stamp label, and then compare it with the steel stamp task table and inspection records issued according to the plan to check whether the steel stamp production label has any engraved errors or repeated engraved situations. However, the manual reading of steel stamp characters has the disadvantages of complicated steps, low efficiency and strong subjectivity. In the context of large-scale industrial production, employees will make recognition errors due to visual fatigue caused by high-intensity work. The cost of rework found in the later stage will be greatly increased, and if the products flow out, it will also cause a large amount of economic losses, which cannot meet the requirements of intelligent and efficient production control. SUMMARY

[0003] The purpose of the embodiments of the present application is to provide a steel stamp character recognition method and device, a mobile device, a storage medium and a processor.

[0004] In order to achieve the above-mentioned purpose, the first aspect of the present application provides a steel stamp character recognition method, which comprises the following steps:

[0005] obtaining a steel stamp character image to be recognized;

[0006] inputting the steel stamp character image into an image character positioning model, determining a rotation angle of a detection box of the steel stamp character in the steel stamp character image through an angle prediction branch of the image character positioning model, and cutting out a region image containing the steel stamp character in the steel stamp character image based on the rotation angle;

[0007] obtaining the region image output by the image character positioning model as a target region image;

[0008] inputting the target region image into a character recognition module to output character information of the steel stamp character in the target region image through the character recognition module.

[0009] In an embodiment of the present application, a stamped character image is input into an image character positioning model, and the rotation angle of a detection frame for the stamped characters in the stamped character image is determined through the angle prediction branch of the image character positioning model, including: inputting the stamped character image into the image character positioning model, and extracting multiple first candidate frames corresponding to the stamped character image through the region candidate network of the image character positioning model; predicting a first prediction probability of each first candidate frame being a foreground through the classification branch of the image character positioning model, wherein the foreground refers to a frame containing stamped characters; determining a detection frame for the stamped characters in the stamped character image based on the first candidate frame whose first prediction probability is greater than a preset threshold; and determining the rotation angle of the detection frame through the angle prediction branch of the image character positioning model.

[0010] In an embodiment of the present application, the recognition method also includes a training step of an image character positioning model, the training step including: obtaining multiple sample images with stamped characters, and marking the character area where the stamped characters are located in each sample image; for each sample image, extracting multiple second candidate frames; for each sample image, predicting the second prediction probability of each second candidate frame being the foreground or background through the classification branch of the image character positioning model, wherein the foreground refers to the area containing the stamped characters, and the background refers to the area not containing the stamped characters; for each sample image, determining the predicted regression frame and predicted regression angle of the sample image based on the position information and angle information of each character area, and all the second prediction probabilities; determining the loss function based on the position information, angle information, second prediction probability, predicted regression frame, and predicted regression angle of all the sample images; and iteratively training the image character positioning model based on the loss function to obtain a trained image character positioning model.

[0011] In an embodiment of the present application, the recognition method includes: determining the aspect ratio of the character area where the stamped characters are located in each sample image; and determining the aspect ratio of the second candidate frame based on the aspect ratio of all character areas.

[0012] In the embodiment of the present application, the loss function is expressed as shown in formula (1):

[0013] Loss = αL SmoothL1 (θ,θ′)+tβL SmoothL1 (P r ,t)+λt∑ i∈(x，y，w，h) L CIoU (V i , V i ′)+ηLoss RPN (1)

[0014] Among them, Loss refers to the function value of the loss function, θ refers to the angle information of the detection box, θ′ refers to the angle information of the predicted regression box, V iRefers to the position information of the detection frame (x, y, w, h), V i ′ refers to the position information of the predicted regression box (x′, y′, w′, h′), t∈{1, 0}, t=1 represents the foreground, t=0 represents the background, P r It refers to the predicted probability that the predicted regression box is the foreground or background, and α, λ, and η are weight parameters.

[0015] In an embodiment of the present application, obtaining multiple sample images with stamped characters includes: obtaining multiple original images with stamped characters, marking a bounding box surrounding the stamped characters in each original image; for each original image, perturbing the bounding box to obtain multiple processed images corresponding to the original image; and obtaining all original images and all processed images as sample images.

[0016] In an embodiment of the present application, the text recognition module includes a feature extraction layer and a Transformer model, the Transformer model includes an encoder and a decoder, and the target area image is input into the text recognition module to output the character information of the stamped characters in the target area image through the text recognition module, including: extracting multiple image features related to the target area image through the feature extraction layer; performing feature fusion on the multiple image features through the encoder of the Transformer model to obtain a feature result vector corresponding to the multiple image features, and the feature result vector includes text information in the target area image; decoding the feature result vector through the decoder of the Transformer model to calculate the probability vector of the stamped characters in the target area image, and outputting the character information based on the probability vector.

[0017] In an embodiment of the present application, the recognition method also includes: after outputting the character information, determining whether there is target steel stamp character information identical to the steel stamp character in the task work order based on the character information, wherein the task work order refers to the steel stamp task list planned to be engraved by the workshop; if the target steel stamp character does not exist, issuing a corresponding prompt message.

[0018] A second aspect of the present application provides a processor configured to execute the above-mentioned method for recognizing steel stamp characters.

[0019] A third aspect of the present application provides a device for recognizing stamped characters, comprising the above-mentioned processor configured to execute the above-mentioned method for recognizing stamped characters.

[0020] A fourth aspect of the present application provides a mobile device, comprising:

[0021] An image acquisition device for acquiring images of steel-stamped characters; and a device for recognizing steel-stamped characters.

[0022] In an embodiment of the present application, the mobile device further includes: a light source for providing a uniformly illuminated acquisition environment for the image acquisition device.

[0023] A fifth aspect of the present application provides a machine-readable storage medium having instructions stored thereon, which, when executed by a processor, configures the processor to execute the above-mentioned method for recognizing steel stamp characters.

[0024] Through the above technical solution, by obtaining a steel-stamped character image to be identified; inputting the steel-stamped character image into an image character positioning model, determining the rotation angle of the detection frame for the steel-stamped characters in the steel-stamped character image through the angle prediction branch of the image character positioning model, and cropping the area image containing the steel-stamped characters in the steel-stamped character image based on the rotation angle; obtaining the area image output by the image character positioning model as the target area image; inputting the target area image into a text recognition module, so that the text recognition module outputs the character information of the steel-stamped characters in the target area image. By using intelligent visual means, the steel-stamped characters engraved on the surface of metal objects can be automatically and quickly identified with high precision, which can significantly reduce labor costs and time costs, improve product management efficiency, and achieve optimized control of the production process.

[0025] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following detailed description, they are used to explain the embodiments of the present application but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:

[0027] Figure 1 The following schematically shows a flow chart of a method for recognizing steel stamp characters according to an embodiment of the present application;

[0028] Figure 2 A schematic diagram schematically shows a detection frame that is not rotated according to an embodiment of the present application;

[0029] Figure 3 A schematic diagram schematically illustrates the rotation of a detection frame according to an embodiment of the present application;

[0030] Figure 4 Schematically shows a network framework diagram of an image character positioning model according to an embodiment of the present application;

[0031] Figure 5 A schematic diagram of a network framework of a text recognition module according to an embodiment of the present application is shown;

[0032] Figure 6Schematically shows a schematic diagram of the steps of checking character information according to an embodiment of the present application;

[0033] Figure 7 The following schematically shows a structural block diagram of a device for recognizing stamped characters according to an embodiment of the present application;

[0034] Figure 8 The structure of a mobile device according to an embodiment of the present application is schematically shown;

[0035] Figure 9 A schematic diagram of a shooting limit frame according to an embodiment of the present application is schematically shown;

[0036] Figure 10 A schematic diagram of a fill light source according to an embodiment of the present application is schematically shown;

[0037] Figure 11 The internal structure diagram of a computer device according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0038] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0039] Figure 1 The following schematically shows a flow chart of a method for recognizing stamped characters according to an embodiment of the present application. Figure 1 As shown, in one embodiment of the present application, a method for recognizing stamped characters is provided. This embodiment mainly uses the method applied to a processor as an example, and includes the following steps:

[0040] S102, obtaining a steel stamp character image to be recognized;

[0041] S104: Input the stamped character image into an image character localization model, determine the rotation angle of the detection frame for the stamped character in the stamped character image through the angle prediction branch of the image character localization model, and crop the area image containing the stamped character in the stamped character image based on the rotation angle;

[0042] S106, obtaining the region image output by the image character localization model as the target region image;

[0043] S108: Input the target area image to a text recognition module, so that the text recognition module outputs character information of the stamped characters in the target area image.

[0044] During the manufacturing process of large-scale construction machinery, it is necessary to imprint characters on the surface of the product and its metal components. This is accomplished through a one-to-one code approach to facilitate product quantity counting, vehicle assembly, quality monitoring, and traceability management. A stamped character image refers to an image containing the stamped characters, representing the product's production number. However, computers cannot directly capture the stamped characters in an image; a series of image processing is required to identify the stamped characters. A processor can capture the stamped character image to be recognized. Since stamped characters are primarily imprinted on metal components, the captured stamped character image may produce different visual effects depending on the angle, resulting in some deviation in the displayed stamped character image. Therefore, the processor can first input the stamped character image into an image character localization model to locate the stamped characters within the image. An image character localization model is a machine learning model that can identify the location of characters within an image. Traditional object detection algorithms can identify the area within an image where stamped characters reside and generate a detection box to calibrate the stamped characters. Since the characters on the steel stamp are regular long strips, and the operator has an angle deviation when taking the image, the steel stamp in the acquired image appears to be rotated at a certain angle. Figure 2 As shown, the stamped characters in the rectangular detection frame are tilted at a certain angle, and the borders of the rectangular frame are not parallel. In order to make the detection frame surrounding the stamped characters fit more closely to the characters, in an embodiment of the present application, the image character positioning model is improved based on the traditional target detection algorithm. Specifically, by adding an angle prediction branch to the traditional target detection algorithm, it is used to predict the rotation angle of the detection frame of the stamped characters in the stamped character image. Then, the processor can input the stamped character image into the image character positioning model, and determine the rotation angle of the detection frame for the stamped characters in the stamped character image through the angle prediction branch of the image character positioning model. As Figure 3 As shown, the detection frame generated by the image character positioning model can be rotated according to the predicted rotation angle. Then, the processor can crop the area image containing the printed characters in the printed character image based on the rotation angle. That is, the printed character image is cropped along the edge of the detection frame. Furthermore, the processor can obtain the area image output by the image character positioning model as the target area image. The target area image is input into the text recognition module to output the character information of the printed characters in the target area image through the text recognition module. The text recognition module (optical character recognition) refers to a module that analyzes and processes image files to obtain character information in the image.

[0045] The image character positioning model with a newly added angle prediction branch first locates the stamped characters in the image, so that the detection frame surrounding the stamped characters fits more closely to the characters. The text recognition module then identifies the character information of the stamped characters in the detection frame. The aforementioned machine vision technology enables automatic and high-precision recognition of production numbers engraved on the surface of metal objects. This is a key step in realizing the intelligentization of key processes and optimized control of production processes in the production and assembly of large-scale engineering machinery, and can replace the current manual reading and copying of stamped characters. This can significantly reduce labor costs and time costs, improve recognition efficiency, and is of great significance to product quantity statistics, vehicle assembly, quality supervision, and tracking management in the production of large-scale equipment.

[0046] In one embodiment, a stamped character image is input into an image character positioning model, and the rotation angle of a detection frame for the stamped characters in the stamped character image is determined through the angle prediction branch of the image character positioning model, including: inputting the stamped character image into the image character positioning model, and extracting multiple first candidate frames corresponding to the stamped character image through the region candidate network of the image character positioning model; predicting a first prediction probability of each first candidate frame being a foreground through the classification branch of the image character positioning model, wherein the foreground refers to a frame containing stamped characters; determining a detection frame for the stamped characters in the stamped character image based on the first candidate frame whose first prediction probability is greater than a preset threshold; and determining the rotation angle of the detection frame through the angle prediction branch of the image character positioning model.

[0047] The processor can input the seal character image to the image character positioning model, and extract a plurality of first candidate boxes corresponding to the seal character image through a region proposal network of the image character positioning model. Then, the processor can predict, through a classification branch of the image character positioning model, a first prediction probability that each first candidate box is a foreground, where the foreground refers to containing the seal character. Then, the processor can determine a detection box of the seal character in the seal character image according to a first candidate box whose first prediction probability is greater than a preset threshold value in all first candidate boxes, and determine a rotation angle of the detection box through an angle prediction branch of the image character positioning model. The preset threshold value refers to a prediction probability threshold value set by the technician according to an experience error range. Specifically, the image character positioning model can be an improved Faster-RCNN model. The region proposal network can extract the first candidate box of the seal character image. The candidate box is the position of the target in the picture initially predicted by the Faster-RCNN model and the confidence that the position contains the target. This prediction of the target position is rough and needs to be further processed by the subsequent network to obtain an accurate position. The first candidate box refers to the candidate box extracted from the seal character image to be recognized. In the embodiments of the present application, an angle prediction branch is added in the Head structure of the Faster-RCNN model, and the angle is enabled based on the traditional target detection framework, which is used to predict the rotation angle of the detection box.

[0048] In one embodiment, the identification method further includes a training step of the image character positioning model, and the training step includes: obtaining a plurality of sample images with seal characters, and marking a character region where the seal character is located in each sample image; for each sample image, extracting a plurality of second candidate boxes; for each sample image, predicting, through a classification branch of the image character positioning model, a second prediction probability that each second candidate box is a foreground or a background, where the foreground refers to containing the seal character, and the background refers to not containing the seal character; for each sample image, determining a prediction regression box and a prediction regression angle of the sample image according to the position information and the angle information of each character region and all second prediction probabilities; determining a loss function according to the position information, the angle information, the second prediction probability, the prediction regression box and the prediction regression angle of all sample images; and iteratively training the image character positioning model based on the loss function to obtain a trained image character positioning model.

[0049] In one embodiment, the identification method includes: determining an aspect ratio of a character region where the seal character is located in each sample image; and determining an aspect ratio of the second candidate box according to the aspect ratios of all character regions.

[0050] To train the image character location model, the processor can obtain multiple sample images with stamped characters and mark the character area where the stamped characters are located in each sample image. The character area includes the position and angle information of the bounding box of the stamped characters in the sample image.

[0051] Then, during the training process, the second candidate frame of each sample image can be extracted through the region candidate network. It can be understood that the second candidate frame is relative to the first candidate frame. Since the steel stamp characters are basically distributed horizontally, the aspect ratio of the candidate frame can be set according to the actual steel stamp size on site. Then, the processor can determine the aspect ratio of the character area where the steel stamp characters are located in each sample image; determine the aspect ratio of the second candidate frame based on the aspect ratio of all character areas. In addition, the aspect ratio of the second candidate frame can also be applied to the extraction process of the first candidate frame. Furthermore, the processor can predict the second prediction probability that each second candidate frame is the foreground or background through the classification branch of the image character positioning model. Then, for each sample image, the predicted regression frame and predicted regression angle of the sample image are determined based on the position information (x, y, w, h) and angle information θ of each character area, as well as all the second prediction probabilities. Specifically, the character area of ​​the sample image can be regressed through the bbox branch. Furthermore, the processor can determine the loss function based on the position information, angle information, second prediction probability, predicted regression box, and predicted regression angle of all sample images, and iteratively train the image character positioning model based on the loss function to obtain a trained image character positioning model.

[0052] Specifically, the aspect ratio of the region proposal (candidate box) generated by the region candidate network is set to at least one of 4:1, 6:1, and 8:1. The image character localization model adds an angle prediction branch to the Head structure of the Faster-RCNN model, adding angle enablement on the basis of the traditional object detection framework to predict the rotation angle of the detection box. Figure 4 The cls branch (classification branch) predicts the probability Pr of each region proposal (candidate box) being the foreground or background. The bbox branch regresses the bounding box, and the angle branch (angle prediction branch) predicts and regresses the rotation angle θ. Both the foreground and background prediction probabilities and the θ parameter regression are optimized using the single-parameter Smooth L1 Loss, using CIoU Loss as the regression loss. During inference, this is combined with the θ and Pr parameters to obtain the bounding box prediction results and prediction scores.

[0053] In the embodiment of the present application, the loss function is expressed as shown in formula (1):

[0054] Loss = αLsmoothL1 (θ,θ′)+βL smoothL1 (Pr, t)+λt∑ i∈(x，y，w , h) L CIou (V i , V i ′)+ηLoss RPN (1)

[0055] Among them, Loss refers to the function value of the loss function, θ refers to the angle information of the detection box, θ′ refers to the angle information of the predicted regression box, V i Refers to the position information of the detection frame (x, y, w, h), V i ′ refers to the position information of the predicted regression box (x′, y′, w′, h′), t∈{1, 0}, t=1 represents the foreground, t=0 represents the background, P r Loss refers to the predicted probability that the regression box is the foreground or background, and α, λ, and η are weight parameters. RPN This refers to the loss for extracting candidate boxes. The RPN loss is calculated using the same method as the original Faster-RCNN algorithm. Training is completed through multiple rounds of iterative training until the calculated value of the loss function reaches the set value, resulting in an image character localization model that meets the required accuracy.

[0056] In one embodiment, obtaining multiple sample images with stamped characters includes: obtaining multiple original images with stamped characters, marking a bounding box surrounding the stamped characters in each original image; for each original image, perturbing the bounding box to obtain multiple processed images corresponding to the original image; and obtaining all original images and all processed images as sample images.

[0057] Since industrial images are difficult to obtain, machine vision technology based on deep learning requires the support of large-scale data. In the process of the processor acquiring multiple sample images with steel-printed characters, multiple original images with steel-printed characters can be acquired first. The original image refers to the steel-printed character image actually collected. The technician can mark the bounding box surrounding the steel-printed characters in each original image. The processor can perform perturbation processing on the bounding box of each original image to obtain multiple processed images corresponding to each original image, so that all original images and all processed images can be obtained as sample images. Specifically, the processor can randomly move the bounding box marked on the original image up and down, and fine-tune the length, width and angle. The original image can also be processed by contrast stretching to improve the contrast of the image and to alleviate the influence of light on image recognition to a certain extent. Specifically, the processor can map the grayscale value of a pixel in the original image to a larger grayscale space according to the following formula (2):

[0058]

[0059] Here, (x, y) refers to the coordinates of each pixel in the original image, I(x, y) refers to the grayscale value of each pixel after transformation, Imin refers to the minimum grayscale value of all pixels in the original image, Imax refers to the maximum grayscale value of all pixels in the original image, and MIN and MAX refer to the minimum and maximum grayscale values ​​in the grayscale space to be stretched. By adding random perturbations to the original data, we can expand the training data and improve the model's ability to resist interference.

[0060] In one embodiment, the text recognition module includes a feature extraction layer and a Transformer model, the Transformer model includes an encoder and a decoder, and the target area image is input into the text recognition module to output the character information of the stamped characters in the target area image through the text recognition module, including: extracting multiple image features related to the target area image through the feature extraction layer; performing feature fusion on the multiple image features through the encoder of the Transformer model to obtain a feature result vector corresponding to the multiple image features, and the feature result vector includes text information in the target area image; decoding the feature result vector through the decoder of the Transformer model to calculate the probability vector of the stamped characters in the target area image, and outputting the character information based on the probability vector.

[0061] For the text recognition module, there are two mainstream algorithms in the era of deep learning, one is the CRNN algorithm and the other is the attention algorithm. CRNN is a convolutional neural network (CNN) + recurrent neural network (RNN) + softmax function, and attention is achieved by replacing the RNN module in CRNN with attention. In the embodiment of the present application, the Transformer framework is used to replace the RNN module in CRNN. ​​The Transformer model uses the attention mechanism to acquire information and has powerful encoding and decoding capabilities. Therefore, applying it to the field of OCR recognition is sufficient to show excellent performance. The network framework of the text recognition module is as follows: Figure 5 As shown, the processor can extract multiple image features related to the target area image through the feature extraction layer, and after performing feature fusion on the multiple image features through the encoder of the Transformer model, obtain a feature result vector corresponding to the multiple image features. The feature result vector includes the text information in the target area image. Furthermore, the processor can decode the feature result vector through the decoder of the Transformer model to calculate the probability vector of the stamped characters in the target area image, and output the character information based on the probability vector.

[0062] Specifically, the EfficientNetV2 model is added as the feature extraction layer of the network to enhance the feature extraction capability of the model, extract multiple image features and generate feature maps. Then the Transformer encoder encodes the feature matrix output by the CNN MODEL and outputs a feature result vector with context information. The Transformer decoder then decodes and calculates the feature result vector, outputs the probability vector of the character, and finds the character corresponding to the index of the maximum probability value in the probability vector to determine the character information in the regional image. Among them, for the Transformer model, each Transformer layer consists of three components: a multi-head attention MultiHead Attention, a fully connected feedforward network Feed Forward and Add&Norm. Considering that different attention heads have different contributions to the network, the direct fusion of the multi-head attention layer of the Transformer model can be improved to a multi-head weighted fusion mechanism. By learning the weight value of each head in the network, the overall performance of the multi-head attention layer is enhanced. The outputs of different attention heads after the fully connected layer FFN are weightedly fused to obtain a multi-head weighted attention head. Specifically, it can be shown as follows (3):

[0063]

[0064] Where h is the number of attention heads, head is the output of different attention heads, α i The weighted weight for each attention head, head i Refers to the output of each attention head.

[0065] However, since the Transformer model is based on the attention mechanism, it discards the temporal information of the traditional recurrent convolutional layer, which will lead to the loss of position information. Therefore, a position encoding module is needed to make up for this defect. The usual method is to use sine and cosine functions of different frequencies at the Positional Encoding of the network to generate a fixed position mark of the target (absolute position encoding). On this basis, considering that there is a certain connection between characters, the effective method of combining relative position marks in absolute position encoding is used to achieve the relative position representation between targets. Specifically, formulas (4)(5)(6) are as follows:

[0066]

[0067]

[0068]

[0069] Among them, Q, K, and V represent the query vector (Q), key vector (K), and value vector (V) of the predicted word in the stamped characters, respectively. Given a query vector related to the task, the attention value is calculated by calculating the attention distribution with the Key and appending it to the Value. i refers to the i-th character, j refers to the j-th character, and e ij refers to the similarity between the i-th character and the j-th character, λ ij Refers to the weight coefficient corresponding to the similarity between the i-th character and the j-th character, y i Refers to the output value of the attention module. and is the newly introduced relative position representation vector, x is the attention head input sequence, y is the output sequence, W Q 、W K 、W V is the parameter matrix. This relative position identification method adds relative position representation to the absolute position encoding of the original Transformer model, thereby compensating for the Transformer model's lack of ability to capture temporal information. This achieves a model accuracy of 99%, making it suitable for recognizing characters engraved on metal surfaces in industrial scenarios.

[0070] like Figure 6 As shown, in one embodiment of the present application, the identification method further includes the following steps:

[0071] S602: After the character information is output, it is determined based on the character information whether the target stamp character information is identical to the stamp character in the task worksheet, where the task worksheet refers to the list of stamp tasks that the workshop plans to engrave. If not, execute S604; if so, execute S606.

[0072] S604: Send corresponding prompt information.

[0073] S606: Compare the character information with the character information in the detection record.

[0074] S608: Determine whether the character information is the same as the character information in the detection record. If so, execute S604; if not, execute S610.

[0075] S610, determining whether the character information in the steel stamp character image is engraved correctly.

[0076] The captured image of the stamped characters to be recognized can be transmitted to the MES via the web. The processor can retrieve the stamped character image from the MES and output the character information using a machine vision recognition algorithm. After outputting the character information, the processor can determine whether the target stamped character information identical to the stamped characters exists in the work order. The target stamped character information is the correct character information recorded on the work order for each product. If it is different, it may indicate a stamped character error or unclear photograph. The processor can issue a prompt message, prompting the technician to retake the photo or record the marking error. The technician can manually identify the character to modify the inspection results and store the modification record as a searchable log file, improving the stability and fault tolerance of the entire recognition system. Furthermore, if the target stamped character information identical to the character information exists in the work order, the processor can compare the character information with the inspection record to determine whether the character information is identical. If it is identical, it indicates a possible stamped character duplication, and the processor can issue a prompt message to inform the technician. If it is different, the processor can determine that the character information in the stamped character image is correctly engraved. In this way, the recognition efficiency of steel stamp characters is greatly improved, and the situation where human recognition errors due to subjective factors are avoided, the orderly management of product production processes and precise control of production processes are achieved, further improving product management efficiency.

[0077] Through the above technical solution, a steel stamp character image to be recognized is obtained; the steel stamp character image is input into the image character positioning model, and the angle prediction branch of the image character positioning model is used to determine the rotation angle of the detection frame for the steel stamp character in the steel stamp character image. Based on the rotation angle, the area image containing the steel stamp character in the steel stamp character image is cropped out; the area image output by the image character positioning model is obtained as the target area image; the target area image is input into the text recognition module, and the text recognition module outputs the character information of the steel stamp character in the target area image. By adding an angle prediction branch to improve the Faster-RCNN model, a rich sample image is obtained through random perturbation processing. Combined with the intelligent visual method of steel stamp OCR recognition improved by the Transformer model, the steel stamp characters engraved on the surface of metal objects are automatically identified, and whether they are engraved correctly or repeatedly is determined, thereby improving product management efficiency and achieving optimized control of the production process. This is of great significance to product quantity statistics, vehicle assembly, quality supervision and tracking management in the production process of large-scale equipment.

[0078] Figure 1 FIG. 1 is a flow chart of a method for recognizing steel stamp characters in one embodiment. It should be understood that although Figure 1The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0079] In one embodiment, Figure 7 As shown, a device 700 for recognizing stamped characters is provided, comprising an image acquisition module, a character positioning module, and a character recognition module, wherein:

[0080] The image acquisition module 702 is used to acquire the image of the stamped characters to be recognized.

[0081] The character positioning module 704 is used to input the stamped character image into the image character positioning model, determine the rotation angle of the detection box for the stamped characters in the stamped character image through the angle prediction branch of the image character positioning model, and crop the area image containing the stamped characters in the stamped character image based on the rotation angle.

[0082] The character recognition module 706 is used to obtain the area image output by the image character positioning model as the target area image; input the target area image to the text recognition module to output the character information of the stamped characters in the target area image through the text recognition module.

[0083] In one embodiment, the character positioning module 704 is also used to: input the stamped character image into the image character positioning model, extract multiple first candidate boxes corresponding to the stamped character image through the region candidate network of the image character positioning model; predict the first prediction probability of each first candidate box being the foreground through the classification branch of the image character positioning model, where the foreground refers to the box containing the stamped character; determine the detection box of the stamped character in the stamped character image based on the first candidate box whose first prediction probability is greater than a preset threshold; and determine the rotation angle of the detection box through the angle prediction branch of the image character positioning model.

[0084] In one embodiment, the device 700 for recognizing stamped characters further includes a training module of an image character positioning model (not shown in the figure), which is used to obtain multiple sample images with stamped characters and mark the character area where the stamped characters are located in each sample image; for each sample image, extract multiple second candidate frames; for each sample image, predict the second prediction probability of each second candidate frame being the foreground or background through the classification branch of the image character positioning model, wherein the foreground refers to the area containing the stamped characters and the background refers to the area not containing the stamped characters; for each sample image, determine the predicted regression frame and predicted regression angle of the sample image based on the position information and angle information of each character area and all the second prediction probabilities; determine the loss function based on the position information, angle information, second prediction probability, predicted regression frame and predicted regression angle of all the sample images; and iteratively train the image character positioning model based on the loss function to obtain a trained image character positioning model.

[0085] In one embodiment, the device 700 for recognizing stamped characters further includes a candidate frame setting module (not shown in the figure), which is used to: determine the aspect ratio of the character area where the stamped characters are located in each sample image; and determine the aspect ratio of the second candidate frame based on the aspect ratio of all character areas.

[0086] In one embodiment, the loss function is expressed as shown in formula (1):

[0087] Loss = αL smoothL1 (θ,θ′)+βL smoothL1 (P r ,t)+λt∑ i∈(x，y，w，h) L CIoU (V i , V i ′)+ηLoss RPN (1)

[0088] Among them, Loss refers to the function value of the loss function, θ refers to the angle information of the detection box, 0′ refers to the angle information of the predicted regression box, V i Refers to the position information of the detection frame (x, y, w, h), V i ′ refers to the position information of the predicted regression box (x′, y′, w′, h′), t∈{1, 0}, t=1 represents the foreground, t=0 represents the background, P r It refers to the predicted probability that the predicted regression box is the foreground or background, and α, λ, and η are weight parameters.

[0089] In one embodiment, the image acquisition module 702 is further used to acquire multiple original images with stamped characters, with a bounding box surrounding the stamped characters marked in each original image; for each original image, the bounding box is perturbed to obtain multiple processed images corresponding to the original image; and all original images and all processed images are acquired as sample images.

[0090] In one embodiment, the text recognition module includes a feature extraction layer and a Transformer model, the Transformer model includes an encoder and a decoder, and the character recognition module 706 is also used to: extract multiple image features related to the target area image through the feature extraction layer; obtain a feature result vector corresponding to the multiple image features after feature fusion of the multiple image features through the encoder of the Transformer model, and the feature result vector includes text information in the target area image; decode the feature result vector through the decoder of the Transformer model to calculate the probability vector of the stamped characters in the target area image, and output character information based on the probability vector.

[0091] In an embodiment of the present application, the steel stamp character recognition device 700 also includes a verification module (not shown in the figure), which is used to determine whether there is target steel stamp character information identical to the steel stamp character in the task work order based on the character information after outputting the character information, wherein the task work order refers to the steel stamp task list planned to be engraved by the workshop; if the target steel stamp character does not exist, a corresponding prompt message is issued.

[0092] The device for recognizing stamped characters includes a processor and a memory. The above-mentioned image acquisition module, character positioning module, and character recognition module are all stored in the memory as program units, and the processor executes the above-mentioned program modules stored in the memory to implement corresponding functions.

[0093] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the method for recognizing the stamped characters can be implemented by adjusting the kernel parameters.

[0094] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0095] In one embodiment, Figure 8 As shown, a mobile device 800 is provided, including:

[0096] Image acquisition device 810, used to acquire images of stamped characters;

[0097] And a device 700 for recognizing stamped characters.

[0098] The mobile device can be a mobile phone, notebook, tablet computer or watch and other devices. The image acquisition device 810 can be a camera, a camcorder, a scanner, or other devices with a camera function (mobile phone, tablet computer, etc.). Since the main content of steel stamp character recognition is the vehicle code and structural part code of the engineering vehicle, and the product is too large, and different types of metal parts are placed in different positions. The recognition method of using a fixed workstation in an assembly line is obviously not in line with the actual situation on site, so the steel stamp character recognition device 700 can be installed in the user's mobile device 800, and the user can rely on the mobile device to collect steel stamp character images anytime and anywhere and recognize their character information. This method is low-cost and simple to operate, and can recognize steel stamp characters in multiple scenarios.

[0099] Mobile devices are suitable for the recognition of stamp characters in distributed and mobile scenarios. Users use their own mobile devices to open the recognition software to collect stamp images. Due to the different shooting habits of different operators, it is easy to cause large differences between the data, and the recognition of stamp characters is easily affected by the image background. In order to restrict the shooting behavior and improve the image quality of the collected data, a rectangular window of appropriate size is designed on the image shooting interface, as follows Figure 9 As shown in the figure, the stamped characters must be within the pre-marked shooting limit box when shooting. On the one hand, this prevents the redundant background in the original image from affecting the character positioning, improving the accuracy of subsequent character positioning. On the other hand, it standardizes the user's shooting behavior, making the collected image data more consistent with the recognition standard.

[0100] In one embodiment, the mobile device 800 further includes a light source 820 for providing a uniformly illuminated acquisition environment for the image acquisition device 810 .

[0101] The vehicle stamp code is located on the chassis frame of the product. When the vehicle is being recognized, it will block part of the light source, resulting in uneven lighting. At the same time, the structural parts are stacked together, and the structural parts at the bottom are darker. In order to alleviate the impact of light on character recognition, the mobile device 800 also includes a light source 820. Figure 10 The Light Source 820 can be used as a small ring-shaped fill light source, nested around the rear camera of a smart mobile device. In the face of poor lighting conditions, it provides supplementary and uniform lighting, improving character recognition accuracy. It is suitable for applications with high space requirements, easy to carry and simple to operate.

[0102] An embodiment of the present application provides a storage medium on which a program is stored, and when the program is executed by a processor, the above-mentioned method for recognizing steel stamp characters is implemented.

[0103] An embodiment of the present application provides a processor, which is used to run a program, wherein the program executes the above-mentioned method for recognizing steel stamp characters when running.

[0104] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 11 As shown. The computer device includes a processor A01, a network interface A02, a memory (not shown in the figure) and a database (not shown in the figure) connected via a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02 and a database (not shown in the figure). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A04. The database of the computer device is used to store data for the recognition of steel-printed characters. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program B02 is executed by the processor A01, a method for recognizing steel-printed characters is implemented.

[0105] Those skilled in the art will understand that Figure 11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0106] An embodiment of the present application provides a device, which includes a processor, a memory, and a program stored in the memory and runnable on the processor. When the processor executes the program, the steps of the method for recognizing steel stamp characters are implemented.

[0107] The present application also provides a computer program product which, when executed on a data processing device, is suitable for executing the program steps of the method for initializing the recognition of stamped characters.

[0108] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0109] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0110] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0111] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0112] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0113] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0114] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0115] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0116] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A method for recognizing stamped characters, characterized in that: The identification method comprises: Obtaining a steel stamp character image to be recognized; Inputting the stamped character image into an image character positioning model, determining the rotation angle of a detection frame for the stamped character in the stamped character image through an angle prediction branch of the image character positioning model, and cropping an area image of the stamped character image containing the stamped character based on the rotation angle; Acquire the region image output by the image character positioning model as a target region image; The target area image is input to a text recognition module, so that the character information of the stamped characters in the target area image is output through the text recognition module. The text recognition module includes a feature extraction layer and a Model, The model includes an encoder and a decoder; The step of inputting the stamped character image into an image character positioning model and determining the rotation angle of the detection frame for the stamped character in the stamped character image through an angle prediction branch of the image character positioning model includes: Inputting the stamped character image into an image character positioning model, and extracting a plurality of first candidate frames corresponding to the stamped character image through a region candidate network of the image character positioning model; Predicting a first prediction probability of each first candidate box being a foreground by a classification branch of the image character localization model, wherein the foreground refers to a box containing a stamped character; Determine a detection frame of the stamped character in the stamped character image according to a first candidate frame whose first prediction probability is greater than a preset threshold; Determining the rotation angle of the detection frame through the angle prediction branch of the image character positioning model; The step of inputting the target area image into a text recognition module so as to output character information of the stamped characters in the target area image through the text recognition module includes: Extracting multiple image features related to the target area image through the feature extraction layer; Through the The encoder of the model performs feature fusion on the multiple image features to obtain a feature result vector corresponding to the multiple image features, wherein the feature result vector includes text information in the target area image; Through the The decoder of the model decodes the feature result vector to calculate the probability vector of the stamped characters in the target area image to output the character information based on the probability vector.

2. The method for recognizing steel stamp characters according to claim 1, characterized in that: The recognition method further includes a step of training the image character positioning model, the training step including: Acquire multiple sample images with stamped characters, and mark the character area where the stamped characters are located in each sample image; For each sample image, extract multiple second candidate frames; For each sample image, predicting a second prediction probability of each second candidate box being a foreground or a background by the classification branch of the image character localization model, wherein the foreground refers to an area containing stamped characters, and the background refers to an area not containing stamped characters; For each sample image, determining a predicted regression frame and a predicted regression angle of the sample image according to the position information and angle information of each character region and all second prediction probabilities; Determine the loss function based on the position information, angle information, second prediction probability, predicted regression box, and predicted regression angle of all sample images; The image character positioning model is iteratively trained based on the loss function to obtain a trained image character positioning model.

3. The method for recognizing steel stamp characters according to claim 2, characterized in that: The identification method comprises: Determine the aspect ratio of the character area where the stamped characters are located in each sample image; The aspect ratio of the second candidate frame is determined according to the aspect ratio of the entire character area.

4. The method for recognizing steel stamp characters according to claim 2, characterized in that: The loss function is expressed as formula (1): （1） Among them, Loss refers to the function value of the loss function, refers to the angle information of the detection frame, Refers to the angle information of the predicted regression frame, Refers to the position information of the detection frame ( ), Refers to the position information of the predicted regression box ( ), , t=1 represents the foreground, t=0 represents the background, It refers to the predicted probability that the predicted regression box is the foreground or background, 、 、 is the weight parameter.

5. The method for recognizing steel stamp characters according to claim 2, characterized in that: The step of obtaining a plurality of sample images with steel-stamped characters includes: Acquire multiple original images with stamped characters, and mark a bounding box surrounding the stamped characters in each original image; For each original image, performing a perturbation process on the bounding box to obtain a plurality of processed images corresponding to the original image; All original images and all processed images are acquired as sample images.

6. The method for recognizing steel stamp characters according to claim 1, characterized in that: Identification methods also include: After outputting the character information, determining whether target steel stamp character information identical to the steel stamp character exists in a task worksheet according to the character information, wherein the task worksheet refers to a list of steel stamp tasks that the workshop plans to engrave; If the target stamped characters do not exist, a corresponding prompt message is issued.

7. A processor, characterized in that: The method is configured to execute the method for recognizing stamped characters according to any one of claims 1 to 6.

8. A device for recognizing stamped characters, characterized in that: comprising the processor of claim 7.

9. A mobile device, characterized in that: include: An image acquisition device, used for acquiring images of characters on a steel stamp; as well as The device for recognizing steel-stamped characters as claimed in claim 8.

10. The mobile device according to claim 9, wherein: Also includes: A light source is used to provide a uniformly illuminated acquisition environment for the image acquisition device.

11. A machine-readable storage medium having instructions stored thereon, characterized in that: When the instruction is executed by a processor, the processor is configured to execute the method for recognizing steel-printed characters according to any one of claims 1 to 6.

Images