Multi-line rotated casting number character recognition method based on direction modeling and transformer decoding

Abstract

The application discloses a multi-line rotating casting blank number character recognition method and device based on direction modeling and Transformer decoding, and relates to the technical field of artificial intelligence and computer vision. The method comprises the following steps: acquiring a single casting blank section image to be recognized and performing pretreatment to obtain a processed image; inputting the processed image into a multi-scale visual Transformer encoder to obtain a multi-scale feature map set; performing detection through an Anchor-free rotating character detection head to output a character candidate set; calculating absolute orientation information of the characters by using a direction distribution prediction head to construct a final character set; predicting a directed sequential relationship between character nodes by using a graph attention network to output a multi-line reading order; obtaining a visual token sequence arranged in a real reading order according to the multi-line reading order and inputting the visual token sequence into a pre-trained Transformer language decoding model to generate a complete casting blank number character sequence in a self-recurrence mode. The application can reduce the error rate of character recognition.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision technology, and in particular to a method and apparatus for recognizing multi-line rotating billet number characters based on orientation modeling and Transformer decoding. Background Technology

[0002] As a vital pillar industry of the national economy, the steel industry is a significant indicator of a nation's industrialization level. In recent years, with the transformation and upgrading of global manufacturing and the rapid development of intelligent manufacturing technologies, the steel industry is gradually moving towards a new stage of digitalization, intelligentization, and greening. Through long-term technological accumulation and industrial layout optimization, the steel industry has formed a complete production system and a massive industrial scale. However, with intensified market competition and the introduction of "dual carbon" targets, traditional steel production models face multiple challenges, including cost reduction and efficiency improvement, energy consumption control, and quality traceability. Improving production efficiency and product quality through refined management and intelligent monitoring of the entire production process using intelligent sensing, data analysis, and automated control has become an important direction for steel production technology research.

[0003] Billets are a crucial intermediate product connecting steelmaking and rolling, and their quality directly impacts the production efficiency of subsequent processes and the performance of the steel. With the rapid development of modern industry and the increasing demand for high-performance steel from downstream industries, higher requirements are being placed on the quality and production efficiency of billets. Material tracking character recognition, such as billet numbers, is an indispensable part of the steel mill production process, playing a vital role in modern steel manufacturing and logistics management. With the continuous development of industrial automation and intelligent manufacturing, accurate and efficient management of the logistics process has become a key factor in improving production efficiency, reducing costs, and ensuring product quality. In this context, the application of character recognition technology is particularly important. It achieves rapid and accurate tracking and management of steel and other materials by automatically identifying the markings on the material surface. In the field of computer vision, billet number character detection and recognition methods use images captured during the production process as input and output the corresponding number on the billet. This process requires first locating the character position and then outputting the characters sequentially. In actual production scenarios, the placement of billets lacks fixed standards, and the randomness of the shooting angle can lead to chaotic character orientations and arbitrary character shapes in the images. Meanwhile, the billet number characters are arranged in multiple rows. When the characters are out of control (such as tilting at too large an angle or even being completely inverted), the traditional reading order from left to right and from top to bottom is no longer applicable, which brings certain challenges to the character detection and recognition of the billet number.

[0004] In summary, in actual production environments, billet number character recognition faces the following typical challenges: (1) Character orientation: Due to the random placement of the ladle or billet, the flipping during transportation, and the unfixed shooting angle of the camera, the numbered characters may be tilted or even inverted at any angle; (2) Fixed order in multiple lines: Numbering is often sprayed in multiple lines (e.g., two or three lines). The order within a line and the order between lines are fixed in a physical sense, but when the whole is rotated or inverted, the general reading order from left to right and from top to bottom cannot be directly used. (3) Complex background interference: The steel plant site has problems such as high temperature radiation, strong reflection, dust and smoke, surface oxide scale and spraying defects, which cause the character edges to be blurred, some strokes to be missing or the contrast to be extremely low. (4) Numbers are the main type but easily confused with letters: The ladle number and billet number are usually pure numbers or numbers are the main type, but under visual noise, the number-letter confusion error of "0 / O, 1 / I, 5 / S or 8 / B" is easy to occur.

[0005] Existing technologies mainly fall into two categories: The first category consists of schemes based on text line detection and sequence recognition, such as combinations of CTPN, EAST, and CRNN. ​​These methods first detect horizontal or slightly tilted text lines, then perform sequence recognition on the cropped text line images. Because the model relies on the integrity of the text line region and the assumption of horizontality, text line detection is prone to deviation or breakage when the character tilt angle is too large or the characters are inverted. Furthermore, these methods cannot reliably recover the physical order between multiple lines, and the concatenation of detection and recognition leads to accumulated errors, making it difficult to meet the accurate recognition requirements of multiple lines of numbers in arbitrary directions in steel plants. The second category consists of schemes based on character-level rotation target detection, such as R3Det or RoI-Transformer. These methods output rotation box parameters (center point, width, height, and angle) and character categories within the rotation detection framework, thereby achieving the localization and classification of characters at arbitrary angles. However, the angle parameter of the rotating rectangle is usually only represented within a 180° range (e.g., [-90°, 90°]). Since the long and short sides of the rectangle can be interchanged, it is impossible to distinguish between upright and inverted characters, thus making it impossible to determine the reading direction of multi-line numbering, ultimately leading to line or character order errors. Some solutions use corner regression branches to predict character corners to complete the 360° orientation information, but corner regression parameters have high dimensionality, are sensitive to noise, and are difficult to train, still exhibiting insufficient robustness in the strong interference scenario of a steel plant.

[0006] In summary, there is an urgent need for a new, simpler, and more robust method for recognizing multi-line numbers in any direction, which can recover the 360° orientation of characters without relying on corner regression, and further utilize the numbering format and sequence context to reduce common errors such as misidentifying numbers as letters. Summary of the Invention

[0007] To address the high error rate caused by arbitrary character orientation, complex background interference, and confusion between numbers and letters in existing technologies, this invention provides a method and apparatus for recognizing multi-line rotating billet number characters based on orientation modeling and Transformer decoding. The technical solution is as follows:

[0008] On the one hand, a method for recognizing multi-line rotating billet number characters based on orientation modeling and Transformer decoding is provided. This method is implemented by a multi-line rotating billet number character recognition device based on orientation modeling and Transformer decoding. The method includes: S1. Obtain the on-site image dataset of the billet cross-section; preprocess the on-site image dataset of the billet cross-section to obtain the preprocessed image dataset; construct a multi-line rotating billet number character recognition model; wherein, the recognition model includes: a multi-scale visual Transformer encoder, an anchor-free rotating character detection head, an orientation distribution prediction head, and a pre-trained Transformer language decoding model; S2. The preprocessed image dataset is used to train the multi-line rotating billet number character recognition model to obtain the trained recognition model. S3. Obtain the cross-sectional image of the single billet to be identified, preprocess the cross-sectional image of the single billet to be identified, and obtain the processed image; input the processed image into the multi-scale visual Transformer encoder in the trained recognition model for feature extraction, and obtain a set of multi-scale feature maps. S4. Input the multi-scale feature map set into the Anchor-free rotating character detection head, detect each character instance in the image, and output a character candidate set; wherein, the character candidate set includes: the character type, center point coordinates, character width, basic rotation angle and confidence score of each character; S5. Use the direction distribution prediction head to calculate the two-dimensional direction vector corresponding to the true top direction of each character instance in the character candidate set, and perform normalization processing to obtain the unit direction vector of each character. S6. Calculate the absolute orientation angle of each character based on the unit orientation vector of each character; verify the consistency between the absolute orientation angle of each character and the basic rotation angle of each character to obtain the absolute orientation information of the characters; construct the final character set based on the absolute orientation information of the characters. S7. Treat each character instance in the character set as a node in a graph; predict the directed order relationship between character nodes using a graph attention network based on the nodes to obtain a directed graph of reading order; perform topological sorting on the directed graph of reading order to restore the in-line character order and output the multi-line reading order. S8. Convert each character instance into a visual token according to the multi-line reading order to obtain a visual token sequence arranged in the actual reading order; input the visual token sequence into the pre-trained Transformer language decoding model to generate a complete billet number character sequence through autoregression.

[0009] On the other hand, a multi-line rotating billet number character recognition device based on orientation modeling and Transformer decoding is provided. This device is applied to the multi-line rotating billet number character recognition method based on orientation modeling and Transformer decoding. The device includes: The acquisition and construction unit is used to acquire a dataset of on-site images of the cross-section of the billet; preprocess the on-site image dataset of the cross-section of the billet to obtain a preprocessed image dataset; and construct a multi-line rotating billet number character recognition model; wherein, the recognition model includes: a multi-scale visual Transformer encoder, an anchor-free rotating character detection head, an orientation distribution prediction head, and a pre-trained Transformer language decoding model. The training unit is used to train the multi-line rotating billet number character recognition model using the preprocessed image dataset to obtain the trained recognition model. The feature extraction unit is used to acquire a cross-sectional image of a single billet to be identified, preprocess the cross-sectional image of the single billet to be identified to obtain a processed image, and input the processed image into the multi-scale visual Transformer encoder in the trained recognition model for feature extraction to obtain a set of multi-scale feature maps. The first output unit is used to input a multi-scale feature map set into the Anchor-free rotating character detection head, detect each character instance in the image, and output a character candidate set; wherein, the character candidate set includes: the character type, center point coordinates, character width, basic rotation angle, and confidence score of each character; The first calculation unit is used to calculate the two-dimensional direction vector corresponding to the true top direction of each character instance in the character candidate set using the direction distribution prediction head, and to perform normalization processing to obtain the unit direction vector of each character. The second calculation unit is used to calculate the absolute orientation angle of each character based on the unit orientation vector of each character; to verify the consistency between the absolute orientation angle of each character and the basic rotation angle of each character to obtain the absolute orientation information of the characters; and to construct the final character set based on the absolute orientation information of the characters. The second output unit is used to treat each character instance in the character set as a node in a graph; based on the nodes, a graph attention network is used to predict the directed order relationship between character nodes to obtain a reading order directed graph; the reading order directed graph is topologically sorted to restore the in-line character order and output the multi-line reading order; The generation unit is used to convert each character instance into a visual token according to the reading order of multiple lines, and obtain a visual token sequence arranged in the actual reading order; the visual token sequence is input into a pre-trained Transformer language decoding model to generate a complete billet number character sequence through autoregression.

[0010] On the other hand, a multi-line rotating billet number character recognition device based on orientation modeling and Transformer decoding is provided. The multi-line rotating billet number character recognition device based on orientation modeling and Transformer decoding includes: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the methods described above for multi-line rotating billet number character recognition based on orientation modeling and Transformer decoding is implemented.

[0011] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement any of the above-described methods for recognizing multi-line rotating billet number characters based on orientation modeling and Transformer decoding.

[0012] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: In the field identification of billet numbers, achieving stable recognition under conditions of strong reflection, high-temperature thermal disturbance, coating defects, and arbitrary rotation (including inversion) is a core challenge for industrial application. Existing methods mostly rely on text line detection or character rotation detection followed by geometric sorting. While these approaches perform reasonably well with a relatively uniform training set distribution, they are prone to misdetection of line frames, disordered order, and missed detections / false detections when faced with inversion, cross-line adhesion, partial character loss, and background texture interference. This invention proposes a model that recognizes the common geometrical pattern of steel plant numbering: "fixed order across multiple lines + consistent character scale + consistent orientation." Therefore, it simultaneously models character positioning, orientation recovery, and order inference within a unified model. This allows the model to leverage structural priors to resist complex working conditions, significantly improving cross-scene robustness and overall recognition accuracy without increasing the cost of manual rule-making.

[0013] Existing rotation frame detection methods only provide angles within a 180° range, leading to ambiguity regarding upright / inverted orientation. Some methods extend this to 360° through corner point regression, but the high dimensionality of corner points makes them sensitive to noise and prone to orientation jumps under conditions such as missing paint strokes, smudges, and reflections, resulting in global sorting errors. This invention employs a combination of "180° basic half-angle rotation detection + orientation distribution regression of the top direction unit vector d." By replacing high-dimensional corner point regression with a 2D continuous unit vector, the orientation prediction space becomes lower-dimensional, smoother, and more convergent. Consistency verification achieves 360° unique orientation recovery, eliminating orientation uncertainty in inverted scenarios from the source and improving the stability of detection and subsequent reading order recovery for arbitrary rotation numbers.

[0014] Existing geometric sorting methods based on coordinate thresholds are unstable under conditions of uneven line spacing, overall rotation / inversion, and local missing characters, easily leading to cross-line splicing or character order reversal. This invention constructs a directed graph of reading order based on character centers and absolute orientation. It explicitly learns the "adjacent-pointing" relationship between characters through a graph attention network, and restores intra-line links by combining loop breaking and topological sorting. Inter-line order is restored through main direction projection clustering. This maintains global reading link consistency even in complex arrangements and missing characters, reducing the engineering failure rate of multi-line numbering order disorder and improving interpretability.

[0015] The billet numbers are primarily composed of pure numbers or numbers predominantly, but traditional visual character-by-character classification lacks sequence priors, making it prone to confusion between numbers and letters such as "0 / O, 1 / I, 5 / S, 8 / B" under noise, and difficult to self-correct. This invention introduces a pre-trained Transformer language decoding model compatible with the DeepSeek architecture. The visual token sequence is input into the decoder as the encoder memory. During generation, contextual probabilities are used for sequence-level discrimination, and a number-priority logit penalty and beam search constraints on the numbering format are applied. This simultaneously suppresses paths that do not conform to the number distribution and format rules at both the probability and search levels, enabling automatic error correction and reordering even under visual uncertainty. This significantly reduces errors such as misidentifying numbers as letters, incorrect number of digits, or inconsistent row structure, improving the reliability of industrial numbering recognition.

[0016] To ensure both accuracy and deployability, this invention employs an end-to-end joint training and phased optimization strategy: first, the detection, orientation, and order modules on the visual side are trained to ensure sufficient convergence in character localization, 360° orientation recovery, and multi-line sorting; then, a pre-trained DeepSeek Decoder is loaded, with only shallow and alignment projection layers fine-tuned, enabling the language side to quickly adapt to the steel plant numbering distribution without overfitting small-scale field data. This strategy fully utilizes the generalization ability of the pre-trained language model without increasing annotation and training costs. Simultaneously, by adjusting the feature pyramid network scale, graph neural network layer count, and beam width, real-time inference deployment under different computing power conditions is achieved, resulting in a unified balance of "high robustness, high accuracy, and low latency." Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a method for recognizing multi-line rotating billet number characters based on orientation modeling and Transformer decoding provided by an embodiment of the present invention; Figure 2 This is an overall flowchart of a multi-row arbitrary direction numbering recognition method provided by an embodiment of the present invention; Figure 3 This is a structural diagram of a multi-scale visual Transformer encoder provided in an embodiment of the present invention; Figure 4 This is a structural diagram of an Anchor-free rotating character detection module provided in an embodiment of the present invention; Figure 5 This invention provides a character 360° orientation recovery map based on directional distribution modeling. Figure 6 This invention provides a method for constructing a multi-line reading order graph and a character sorting graph. Figure 7 This is a character sequence generation graph based on visual conditional language decoding provided in an embodiment of the present invention; Figure 8 This is a block diagram of a multi-line rotating billet number character recognition device based on direction modeling and Transformer decoding provided in an embodiment of the present invention; Figure 9This is a schematic diagram of the structure of a multi-line rotating billet number character recognition device based on direction modeling and Transformer decoding provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0020] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0021] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0022] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0023] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0024] This invention provides a method for recognizing multi-line rotating billet number characters based on direction modeling and Transformer decoding. This method can be implemented by a multi-line rotating billet number character recognition device based on direction modeling and Transformer decoding, which can be a terminal or a server. Figure 1 The flowchart shown is for a multi-line rotating billet number character recognition method based on orientation modeling and Transformer decoding. The processing flow of this method may include the following steps:

[0025] S1. Obtain the on-site image dataset of the billet cross-section; preprocess the on-site image dataset of the billet cross-section to obtain the preprocessed image dataset; construct a multi-line rotating billet number character recognition model; wherein, the recognition model includes: a multi-scale visual Transformer encoder, an anchor-free rotating character detection head, an orientation distribution prediction head, and a pre-trained Transformer language decoding model.

[0026] In one feasible implementation, the industrial site is subject to high temperature radiation, smoke and dust interference, uneven lighting, and blurred imaging. Therefore, the on-site image of the billet cross-section is normalized to obtain a normalized image. The normalized image is then subjected to brightness and contrast enhancement, adaptive histogram equalization, and Gamma correction to improve the recognizability of the character area, thus obtaining a preprocessed image.

[0027] In one feasible implementation, this embodiment of the invention uses billet number images collected from actual industrial sites as the source of training and testing data. The collected billet number images include numbering instances under different shooting distances, lighting conditions, and posture conditions. The characters in the dataset may be rotated, inverted, or arranged in multiple rows, and include complex situations such as uneven coating, wear, and occlusion.

[0028] In the training phase of the end-to-end trainable multi-line rotating billet number character recognition model, a large-scale source domain number images are used as the main training data, while a small number of samples collected from the target site are introduced as supplementary data to enhance the model's generalization ability in the target application scenario. During the model inference phase, only single-frame images collected in real-time on-site are used as input, without relying on multi-frame or temporal information.

[0029] In this embodiment of the invention, a deep learning framework is used to implement an end-to-end trainable multi-line rotating billet number character recognition model structure, and an adaptive optimization algorithm is used to train the model parameters. During training, a fixed batch size and learning rate strategy is adopted, and the model performance is gradually optimized through multiple iterations. No data augmentation is introduced during the model inference stage; the entire recognition process from character detection to number sequence generation is completed based solely on a single frame of input image.

[0030] In one feasible implementation, this invention constructs an end-to-end trainable multi-line rotating billet number character recognition model, including: a multi-scale visual Transformer encoder, an anchor-free rotating character detection head, an orientation distribution prediction head, and a pre-trained Transformer language decoding model. During the model training phase, to enhance the model's robustness to complex scenes, the image is further subjected to random rotation, thermal ripple perturbation, motion blur perturbation, and random occlusion data augmentation operations to obtain a pre-processed image. During the model inference phase, only the input image is subjected to size adjustment and pixel normalization operations to ensure inference efficiency and result stability.

[0031] in, Figure 2 This is an overall flowchart of a multi-line arbitrary direction number recognition method provided by an embodiment of the present invention. In one feasible implementation, the number recognition process of this embodiment includes four stages: image preprocessing, model building, model training, and model inference. The model building stage includes multi-scale visual feature encoding, rotated character detection, character direction distribution modeling, multi-line reading order inference, and language model decoding generation. Through the above process, stable recognition of ladle numbers or billet numbers under arbitrary rotation and multi-line arrangement conditions is achieved.

[0032] S2. The preprocessed image dataset is used to train the multi-line rotating billet number character recognition model to obtain the trained recognition model.

[0033] S3. Obtain the cross-sectional image of the single billet to be identified, preprocess the cross-sectional image of the single billet to be identified to obtain the processed image; input the processed image into the multi-scale visual Transformer encoder in the trained recognition model for feature extraction to obtain a set of multi-scale feature maps.

[0034] in, Figure 3 This is a structural diagram of a multi-scale visual Transformer encoder provided in an embodiment of the present invention. In one feasible implementation, the preprocessed image is input into the multi-scale visual Transformer encoder for feature extraction. Taking an input image of size H×W×3 as an example, it is divided into multiple fixed-size image blocks, and each image block is linearly mapped to form a patch token representation. The global dependencies between patch tokens are modeled through multi-layer self-attention computation, and feature representations containing local character details and overall spatial layout are extracted level by level. The features output from different layers are fused through a feature pyramid network to form a multi-scale feature map for subsequent character detection and orientation prediction.

[0035] Optionally, the specific implementation process of S3 includes S31-S33: S31. Input the processed image into the multi-scale visual Transformer encoder, divide the preprocessed image into multiple fixed-size image blocks, and map the multiple fixed-size image blocks into a patch token sequence. S32. Based on the patch token sequence, a multi-head self-attention mechanism with multiple encoding stages is used to model the global dependency relationship between the patch token sequences and obtain an intermediate feature representation containing local character morphology information and global layout information. S33. Input the intermediate feature representation into the feature pyramid network for feature fusion to obtain a multi-scale feature map set.

[0036] Among them, the multi-scale feature map set is used to simultaneously cover small-sized character details and the overall spatial structure of multi-line numbering.

[0037] S4. Input the multi-scale feature map set into the Anchor-free rotating character detection head, detect each character instance in the image, and output a character candidate set; wherein, the character candidate set includes: the character type, center point coordinates, character width, basic rotation angle and confidence score of each character.

[0038] in, Figure 4 This is a structural diagram of an anchor-free rotated character detection module provided in an embodiment of the present invention. In one feasible implementation, an anchor-free rotated character detection head is constructed on a multi-scale feature map to detect single character instances in an image. The detection head outputs the character category, character center position, character width and height, and a basic rotation angle, wherein the basic rotation angle is represented by sine and cosine regression to enhance the stability of angle prediction. Redundant detection boxes are removed by rotation non-maximum suppression to obtain a character candidate set. For example, Figure 4 As shown in (a), the base rotation angle describes the tilt direction of the character's outer rotation frame relative to the horizontal axis, and its value is limited to [0, 180°]. Since the geometry of the character's outer rotation frame remains consistent after a 180° rotation, the base rotation angle alone cannot distinguish between the character's upright and inverted orientation. Figure 4 As shown in (b), for the same character instance, when their actual orientations differ by 180°, the regressed base rotation angles may still be consistent, leading to ambiguity in the character's top orientation. This ambiguity is particularly pronounced when the character is completely inverted or multiple lines of numbering are rotated, easily causing errors in subsequent reading order inference and character recognition. Therefore, this embodiment of the invention, based on rotated character detection, further introduces character orientation distribution modeling to achieve a unique determination of the character's true orientation.

[0039] S5. Use the direction distribution prediction head to calculate the two-dimensional direction vector corresponding to the true top direction of each character instance in the character candidate set, and perform normalization processing to obtain the unit direction vector of each character.

[0040] in, Figure 5 This invention provides a 360° character orientation recovery map based on orientation distribution modeling. To address the ambiguity of character orientation within a 180° range, an orientation distribution modeling module is introduced based on character detection results. This module predicts a two-dimensional orientation vector corresponding to the top orientation of each character and obtains a unit orientation vector through normalization. The absolute 360° orientation angle of the character is calculated based on the unit orientation vector and its consistency is verified with the base rotation angle, thereby obtaining uniquely determined character orientation information.

[0041] In one feasible implementation, for each character instance in the character candidate set, an orientation distribution prediction head is introduced at its corresponding positive sample position to regress the two-dimensional orientation vector corresponding to the true top orientation of the character.

[0042] In this process, the two-dimensional direction vector is L2 normalized to obtain the unit direction vector.

[0043] Optionally, the process of S5 using the direction distribution prediction head to calculate the two-dimensional direction vector corresponding to the true top direction of each character instance in the character candidate set, and performing normalization processing to obtain the unit direction vector of each character, is represented by the following formulas (1)-(2): (1) (2) in, Indicates the first k The character top direction prediction vector corresponding to each character instance; This represents the horizontal component of the character top direction prediction vector; This represents the vertical component of the character top direction prediction vector; This represents the unit direction vector obtained by normalizing the character top direction prediction vector; This represents the horizontal component of the unit direction vector; This represents the vertical component of the unit direction vector.

[0044] The unit direction vector points to the true top direction of the character, and this direction vector is supervised during model training using a direction consistency loss function based on cosine similarity.

[0045] S6. Calculate the absolute orientation angle of each character based on the unit orientation vector of each character; verify the consistency between the absolute orientation angle of each character and the basic rotation angle of each character to obtain the absolute orientation information of the characters; construct the final character set based on the absolute orientation information of the characters.

[0046] Optionally, the final character set includes: character category, spatial location, scale information, and absolute orientation information.

[0047] Optionally, the process of S6 calculating the absolute orientation angle of each character based on the unit orientation vector of each character is represented by the following formula (3): (3) in, Indicates the first k The final absolute orientation angle corresponding to each character instance is used to uniquely characterize the true rotation direction of the character in the image plane; This represents the horizontal component of the unit direction vector; This represents the vertical component of the unit direction vector; This represents the two-parameter arctangent function, used to calculate the corresponding polar angle based on the component relationships of the direction vector in a two-dimensional plane, and to map the angle range to [0, 360°).

[0048] In one feasible implementation, when there is a 180° quadrant conflict between the absolute orientation angle and the base rotation angle, the base rotation angle is flipped and corrected by ±180° to obtain a unique and stable 360° character orientation.

[0049] S7. Treat each character instance in the character set as a node in the graph; use a graph attention network based on the nodes to predict the directed order relationship between character nodes and obtain the reading order directed graph; perform topological sorting on the reading order directed graph to restore the in-line character order and output the multi-line reading order.

[0050] in, Figure 6 This invention provides a method for constructing a multi-line reading order graph and a character sorting graph. In one feasible implementation, after obtaining the spatial position, visual features, and absolute orientation of character instances, each character instance is considered as a node in the graph, thus constructing a multi-line reading order graph. Node features consist of the character's visual feature vector, position, scale, and orientation information. The directed order relationship between character nodes is predicted using a graph neural network, forming a directed reading order graph. The directed graph is then topologically sorted to restore the character order within each line, and the multi-line arrangement order is determined by combining the spatial relationship between character lines.

[0051] Optionally, the specific implementation process of S7 includes S71-S75: S71. Treat each character instance in the character set as a node in a graph; where the node features are composed of the character's ROI visual feature vector, center position coordinates, character width and height, absolute orientation information, and character category embedding. S72. Based on the conditions that the spatial distance between character instances is less than a preset distance threshold and the absolute orientation difference is less than a preset orientation threshold, establish candidate adjacent edges between nodes. S73. Based on each node and candidate adjacent edges, a graph attention network is used to predict the directed order relationship between character nodes to obtain a directed graph of reading order. S74. Perform loop breaking on the directed graph of reading order and perform topological sorting to restore the sequential links of characters within the line; S75. Based on the sequential link and the projection result of the character in the main reading direction, the line is segmented, and the line is sorted according to fixed rules for each character line, and finally the multi-line reading order sequence set is output.

[0052] S8. Convert each character instance into a visual token according to the multi-line reading order to obtain a visual token sequence arranged in the actual reading order; input the visual token sequence into the pre-trained Transformer language decoding model to generate a complete billet number character sequence through autoregression.

[0053] Each visual token is composed of the visual feature vector of the corresponding character and its spatial position and orientation information.

[0054] in, Figure 7 This invention provides a character sequence generation graph based on visual conditional language decoding. In one feasible implementation, after determining the final reading order of characters, character instances are sequentially converted into a sequence of visual tokens, which are then used as conditional inputs to a pre-trained Transformer language decoding model. The language decoding model models character historical dependencies through a masked self-attention mechanism and introduces constraint information from visual tokens through a visual conditional cross-attention mechanism. It then generates character sequences step by step in an autoregressive manner until the output sequence ends, obtaining a complete number recognition result.

[0055] In one feasible implementation, the pre-trained Transformer language decoding model includes: a mask self-attention module, a visual conditional cross-attention module, and a feedforward neural network module; wherein, the visual conditional cross-attention module is used to introduce conditional constraints provided by visual tokens during the character generation process.

[0056] In a feasible implementation manner, the specific implementation process of the pre-trained Transformer language decoding model includes: taking the starting token of the visual token sequence as the initial input of the decoding model, and jointly inputting it with the generated historical character sequence into the masked self-attention module. By imposing a causal mask constraint on the historical character positions, the current moment can only focus on the character information generated before the current character, thereby modeling the context dependencies within the character sequence, avoiding the leakage of future character information, and ensuring the autoregressive nature of the generation process.

[0057] Input the output result of the masked self-attention module into the visual conditional cross-attention module. Use the current character generation state as the query vector, and the visual token sequence as the key and value. Through the attention weighting mechanism, calculate the correlation weights between the current character and each visual token, thereby dynamically introducing the conditional constraints provided by the character visual features, spatial positions, and orientation information during the character generation process. In this way, the decoding model can combine the visual information of the corresponding character instance when generating each character, reducing the recognition ambiguity caused by similar character appearances or complex arrangements.

[0058] After completing the visual conditional cross-attention calculation, input the feature vector integrating the visual conditional information into the feed-forward neural network module to perform non-linear mapping and reconstruction of the features, enhancing the model's expression ability for character category discrimination, and outputting the predicted probability distribution of each candidate character at the current moment.

[0059] Based on the predicted probability distribution of each candidate character at the current moment, gradually generate a complete bloom number character sequence in an autoregressive manner.

[0060] Optionally, during the process of generating the complete bloom number character sequence, the conditional constraints provided by the visual tokens are satisfied, which is represented by the following formula (4): (4) Where, represents the character generated at the current moment; t represents the moment; y<t represents the character sequence composed of all the historical characters generated before the current moment; {z t} represents the visual token sequence; i represents the index number in the visual token sequence. , t , ,

[0061] , , i ,

[0062] } represents the visual token sequence; i represents the index number in the visual token sequence.

[0061] Among them, during the character decoding process in step S8, considering that the ladle number or bloom number is usually encoded in a fixed format mainly consisting of numbers, and the number structure has clear digit constraints and character type constraints, the embodiment of the present invention further introduces a digital priority and number format constraint mechanism in the decoding stage to improve the reliability and consistency of the number recognition results.

[0062] Specifically, after obtaining the predicted probability distribution of candidate characters at each decoding time step, a logit penalty is applied to the predicted probability of alphabetic characters to reduce the probability of alphabetic characters being selected at the current decoding position, thereby increasing the prior advantage of numeric characters as candidate outputs at the probability level. The logit penalty is implemented by additively shifting or scaling the logit value corresponding to alphabetic characters, without changing the model structure.

[0063] During the decoding process, a Beam Search strategy is incorporated, and a numbering rule mask is introduced to constrain candidate paths that do not conform to the preset numbering format. The numbering rule mask performs pruning or weighting on candidate characters or paths that do not meet the conditions based on the bit length requirements, character type requirements, and overall numbering structure rules corresponding to the current decoding position. For example, if the current position should output numeric characters, letter-type candidates are directly masked; when a candidate path violates the numbering length or line structure constraints, the path is terminated early.

[0064] In the candidate path selection process, the remaining candidate sequences are reordered by combining the context probabilities given by the language model. Candidate paths that simultaneously meet the visual conditions, language context probabilities, and numbering format constraints are retained first. This significantly suppresses the confusion between numbers and letters such as "0 / O, 1 / I, 5 / S or 8 / B", as well as output results with digit errors or abnormal line structures.

[0065] Through the above decoding control mechanism, the final generated number character sequence not only satisfies visual feature matching but also strictly conforms to the prior format of industrial numbering, thereby improving the stability and engineering usability of the overall recognition results.

[0066] In the field identification of billet numbers, achieving stable recognition under conditions of strong reflection, high-temperature thermal disturbance, coating defects, and arbitrary rotation (including inversion) is a core challenge for industrial application. Existing methods mostly rely on text line detection or character rotation detection followed by geometric sorting. While these approaches perform reasonably well with a relatively uniform training set distribution, they are prone to misdetection of line frames, disordered order, and missed detections / false detections when faced with inversion, cross-line adhesion, partial character loss, and background texture interference. This invention proposes a model that recognizes the common geometrical pattern of steel plant numbering: "fixed order across multiple lines + consistent character scale + consistent orientation." Therefore, it simultaneously models character positioning, orientation recovery, and order inference within a unified model. This allows the model to leverage structural priors to resist complex working conditions, significantly improving cross-scene robustness and overall recognition accuracy without increasing the cost of manual rule-making.

[0067] Existing rotation frame detection methods only provide angles within a 180° range, leading to ambiguity regarding upright / inverted orientation. Some methods extend this to 360° through corner point regression, but the high dimensionality of corner points makes them sensitive to noise and prone to orientation jumps under conditions such as missing paint strokes, smudges, and reflections, resulting in global sorting errors. This invention employs a combination of "180° basic half-angle rotation detection + orientation distribution regression of the top direction unit vector d." By replacing high-dimensional corner point regression with a 2D continuous unit vector, the orientation prediction space becomes lower-dimensional, smoother, and more convergent. Consistency verification achieves 360° unique orientation recovery, eliminating orientation uncertainty in inverted scenarios from the source and improving the stability of detection and subsequent reading order recovery for arbitrary rotation numbers.

[0068] Existing geometric sorting methods based on coordinate thresholds are unstable under conditions of uneven line spacing, overall rotation / inversion, and local missing characters, easily leading to cross-line splicing or character order reversal. This invention constructs a directed graph of reading order based on character centers and absolute orientation. It explicitly learns the "adjacent-pointing" relationship between characters through a graph attention network, and restores intra-line links by combining loop breaking and topological sorting. Inter-line order is restored through main direction projection clustering. This maintains global reading link consistency even in complex arrangements and missing characters, reducing the engineering failure rate of multi-line numbering order disorder and improving interpretability.

[0069] The billet numbers are primarily composed of pure numbers or numbers predominantly, but traditional visual character-by-character classification lacks sequence priors, making it prone to confusion between numbers and letters such as "0 / O, 1 / I, 5 / S, 8 / B" under noise, and difficult to self-correct. This invention introduces a pre-trained Transformer language decoding model compatible with the DeepSeek architecture. The visual token sequence is input into the decoder as the encoder memory. During generation, contextual probabilities are used for sequence-level discrimination, and a number-priority logit penalty and beam search constraints on the numbering format are applied. This simultaneously suppresses paths that do not conform to the number distribution and format rules at both the probability and search levels, enabling automatic error correction and reordering even under visual uncertainty. This significantly reduces errors such as misidentifying numbers as letters, incorrect number of digits, or inconsistent row structure, improving the reliability of industrial numbering recognition.

[0070] To ensure both accuracy and deployability, this invention employs an end-to-end joint training and phased optimization strategy: first, the detection, orientation, and order modules on the visual side are trained to ensure sufficient convergence in character localization, 360° orientation recovery, and multi-line sorting; then, a pre-trained DeepSeek Decoder is loaded, with only shallow and alignment projection layers fine-tuned, enabling the language side to quickly adapt to the steel plant numbering distribution without overfitting small-scale field data. This strategy fully utilizes the generalization ability of the pre-trained language model without increasing annotation and training costs. Simultaneously, by adjusting the feature pyramid network scale, graph neural network layer count, and beam width, real-time inference deployment under different computing power conditions is achieved, resulting in a unified balance of "high robustness, high accuracy, and low latency."

[0071] Figure 8 This is a block diagram of a multi-line rotating billet number character recognition device based on direction modeling and Transformer decoding, provided by an embodiment of the present invention. This device is used for a multi-line rotating billet number character recognition method based on direction modeling and Transformer decoding. (Refer to...) Figure 8 The device includes an acquisition and construction unit 810, a training unit, a feature extraction unit 830, a first output unit 840, a first calculation unit 850, a second calculation unit 860, a second output unit 870, and a generation unit 880. Wherein:

[0072] The acquisition and construction unit 810 is used to acquire a dataset of on-site images of the billet cross-section; preprocess the on-site image dataset of the billet cross-section to obtain a preprocessed image dataset; and construct a multi-line rotating billet number character recognition model; wherein, the recognition model includes: a multi-scale visual Transformer encoder, an anchor-free rotating character detection head, an orientation distribution prediction head, and a pre-trained Transformer language decoding model. Training unit 820 is used to train the multi-line rotating billet number character recognition model using the preprocessed image dataset to obtain the trained recognition model; The feature extraction unit 830 is used to acquire a cross-sectional image of a single billet to be identified, preprocess the cross-sectional image of the single billet to be identified to obtain a processed image, and input the processed image into the multi-scale visual Transformer encoder in the trained recognition model for feature extraction to obtain a set of multi-scale feature maps. The first output unit 840 is used to input a multi-scale feature map set into the Anchor-free rotating character detection head, detect each character instance in the image, and output a character candidate set; wherein, the character candidate set includes: the character type, center point coordinates, character width, basic rotation angle and confidence score of each character; The first calculation unit 850 is used to calculate the two-dimensional direction vector corresponding to the true top direction of each character instance in the character candidate set using the direction distribution prediction head, and to perform normalization processing to obtain the unit direction vector of each character. The second calculation unit 860 is used to calculate the absolute orientation angle of each character based on the unit orientation vector of each character; to verify the consistency between the absolute orientation angle of each character and the basic rotation angle of each character to obtain the absolute orientation information of the characters; and to construct the final character set based on the absolute orientation information of the characters. The second output unit 870 is used to treat each character instance in the character set as a node in a graph; based on the nodes, a graph attention network is used to predict the directed order relationship between character nodes to obtain a reading order directed graph; the reading order directed graph is topologically sorted to restore the in-line character order and output the multi-line reading order; The generation unit 880 is used to convert each character instance into a visual token according to the multi-line reading order to obtain a visual token sequence arranged in the actual reading order; the visual token sequence is input into a pre-trained Transformer language decoding model to generate a complete billet number character sequence through autoregression.

[0073] Optionally, the feature extraction unit 830 is used for: The processed image is input into a multi-scale visual Transformer encoder, which divides the preprocessed image into multiple fixed-size image blocks and maps these fixed-size image blocks into a sequence of patch tokens. Based on the patch token sequence, a multi-head self-attention mechanism with multiple encoding stages is used to model the global dependency relationship between the patch token sequences and obtain intermediate feature representations containing local character morphological information and global layout information. The intermediate feature representations are input into the feature pyramid network for feature fusion to obtain a multi-scale feature map set.

[0074] Optionally, the process of using the direction distribution prediction head to calculate the two-dimensional direction vector corresponding to the true top direction of each character instance in the character candidate set, and performing normalization processing to obtain the unit direction vector of each character, is represented by the following formulas (1)-(2): (1) (2) in, Indicates the first k The character top direction prediction vector corresponding to each character instance; This represents the horizontal component of the character top direction prediction vector; This represents the vertical component of the character top direction prediction vector; This represents the unit direction vector obtained by normalizing the character top direction prediction vector; This represents the horizontal component of the unit direction vector; This represents the vertical component of the unit direction vector.

[0075] Optionally, the process of calculating the absolute orientation angle of each character based on the unit orientation vector of each character is represented by the following formula (3): (3) in, Indicates the first k The final absolute orientation angle corresponding to each character instance is used to uniquely characterize the true rotation direction of the character in the image plane; This represents the horizontal component of the unit direction vector; This represents the vertical component of the unit direction vector; This represents the two-parameter arctangent function, used to calculate the corresponding polar angle based on the component relationships of the direction vector in a two-dimensional plane, and to map the angle range to [0, 360°).

[0076] Optionally, the second output unit 870 is used for: Each character instance in the character set is considered as a node in a graph; the node features are composed of the character's ROI visual feature vector, center position coordinates, character width and height, absolute orientation information, and character category embedding. Candidate adjacent edges are established between nodes based on the conditions that the spatial distance between character instances is less than a preset distance threshold and the absolute orientation difference is less than a preset orientation threshold. Based on each node and candidate adjacent edges, a graph attention network is used to predict the directed order relationship between character nodes to obtain a directed graph of reading order. The reading order directed graph is broken by performing loop breaking and topological sorting to restore the sequential links of characters within a line; Line segmentation is performed based on the sequential link and the projection results of characters in the main reading direction. Then, a fixed rule is applied to the inter-line sorting of each character line, and finally, a set of multi-line reading order sequences is output.

[0077] Optionally, the final character set includes: character category, spatial location, scale information, and absolute orientation information.

[0078] Optionally, the process of generating the complete billet number character sequence satisfies the conditional constraints provided by the visual token, as expressed by the following formula (4): (4) wherein represents the character generated at the current moment; t represents the moment; y<t represents the character sequence composed of all historical characters generated before the current moment t ; {z i} represents the visual token sequence; i represents the index number in the visual token sequence.

[0079] In the problem of on-site recognition of billet numbers, how to achieve stable recognition under the conditions of strong reflection, high-temperature thermal disturbance, spraying defects and arbitrary rotation (including inversion) is the core difficulty in industrial implementation. Most existing methods rely on text line detection or character rotation detection followed by geometric sorting. Such solutions have acceptable effects when the training set distribution is relatively single, but are prone to misdetection of line frames, disorder of order, and missed and false detections when facing inversion,跨行粘连、局部缺字和背景纹理干扰时容易产生行框错检、顺序错乱和漏检误检。本发明实施例认为钢厂编号在几何结构上具有“多行固定顺序+字符尺度一致+方向一致”的共性规律,因此在统一模型中同时建模字符定位、朝向恢复与顺序推断,使模型能够利用结构先验抵抗复杂工况扰动,从而在不增加额外人工规则成本的前提下显著提升跨场景鲁棒性与整体识别准确率。

[0080] Existing rotated bounding box detection can only provide an angle range of 180°, resulting in upright / inverted ambiguity; some methods extend to 360° through corner regression, but the corner dimension is high, sensitive to noise, and prone to orientation jumps under spraying missing strokes, stain occlusion and reflection,继而引发全局排序错误。本发明实施例采用“180°基础半角旋转检测+顶部方向单位向量d的方向分布回归”的组合方式,以2维连续单位向量替代高维角点回归,使方向预测空间更低维、更平滑以及更易收敛,并通过一致性校验实现360°唯一朝向恢复,从源头消除倒置场景下的方向不确定性,提高任意旋转编号的检测与后续阅读顺序恢复稳定性。

[0081] Existing geometric sorting based on coordinate thresholds is unstable in the case of uneven line spacing, overall rotation / inversion and local deletion, and is prone to跨行拼接或字序反转。本发明实施例构建基于字符中心与绝对朝向的阅读顺序有向图,通过图注意力网络显式学习字符间“相邻—指向”关系,并结合断环与拓扑排序恢复行内链路,通过主方向投影聚类恢复行间顺序,从而在复杂排列和缺字情况仍可维持全局阅读链路一致,降低多行编号顺序错乱的工程失败率并提升可解释性。

[0082] It should be noted that there are some Chinese characters in the original text that seem to be incorrect or incomplete in the description part. I have translated them as accurately as possible according to the context. You may need to check and correct the original text for more accurate understanding.The billet numbers are primarily composed of pure numbers or numbers predominantly, but traditional visual character-by-character classification lacks sequence priors, making it prone to confusion between numbers and letters such as "0 / O, 1 / I, 5 / S, 8 / B" under noise, and difficult to self-correct. This invention introduces a pre-trained Transformer language decoding model compatible with the DeepSeek architecture. The visual token sequence is input into the decoder as the encoder memory. During generation, contextual probabilities are used for sequence-level discrimination, and a number-priority logit penalty and beam search constraints on the numbering format are applied. This simultaneously suppresses paths that do not conform to the number distribution and format rules at both the probability and search levels, enabling automatic error correction and reordering even under visual uncertainty. This significantly reduces errors such as misidentifying numbers as letters, incorrect number of digits, or inconsistent row structure, improving the reliability of industrial numbering recognition.

[0083] To ensure both accuracy and deployability, this invention employs an end-to-end joint training and phased optimization strategy: first, the detection, orientation, and order modules on the visual side are trained to ensure sufficient convergence in character localization, 360° orientation recovery, and multi-line sorting; then, a pre-trained DeepSeek Decoder is loaded, with only shallow and alignment projection layers fine-tuned, enabling the language side to quickly adapt to the steel plant numbering distribution without overfitting small-scale field data. This strategy fully utilizes the generalization ability of the pre-trained language model without increasing annotation and training costs. Simultaneously, by adjusting the feature pyramid network scale, graph neural network layer count, and beam width, real-time inference deployment under different computing power conditions is achieved, resulting in a unified balance of "high robustness, high accuracy, and low latency."

[0084] Figure 9 This is a schematic diagram of the structure of a multi-line rotating billet number character recognition device based on direction modeling and Transformer decoding provided in an embodiment of the present invention, as shown below. Figure 9 As shown, a multi-line rotating billet number character recognition device based on orientation modeling and Transformer decoding may include the above-mentioned... Figure 8 The illustrated device is a multi-line rotating billet number character recognition device based on orientation modeling and Transformer decoding. Optionally, the multi-line rotating billet number character recognition device 910 based on orientation modeling and Transformer decoding may include a first processor 2001.

[0085] Optionally, the multi-line rotary billet number character recognition device 910 based on orientation modeling and Transformer decoding may also include a memory 2002 and a transceiver 2003.

[0086] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0087] The following is combined with Figure 9 A detailed introduction to each component of the 910 multi-line rotating billet number character recognition device based on orientation modeling and Transformer decoding is provided below: The first processor 2001 is the control center of the multi-line rotating billet number character recognition device 910 based on direction modeling and Transformer decoding. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0088] Optionally, the first processor 2001 can execute various functions of the multi-line rotary billet number character recognition device 910 based on orientation modeling and Transformer decoding by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0089] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 9 CPU0 and CPU1 are shown in the diagram.

[0090] In a specific implementation, as one example, the multi-line rotating billet number character recognition device 910 based on direction modeling and Transformer decoding may also include multiple processors, for example... Figure 9 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0091] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0092] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or exist independently, and may be connected via the interface circuit of the multi-line rotating billet number character recognition device 910 based on orientation modeling and Transformer decoding. Figure 9 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0093] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0094] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 9 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0095] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and can be connected to the interface circuit of the multi-line rotating billet number character recognition device 910 based on orientation modeling and Transformer decoding. Figure 9 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0096] It should be noted that, Figure 9 The structure of the multi-line rotating billet number character recognition device 910 based on orientation modeling and Transformer decoding shown in the figure does not constitute a limitation on the router. The actual multi-line rotating billet number character recognition device based on orientation modeling and Transformer decoding may include more or fewer components than shown, or combine some components, or have different component arrangements.

[0097] Furthermore, the technical effect of the multi-line rotating billet number character recognition device 910 based on orientation modeling and Transformer decoding can be referred to the technical effect of the multi-line rotating billet number character recognition method based on orientation modeling and Transformer decoding described in the above method embodiments, and will not be repeated here.

[0098] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or it may be any conventional processor, etc.

[0099] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0100] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0101] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0102] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0103] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0104] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0105] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0106] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0107] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0108] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0109] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0110] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for recognizing multi-line rotating billet number characters based on direction modeling and Transformer decoding, characterized in that, The method includes: S1. Obtain the on-site image dataset of the billet cross-section; preprocess the on-site image dataset of the billet cross-section to obtain the preprocessed image dataset; construct a multi-line rotating billet number character recognition model; wherein, the recognition model includes: a multi-scale visual Transformer encoder, an anchor-free rotating character detection head, an orientation distribution prediction head, and a pre-trained Transformer language decoding model; S2. The preprocessed image dataset is used to train the multi-line rotating billet number character recognition model to obtain the trained recognition model. S3. Obtain the cross-sectional image of the single billet to be identified, preprocess the cross-sectional image of the single billet to be identified, and obtain the processed image; input the processed image into the multi-scale visual Transformer encoder in the trained recognition model for feature extraction, and obtain a set of multi-scale feature maps. S4. Input the multi-scale feature map set into the Anchor-free rotating character detection head, detect each character instance in the image, and output a character candidate set; wherein, the character candidate set includes: the character type, center point coordinates, character width, basic rotation angle and confidence score of each character; S5. Use the direction distribution prediction head to calculate the two-dimensional direction vector corresponding to the true top direction of each character instance in the character candidate set, and perform normalization processing to obtain the unit direction vector of each character. S6. Calculate the absolute orientation angle of each character based on the unit orientation vector of each character; verify the consistency between the absolute orientation angle of each character and the basic rotation angle of each character to obtain the absolute orientation information of the characters; construct the final character set based on the absolute orientation information of the characters. S7. Treat each character instance in the character set as a node in a graph; predict the directed order relationship between character nodes using a graph attention network based on the nodes to obtain a directed graph of reading order; perform topological sorting on the directed graph of reading order to restore the in-line character order and output the multi-line reading order. S8. Convert each character instance into a visual token according to the multi-line reading order to obtain a visual token sequence arranged in the actual reading order; input the visual token sequence into the pre-trained Transformer language decoding model to generate a complete billet number character sequence through autoregression.

2. The method for recognizing multi-line rotating billet number characters based on direction modeling and Transformer decoding according to claim 1, characterized in that, S3 inputs the processed image into the multi-scale visual Transformer encoder in the trained recognition model for feature extraction, obtaining a multi-scale feature map set, including: S31. Input the processed image into the multi-scale visual Transformer encoder, divide the preprocessed image into multiple fixed-size image blocks, and map the multiple fixed-size image blocks into a patch token sequence. S32. Based on the patch token sequence, a multi-head self-attention mechanism with multiple encoding stages is used to model the global dependency relationship between the patch token sequences and obtain intermediate feature representations containing local character morphological information and global layout information. S33. Input the intermediate feature representation into the feature pyramid network for feature fusion to obtain a multi-scale feature map set.

3. The method for recognizing multi-line rotating billet number characters based on direction modeling and Transformer decoding according to claim 1, characterized in that, The process of S5, which uses a direction distribution prediction head to calculate the two-dimensional direction vector corresponding to the true top direction of each character instance in the character candidate set and performs normalization processing to obtain the unit direction vector of each character, is represented by the following formulas (1)-(2): （1） （2） in, Indicates the first k The character top direction prediction vector corresponding to each character instance; This represents the horizontal component of the character top direction prediction vector; This represents the vertical component of the character top direction prediction vector; This represents the unit direction vector obtained by normalizing the character top direction prediction vector; This represents the horizontal component of the unit direction vector; This represents the vertical component of the unit direction vector.

4. The method for recognizing multi-line rotating billet number characters based on direction modeling and Transformer decoding according to claim 1, characterized in that, The process of S6 calculating the absolute orientation angle of each character based on the unit orientation vector of each character is represented by the following formula (3): （3） in, Indicates the first k The final absolute orientation angle corresponding to each character instance is used to uniquely characterize the true rotation direction of the character in the image plane; This represents the horizontal component of the unit direction vector; This represents the vertical component of the unit direction vector; This represents the two-parameter arctangent function, used to calculate the corresponding polar angle based on the component relationships of the direction vector in a two-dimensional plane, and to map the angle range to [0, 360°).

5. The method for recognizing multi-line rotating billet number characters based on direction modeling and Transformer decoding according to claim 1, characterized in that, S7 treats each character instance in the character set as a node in a graph; based on the nodes, a graph attention network is used to predict the directed order relationship between character nodes to obtain a directed graph of reading order. Perform a topological sort on the directed graph of reading order, restore the character order within each line, and output the reading order of multiple lines, including: S71. Treat each character instance in the character set as a node in a graph; where the node features are composed of the character's ROI visual feature vector, center position coordinates, character width and height, absolute orientation information, and character category embedding. S72. Based on the conditions that the spatial distance between character instances is less than a preset distance threshold and the absolute orientation difference is less than a preset orientation threshold, establish candidate adjacent edges between nodes. S73. Based on each node and candidate adjacent edges, a graph attention network is used to predict the directed order relationship between character nodes to obtain a directed graph of reading order. S74. Perform loop breaking on the directed graph of reading order and perform topological sorting to restore the sequential links of characters within the line; S75. Based on the sequential link and the projection result of the character in the main reading direction, the line is segmented, and the line is sorted according to fixed rules for each character line, and finally the multi-line reading order sequence set is output.

6. The method for recognizing multi-line rotating billet number characters based on direction modeling and Transformer decoding according to claim 1, characterized in that, The final character set includes: character category, spatial location, scale information, and absolute orientation information.

7. The method for recognizing multi-line rotating billet number characters based on direction modeling and Transformer decoding according to claim 1, characterized in that, The process of generating the complete billet number character sequence satisfies the conditional constraints provided by the visual token, as expressed by the following formula (4): （4） Among them, represents the character generated at the current moment; t represents the moment; y<t represents the current moment t is the character sequence composed of all historical characters that have been generated before; {z i} represents the visual token sequence; i represents the index number in the visual token sequence.

8. A multi-line rotating billet number character recognition device based on direction modeling and Transformer decoding, wherein the multi-line rotating billet number character recognition device based on direction modeling and Transformer decoding is used to implement the multi-line rotating billet number character recognition method based on direction modeling and Transformer decoding as described in any one of claims 1-7, characterized in that, The device includes: The acquisition and construction unit is used to acquire a dataset of on-site images of the cross-section of the billet; preprocess the on-site image dataset of the cross-section of the billet to obtain a preprocessed image dataset; and construct a multi-line rotating billet number character recognition model; wherein, the recognition model includes: a multi-scale visual Transformer encoder, an anchor-free rotating character detection head, an orientation distribution prediction head, and a pre-trained Transformer language decoding model. The training unit is used to train the multi-line rotating billet number character recognition model using the preprocessed image dataset to obtain the trained recognition model. The feature extraction unit is used to acquire a cross-sectional image of a single billet to be identified, preprocess the cross-sectional image of the single billet to be identified to obtain a processed image, and input the processed image into the multi-scale visual Transformer encoder in the trained recognition model for feature extraction to obtain a set of multi-scale feature maps. The first output unit is used to input a multi-scale feature map set into the Anchor-free rotating character detection head, detect each character instance in the image, and output a character candidate set; wherein, the character candidate set includes: the character type, center point coordinates, character width, basic rotation angle, and confidence score of each character; The first calculation unit is used to calculate the two-dimensional direction vector corresponding to the true top direction of each character instance in the character candidate set using the direction distribution prediction head, and to perform normalization processing to obtain the unit direction vector of each character. The second calculation unit is used to calculate the absolute orientation angle of each character based on the unit orientation vector of each character; to verify the consistency between the absolute orientation angle of each character and the basic rotation angle of each character to obtain the absolute orientation information of the characters; and to construct the final character set based on the absolute orientation information of the characters. The second output unit is used to treat each character instance in the character set as a node in a graph; based on the nodes, a graph attention network is used to predict the directed order relationship between character nodes to obtain a reading order directed graph; the reading order directed graph is topologically sorted to restore the in-line character order and output the multi-line reading order; The generation unit is used to convert each character instance into a visual token according to the reading order of multiple lines, and obtain a visual token sequence arranged in the actual reading order; the visual token sequence is input into a pre-trained Transformer language decoding model to generate a complete billet number character sequence through autoregression.

9. A multi-line rotating billet number character recognition device based on direction modeling and Transformer decoding, characterized in that, The multi-line rotary billet number character recognition device based on orientation modeling and Transformer decoding includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 7.

Images