A billet number recognition method and system based on 3D vision and multiple model series connection

Abstract

This invention discloses a billet number recognition method and system based on 3D vision and multi-model concatenation, belonging to the field of visual inspection technology. The invention employs a 3D camera with strong anti-light performance to acquire billet position information. The acquired depth information accurately detects the end position of the billet. Then, using the detected position information, the billet end image is cropped from an RGB image. A classification network is used to determine the presence and rotation direction of text on the billet. A YOLO object detection algorithm is used to detect multiple lines of billet number regions, followed by proportional slicing to obtain a complete single-line billet number sequence image. A convolutional recurrent neural network is used to batch recognize the billet numbers, and the results are sent back to a material tracking platform for verification. This invention utilizes both 2D and 3D data to improve the billet number recognition accuracy. Simultaneously, the multi-model concatenation billet number recognition algorithm has stronger generalization capabilities, can meet concurrent business recognition requirements, and offers higher system robustness.

Description

Technical Field

[0001] This invention belongs to the field of visual inspection technology, specifically relating to a billet number recognition method and system based on 3D vision and multi-model cascading. Background Technology

[0002] The information disclosed in this background section is intended only to enhance understanding of the overall background of the invention and is not necessarily to be construed as an admission or in any way implying that such information constitutes prior art known to those skilled in the art.

[0003] In steel production, billet numbers facilitate billet production, quality management, and traceability. Through billet numbers, steel mills can track the production status of each batch of billets, including raw material sources, production time, and processing technology. This helps improve production efficiency and reduce defect rates. Simultaneously, billet numbers serve as a quality management tool, facilitating the tracking and verification of billet quality. Furthermore, billet numbers act as an important document in trade, ensuring the rights and security of both parties in a transaction. Therefore, the identification of billet numbers plays a crucial role in steel production management.

[0004] Currently, there are many technologies for billet number character recognition systems, and they are widely used. Commonly used recognition methods include OCR recognition based on traditional image processing algorithms, image OCR recognition based on convolutional neural networks, and image OCR recognition based on template matching. All of these methods rely on recognition based on two-dimensional images captured by a camera. The quality of the acquired images is easily affected by factors such as lighting and noise, which in turn affects the billet number recognition rate. Summary of the Invention

[0005] The purpose of this invention is to realize billet number identification in the billet feeding process of steel rolling, and then obtain material information such as raw material source, production time, and processing technology through the billet number. This invention proposes a billet number identification method and system based on 3D vision and multi-model cascading.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] The first aspect of the present invention provides a billet number identification method based on 3D vision and multi-model concatenation, comprising the following steps:

[0008] (1) When the billet just enters the furnace roller conveyor, point cloud P1 is collected; after the billet enters the furnace roller conveyor, point cloud P2 and RGB image are collected at the same time.

[0009] (2) Using the collected point clouds P1 and P2, the position of the billet is calculated using a 3D vision algorithm; combined with the position of the billet, the end image of the billet in the RGB image is extracted.

[0010] (3) The state of the obtained billet end image is adjusted by using a classification network, and the billet end image is rotated to 0°.

[0011] (4) The YOLO target detection algorithm is used to detect the end image of the billet and obtain a complete single-row billet number sequence image;

[0012] (5) Identify the complete single-row billet number sequence image and match the identified billet number with the billet number of the current batch to determine the correctness of the identification.

[0013] In some embodiments of the present invention, a 3D camera is used to acquire point clouds.

[0014] In some embodiments of the present invention, the 3D vision algorithm includes:

[0015] A kd-tree structure is built for the collected point cloud P2. The background point cloud P1 is traversed to find the nearest neighbor point d. A distance threshold dmax is set. If the distance d from the nearest neighbor point is less than dmax, it is considered a background point and the point is deleted. If it is greater than dmax, it is considered a billet position point and the point is retained. The final billet position is obtained by using the aspect ratio feature of the fixed circumscribed rectangle of the billet.

[0016] In some embodiments of the present invention, the state of the billet end image includes: 0-no inkjet printing, 1-unclear inkjet printing, 2-0°, 3-90°, 4-180°, 5-270°. If the identification result is 3-5, rotate it to the 0° state. If the identification result is no inkjet printing or unclear inkjet printing, manual confirmation and supplementary recording are performed.

[0017] In some embodiments of the present invention, the YOLO target detection algorithm is used to detect the billet number position of the input billet end image to obtain a rectangular billet number image. The rectangular billet number image is then divided into equidistant segments, straightened, and stitched together to obtain a complete single-row billet number sequence image.

[0018] In some embodiments of the present invention, the rectangular billet number image is divided into equal segments along the Y direction according to the number of rows, taking advantage of the characteristics of the rectangular billet number. The image after equal segmentation can completely cover the billet number of the row without any character truncation. The segmented images are straightened and spliced ​​to obtain a complete single-row billet number sequence image.

[0019] In some embodiments of the present invention, a convolutional recurrent neural network is used to identify the billet number of the input complete single-line billet number sequence image and output the billet number identification result. At the same time, a generative adversarial network is used to replace the text in the original billet number image with custom text while preserving the style and position of the billet number text and the background of the billet end, so as to prevent the model from overfitting.

[0020] In some embodiments of the present invention, the correctness of the identified billet number is verified by the billet number compilation rules, and at the same time, the billet numbers in the current batch are searched, and the maximum similarity calculation is performed on the billet numbers that are not searched, so as to match the most likely billet number.

[0021] Preferably, the formula for calculating the maximum similarity is:

[0022]

[0023] In the formula, X represents the identified billet number, and Y represents the billet number in the current production batch list.

[0024] A second aspect of the present invention provides a billet number recognition system based on 3D vision and multi-model cascading, including a furnace feed roller conveyor, a first proximity switch disposed at the inlet of the furnace feed roller conveyor, a second proximity switch disposed in the middle and rear part of the furnace feed roller conveyor, a 3D camera disposed on the upper part of the furnace feed roller conveyor, and a system server.

[0025] The 3D camera is used to capture point cloud P1 when the billet just enters the furnace roller conveyor and passes the first proximity switch; when the billet enters the furnace roller conveyor and passes the second proximity switch, the 3D camera captures point cloud P2 and RGB image; and simultaneously transmits the captured point cloud and RGB image to the system server.

[0026] The system server includes a location calculation module, an image adjustment module, a sequence extraction module, and a verification and correction module;

[0027] The position calculation module is used to calculate the position of the billet using point clouds P1 and P2 acquired by the 3D camera and a 3D vision algorithm; and to extract the end image of the billet from the RGB image based on the billet position.

[0028] The image adjustment module is used to adjust the state of the obtained billet end image using a classification network, and rotate the billet end image to 0°.

[0029] The sequence extraction module is used to detect the end image of the billet using the YOLO target detection algorithm to obtain a complete single-row billet number sequence image;

[0030] The verification and correction module is used to identify a complete single-row billet number sequence image and match the identified billet number with the billet number of the current batch to determine the correctness of the identification.

[0031] In some embodiments of the present invention, the state of the billet end image includes: 0-no inkjet printing, 1-unclear inkjet printing, 2-0°, 3-90°, 4-180°, 5-270°. If the identification result is 3-5, rotate it to the 0° state. If the identification result is no inkjet printing or unclear inkjet printing, manual confirmation and supplementary recording are performed.

[0032] In some embodiments of the present invention, the YOLO target detection algorithm is used to detect the billet number position of the input billet end image to obtain a rectangular billet number image. The rectangular billet number image is then divided into equidistant segments, straightened, and stitched together to obtain a complete single-row billet number sequence image.

[0033] The beneficial effects of this invention are as follows:

[0034] This invention discloses a billet number recognition method based on 3D vision and multi-model cascading, comprising the following steps: When the billet just enters the furnace feed roller conveyor, point cloud P1 is acquired; after the billet enters the furnace feed roller conveyor, point cloud P2 and RGB images are acquired simultaneously; using the acquired point clouds P1 and P2, a 3D vision algorithm is used to calculate the billet number position; combining the billet number position, the billet number image is extracted from the RGB image; a classification network is used to adjust the state of the obtained billet number image, rotating the billet number image to 0°; a multi-row billet number region detection algorithm is used to detect the billet number image, obtaining a complete single-row billet number sequence image; the complete single-row billet number sequence image is identified, and the identified billet number is matched with the billet number of the current batch to determine the correctness of the identification. This invention employs a 3D camera with strong anti-light performance to acquire billet position information. The acquired depth information enables precise detection of the billet end position. The detected position information is then used to crop the billet end image from an RGB image. A classification network is used to determine the presence and rotation direction of text on the billet. A YOLO object detection algorithm is used to detect multiple lines of billet number regions and then slices them proportionally. A text recognition network (i.e., a convolutional recurrent neural network) is used to batch recognize multiple lines of text. The results are then sent back to the material tracking platform for verification. This invention utilizes both 2D and 3D data to improve the billet number recognition accuracy. Furthermore, a multi-model cascaded billet number recognition algorithm is designed, resulting in stronger generalization capabilities, the ability to handle concurrent business recognition, and higher system robustness. Attached Figure Description

[0035] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0036] Figure 1 This is a flowchart of a billet number identification method based on 3D vision and multi-model cascading according to the present invention.

[0037] Figure 2 This invention relates to a billet number identification method based on 3D vision and multi-model cascading, which implements billet number region detection.

[0038] Figure 3 This is a structural diagram of a billet number recognition system based on 3D vision and multi-model cascading according to the present invention.

[0039] In the diagram, 1-furnace feed roller, 2-first set of proximity switches, 3-second set of proximity switches, 4-3D camera, 5-system server, 6-heating furnace. Detailed Implementation

[0040] To enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention will be described in detail below with reference to specific embodiments.

[0041] Example 1

[0042] A billet number recognition method based on 3D vision and multi-model cascading includes the following steps:

[0043] (1) Obtain material information: Connect with the upstream MES system interface to obtain the current batch billet number and its associated material information from the upstream steelmaking system and transmit it to the rolling mill material tracking platform.

[0044] (2) 3D camera acquisition: Two pairs of proximity switches are installed at the beginning and middle and rear of the furnace feed roller conveyor. When the billet just enters the furnace feed roller conveyor, it passes through the first set of proximity switches and triggers the 3D camera to take pictures and acquire the background point cloud P1 of the roller conveyor without billet. When the billet enters the furnace feed roller conveyor, it passes through the second set of proximity switches and triggers the 3D camera to take pictures again. At this time, the point cloud P2 and RGB image I1 will be acquired at the same time.

[0045] (3) Billet position detection: Using the collected point clouds P1 and P2, the billet position is detected by point cloud difference. Specifically, a kd-tree structure is built for the collected point cloud P2, the background point cloud P1 is traversed, the nearest neighbor point d is found, and a distance threshold dmax is set. If the distance d from the nearest neighbor point is less than dmax, it is considered a background point and deleted; if it is greater than dmax, it is considered a billet position point and the point is retained to obtain the billet position. Then, the precise position of the billet end is extracted based on the rectangular fixed aspect ratio feature of the billet end, and the billet image in the RGB image I1 is cropped to obtain the billet end image.

[0046] (4) Billet Number Status Classification: Since billets may be in different states when transported to the furnace roller conveyor, such as character rotation, no coding, or unclear coding, this invention uses a classification neural network to classify the billet number status in the billet end image. The billet number status types include: 0 - no coding, 1 - unclear coding, 2 - 0°, 3 - 90°, 4 - 180°, and 5 - 270°. For categories with identification results of 3-5, the billet is rotated to the 0° state; for those with no coding or unclear coding, the material tracking platform is notified for manual confirmation and supplementary recording.

[0047] (5) Steel billet grade area inspection: such as Figure 2The billet numbers have characteristics of equal length, equal height, and equal row spacing. To avoid increasing the complexity of the detection model, the YOLO rotation target detection algorithm is used to detect the billet number position in the input billet end image, cropping it into a rectangular billet number image. Then, utilizing the characteristics of the billet numbers, the rectangular billet number image is divided equally along the Y direction according to the number of rows. The divided image can completely cover the billet number in that row without any character truncation, ensuring subsequent character recognition. At the same time, the YOLO rotation target detection algorithm supports situations such as camera rotation due to abnormal conditions, automatically detecting and segmenting rotated character regions, and the model has good generalization ability. The segmented images are straightened and stitched together to obtain a complete single-row billet number sequence image.

[0048] (6) Billet Number Recognition: A Convolutional Recurrent Neural Network (CRNN) is used to recognize the billet numbers in the input complete single-line billet number sequence image, and the output billet number recognition result is output. The CRNN supports characters of arbitrary length and has a small model size and fast speed. At the same time, in order to improve the recognition accuracy of billet sequence numbers, a Generative Adversarial Network (GAN) is used. Custom text is used to replace the text in the original billet number image while preserving the style and position of the billet number text and the background of the billet end. The generated image is indistinguishable from the original image in style, which quickly increases the size and quality of the training dataset and prevents the model from overfitting.

[0049] (7) Billet Number Correction: Each steel mill has its own specific rules for assigning billet numbers. To determine the correctness of the model's recognition, the correctness of the billet numbers identified by the Convolutional Recurrent Neural Network (CRNN) can be quickly determined through rule verification. Then, the closest value is quickly retrieved from the current batch of billet numbers obtained from upstream sources to correct the recognition results. The maximum similarity is calculated for billet numbers that are not retrieved to match the most likely billet number.

[0050] The formula for calculating the maximum similarity is:

[0051]

[0052] In the formula, X represents the identified billet number, and Y represents the billet number in the current production batch list.

[0053] Example 2

[0054] A billet number recognition system based on 3D vision and multi-model cascading, such as Figure 3 The system includes a furnace feed roller conveyor 1, a first proximity switch 2 located at the inlet of the furnace feed roller conveyor 1, a second proximity switch 3 located at the rear of the furnace feed roller conveyor 1, a 3D camera 4 located at the upper part of the furnace feed roller conveyor 1, and a system server 5.

[0055] The 3D camera 4 is a 3D area array TOF camera, which has strong anti-light resistance. The depth information it acquires can be used to accurately detect the end position of the steel billet. The 3D camera 4 is used to capture point cloud P1 when the steel billet just enters the furnace roller conveyor 1 and passes the first set of proximity switches 2; when the steel billet enters the furnace roller conveyor 1 and passes the second set of proximity switches 3, the 3D camera 4 captures point cloud P2 and RGB images; simultaneously, the acquired point cloud and RGB images are transmitted to the system server 5.

[0056] The system server 5 includes a position calculation module, an image adjustment module, a sequence extraction module, and a verification and correction module;

[0057] The position calculation module is used to calculate the billet number position using point clouds P1 and P2 acquired by 3D camera 4 and a 3D vision algorithm; and to extract the billet end image from the RGB image based on the billet number position.

[0058] The image adjustment module is used to adjust the state of the obtained billet end image using a classification network, and rotate the billet end image to a 0° state.

[0059] The sequence extraction module is used to detect the end image of the billet using the YOLO target detection algorithm to obtain a complete single-row billet number sequence image;

[0060] The verification and correction module is used to identify a complete single-row billet number sequence image and match the identified billet number with the billet number of the current batch to determine the correctness of the identification.

[0061] The state of the billet end image includes: 0-no inkjet printing, 1-unclear inkjet printing, 2-0°, 3-90°, 4-180°, 5-270°. For categories with a recognition result of 3-5, rotate the image to the 0° state. If the recognition result is no inkjet printing or unclear inkjet printing, manual confirmation and supplementary recording are required.

[0062] The YOLO object detection algorithm is used to detect the billet number position of the input billet end image, resulting in a rectangular billet number image. The rectangular billet number image is then divided into equidistant segments, straightened, and stitched together to obtain a complete single-row billet number sequence image.

[0063] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A billet number recognition method based on 3D vision and multi-model cascading, characterized in that, Includes the following steps: (1) When the billet just enters the furnace roller conveyor, point cloud P1 is collected; after the billet enters the furnace roller conveyor, point cloud P2 and RGB image are collected at the same time. (2) Using the collected point clouds P1 and P2, the position of the billet is calculated using a 3D vision algorithm; Based on the billet's location, extract the billet end image from the RGB image; The 3D vision algorithm includes: A kd-tree structure is built for the collected point cloud P2. The background point cloud P1 is traversed to find the nearest neighbor point d. A distance threshold dmax is set. If the distance d to the nearest neighbor point is less than dmax, it is considered a background point and the point is deleted. If it is greater than dmax, it is considered a billet position point and the point is retained. The final billet position is obtained by using the aspect ratio feature of the fixed bounding rectangle of the billet. (3) The state of the obtained billet end image is adjusted by using a classification network, and the billet end image is rotated to 0°. The status of the billet image includes: 0-no inkjet printing, 1-unclear inkjet printing, 2-0°, 3-90°, 4-180°, 5-270°. If the recognition result is 3-5, rotate it to the 0° state. If the recognition result is no inkjet printing or unclear inkjet printing, manual confirmation and supplementary recording are required. (4) The YOLO target detection algorithm is used to detect the end image of the billet to obtain a complete single-row billet number sequence image; (5) Identify the complete single-row billet number sequence image, and match the identified billet number with the billet number of the current batch to determine the correctness of the identification; The correctness of the billet number obtained by verifying the billet number is verified by the billet number compilation rules. At the same time, the billet numbers in the current batch are searched, and the maximum similarity of the billet numbers not found is calculated to match the most likely billet number.

2. The billet number recognition method based on 3D vision and multi-model cascading as described in claim 1, characterized in that, Point clouds were collected using a 3D camera.

3. The billet number recognition method based on 3D vision and multi-model cascading as described in claim 1, characterized in that, The YOLO object detection algorithm is used to detect the billet number position of the input billet end image, resulting in a rectangular billet number image. The rectangular billet number image is then divided into equidistant segments, straightened, and stitched together to obtain a complete single-row billet number sequence image.

4. The billet number recognition method based on 3D vision and multi-model concatenation as described in claim 1, characterized in that, By utilizing the characteristics of rectangular billet numbers, the rectangular billet number image is divided into equidistant segments along the Y direction based on the number of rows. The image after equidistant segmentation can completely cover the billet number of that row without any character truncation.

5. The billet number recognition method based on 3D vision and multi-model cascading as described in claim 1, characterized in that, A convolutional recurrent neural network is used to identify billet numbers in the input complete single-line billet number sequence image and output the identified billet number results. At the same time, a generative adversarial network is used to replace the text in the original billet number image with custom text while preserving the style and position of the billet number text and the background at the end of the billet, thus preventing the model from overfitting.

6. A billet number recognition system based on 3D vision and multi-model cascading, characterized in that, It includes a furnace feed roller conveyor, a first proximity switch located at the inlet of the furnace feed roller conveyor, a second proximity switch located in the middle and rear of the furnace feed roller conveyor, a 3D camera located on the upper part of the furnace feed roller conveyor, and a system server. The 3D camera is used to capture point cloud P1 when the billet just enters the furnace roller conveyor and passes the first proximity switch; when the billet enters the furnace roller conveyor and passes the second proximity switch, the 3D camera captures point cloud P2 and RGB image; and simultaneously transmits the captured point cloud and RGB image to the system server. The system server includes a location calculation module, an image adjustment module, a sequence extraction module, and a verification and correction module; The position calculation module is used to calculate the position of the billet using point clouds P1 and P2 collected by the 3D camera and a 3D vision algorithm. Based on the billet's location, extract the billet end image from the RGB image; The 3D vision algorithm includes: A kd-tree structure is built for the collected point cloud P2. The background point cloud P1 is traversed to find the nearest neighbor point d. A distance threshold dmax is set. If the distance d to the nearest neighbor point is less than dmax, it is considered a background point and the point is deleted. If it is greater than dmax, it is considered a billet position point and the point is retained. The final billet position is obtained by using the aspect ratio feature of the fixed bounding rectangle of the billet. The image adjustment module is used to adjust the state of the obtained billet end image using a classification network, and rotate the billet end image to 0°. The status of the billet image includes: 0-no inkjet printing, 1-unclear inkjet printing, 2-0°, 3-90°, 4-180°, 5-270°. If the recognition result is 3-5, rotate it to the 0° state. If the recognition result is no inkjet printing or unclear inkjet printing, manual confirmation and supplementary recording are required. The sequence extraction module is used to detect the end image of the billet using the YOLO target detection algorithm to obtain a complete single-row billet number sequence image; The verification and correction module is used to identify a complete single-line billet number sequence image, and match the identified billet number with the billet number of the current batch to determine the correctness of the identification. The correctness of the billet number obtained by verifying the billet number is verified by the billet number compilation rules. At the same time, the billet numbers in the current batch are searched, and the maximum similarity of the billet numbers not found is calculated to match the most likely billet number.

7. The billet number recognition system based on 3D vision and multi-model cascading as described in claim 6, characterized in that, The YOLO object detection algorithm is used to detect the billet number position of the input billet end image, resulting in a rectangular billet number image. The rectangular billet number image is then divided into equidistant segments, straightened, and stitched together to obtain a complete single-row billet number sequence image.

Images