A slab number high-precision identification method based on a two-stage hierarchical detection model in a complex scene

Abstract

The application discloses a slab number high-precision recognition method based on a two-stage hierarchical detection model in a complex scene and belongs to the field of machine vision industrial detection. The application adopts a two-stage hierarchical double detection model, first detects a character region containing a slab number in an original slab picture by using a slab number character region detection model, then segments the region to form a sub-picture containing only the slab number character region, inputs the sub-picture into a character detection and recognition model, recognizes each character in the slab number, and finally splices the characters according to the character sequence to form a final slab number. The application solves the problem of high-precision recognition of the slab number in a complex scene such as a roller way after continuous casting and cutting, a slab yard, a roller way before a heating furnace, and the like, and compared with an existing recognition method using a single detector, the application can reduce the acquisition requirement of the original picture, and improve the slab number recognition precision of different background pictures obtained by different acquisition devices in different scenes.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of machine vision industrial detection, and relates to a method for recognizing a slab number in multiple scenes such as a roller bed after cutting and marking, a slab yard, and a roller bed before entering a heating furnace. BACKGROUND

[0002] Slab tracking is a key link in logistics between the steelmaking and rolling processes. With the progress of detection technology, the recognition of the slab number has been mostly realized by automatic recognition technology. The recognition method has been transitioned from the traditional machine vision algorithm to a target detection model based on a deep convolutional neural network with higher precision. This method directly finds characters in the slab number in a picture and recognizes them by training a character target detection model. This method requires that the picture used for recognition preferably only contains the slab number character region and does not contain other backgrounds, which has strict requirements for the shooting scene. In actual scenes, the shooting position of the slab number may be multiple, and the picture collection method may also be various, such as fixed type and handheld type. The collected picture may not only contain the slab number character region, but also other objects. In this complex scene, for a single detection model, the slab number character region may be mixed with the similar region in the picture (characters may also appear in the background), causing the recognized slab number to have more or fewer digits. Therefore, the accuracy of the recognition method of the general single detection model is difficult to reach 95% in the complex scene, and manual confirmation of the recognition result is often required in actual use, which reduces the logistics efficiency. SUMMARY

[0003] In view of the above problems, the present application provides a high-precision slab number recognition method based on a two-level hierarchical detection model in a complex scene, which is used to realize high-precision recognition of the slab number in a complex scene, and the recognition accuracy can reach more than 99%.

[0004] The present application provides a high-precision slab number recognition method based on a two-level hierarchical detection model in a complex scene, and the specific steps are as follows:

[0005] Step 1: Collecting a slab picture containing a slab number by using a slab number picture collection module, and uploading and saving the picture in a server.

[0006] Step 2: Labeling the slab number character region in the server picture, and framing the picture region containing the slab number in the picture with a rectangular frame to form a character region sample library.

[0007] Step 3: Train a slab number character region detection model using the training samples obtained in Step 2; the model adopts a two-level structure based on a deep convolutional neural network, the first level uses a deep residual neural network of ResNet34 or ResNet50 for feature extraction of the picture, and the second level uses the fcos algorithm to regress the position coordinates of the detection box based on the features generated by the first level.

[0008] Step 4: Cut out the slab number character region part in the picture labeled with the slab number character region in Step 2 and save it as a slab number character region picture, and label each character region and character value in the slab number character region picture. When labeling, each character is circled by a rectangular frame to form a character detection and recognition sample library.

[0009] Step 5: Train a character detection and recognition model using the training samples obtained in Step 4; the model also adopts a two-level structure based on a deep convolutional neural network, the first level uses a deep residual neural network of ResNet101 or ResNet152 for feature extraction of the picture, and the second level contains two models, the first uses the fcos algorithm to regress the position coordinates of the character detection box based on the features generated by the first level, and the second model trains a fully connected neural network using the features generated by the first level to recognize each character.

[0010] Step 6: Copy the slab number character region detection model file and the character detection and recognition model file to the online server. The online server reads the slab number picture sent by the slab number picture acquisition module, standardizes the picture; then inputs the processed picture into the slab number character region detection model, obtains the rectangular frame coordinates of the slab number character region after processing by the slab number character region detection model, including the coordinates of the four corners of the rectangular frame; finally, the above rectangular frame coordinates and the slab number character region image circled by the rectangular frame are cut out to form a character region sub-picture.

[0011] Step 7: Input the character region sub-picture cut out in Step 6 into the character detection and recognition model, detect the characters contained in the slab number character region image by the character detection and recognition model, obtain the coordinates of each character detection box and the data of the detection box, and determine whether to output the character recognition result according to whether the number of detection boxes is equal to the preset number of characters of the slab number. If the numbers are equal, proceed to Step 9; if the numbers are not equal, it means that the detection is incorrect, proceed to Step 8.

[0012] Step 8: According to the number of output detection boxes being less than or greater than the preset character number, it is judged whether there is omission or overlap. If the number of detection boxes is greater than the preset value, the intersection over union of the detection boxes output by the detection model is reduced to re-output the reserved detection box coordinates, and the judgment process in step 7 is returned. If the number of detection boxes is less than the preset value, the intersection over union of the detection boxes output by the detection model is increased to re-output the reserved detection box coordinates, and the judgment process in step 7 is returned.

[0013] Step 9: The character detection and recognition model is called by the character recognition module, the image in each detection box is subjected to character recognition, and the recognized character is output.

[0014] Step 10: The character splicing module arranges the finally output characters according to the order of the characters appearing in the picture, the character order is arranged from small to large according to the horizontal direction of the upper left corner coordinates of each character, and the final slab number is formed.

[0015] The application also provides a slab number high-precision recognition system based on a two-level hierarchical detection model in a complex scene, which comprises a slab number picture acquisition module, a slab number character region detection model training module, a character detection and recognition model training module, a slab number character region detection module, a character detection module, a character recognition module and a character splicing module.

[0016] The slab number picture acquisition module is composed of a picture acquisition device and a network transmission system, the picture acquisition device is used for photographing the side of the slab with the number sprayed, and uploading the picture to the server. The slab number picture acquisition module is deployed in an industrial site, and the picture is transmitted to an offline server and an online server through an optical fiber network. The offline server is a GPU server, has a slab number character region detection model training module and a character detection and recognition model training module, and is used for training the slab number character region detection model and the character detection and recognition model.

[0017] The slab number character region detection model training module trains a target detection model by using the slab picture, the model only detects and does not recognize, detects the target containing the slab number character region in the picture, and mainly separates the sub-pictures of the character region; and the trained model is used by the slab number character region detection module.

[0018] The character detection and recognition model training module trains a character detection and recognition model by using the slab number character region sub-picture; the model comprises two parts of detection and recognition, and is respectively used for character segmentation and character recognition. The trained model is used by the character detection module and the character recognition module.

[0019] The slab number character region detection module calls the trained slab number character region detection model to detect the collected slab picture and segment out the character region sub-picture containing the slab number, for use by the character detection module.

[0020] The character detection module detects the slab number character region sub-picture using a character detection and recognition model, segments out each character picture, and outputs each character region image for use by the character recognition module.

[0021] The character recognition module recognizes the character picture using a character detection and recognition model and outputs the character of each picture.

[0022] The character splicing module splices each output character into a slab number according to the character order, wherein the character order is determined according to the picture coordinates of each character.

[0023] The slab number high-precision recognition method based on a two-level hierarchical detection model in a complex scene provided by the application can realize high-precision recognition of slab numbers in a complex scene, and compared with the existing recognition method using a single detector, can reduce the acquisition requirements of the original picture and improve the recognition accuracy of slab number pictures of different backgrounds obtained by different acquisition devices in different scenes. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 The structure diagram of the slab number high-precision recognition method based on a two-level hierarchical detection model in a complex scene.

[0025] Figure 2 The flowchart of the slab number high-precision recognition method based on a two-level hierarchical detection model in a complex scene. DETAILED DESCRIPTION

[0026] The application will be further described in detail below with reference to the accompanying drawings.

[0027] The application provides a slab number high-precision recognition method based on a two-level hierarchical detection model in a complex scene, which is realized based on a recognition system composed of a slab number picture acquisition module, a slab number character region detection model training module, a character detection and recognition model training module, a slab number region detection module, a character detection module, a character recognition module, and a character splicing module, as shown in Figure 1 .

[0028] The slab number picture acquisition module is composed of a picture acquisition device and a network transmission system. The picture acquisition device includes an industrial camera, an industrial camera or a handheld camera device, which is used to take pictures of the side of the slab with numbers printed and upload the pictures to the server. The slab number picture acquisition module is deployed in the industrial field (slab yard, heating furnace front roller way), and the pictures are transmitted to the offline server and the online server through the optical fiber network. The offline server is a GPU server, which has a slab number character region detection model training module and a character detection and recognition model training module, which are used to train the slab number character region detection model and the character detection and recognition model, and use pytorch or tensorflow training environment; the model training result is output as a pth model file, and the model file needs to be copied to the online server for the slab number region detection module, the character detection module and the character recognition module deployed in the online server to call.

[0029] The above slab number character region detection model training module trains a target detection model using slab pictures. The model only detects and does not recognize, and detects the target containing the slab number character region in the picture, and the main task is to segment the character region sub-picture. Since the model only detects and does not recognize, ResNet34 or ResNet50 with faster speed is generally used as the feature extraction network. The training process includes two steps of labeling the slab number character region and training the model. The trained model is used for the slab number character region detection module, which belongs to offline application. The slab number character region detection model training module is a program code installed on a server containing GPU. The module does not participate in online operation, and is a model training program in offline environment. The module and the slab number character region detection module are deployed on different servers.

[0030] The character detection and recognition model training module trains a target detection and recognition model using slab number character region sub-pictures. The model includes detection and recognition, which are used for character segmentation and character recognition, respectively. Since the model detects and recognizes, ResNet101 or ResNet152 with faster speed is generally used as the feature extraction network. The training process includes two steps of character labeling and model training. The trained model is used for the character detection module and the character recognition module, which belong to offline application. The character detection and recognition model training module is a program code installed on a server containing GPU, which is generally installed on the same server as the slab number character region detection model training module. It is also a model training program in offline environment and does not participate in online operation.

[0031] The ResNet34, ResNet50, ResNet101 and ResNet152 are residual convolutional neural networks containing 34 layers, 50 layers, 101 layers and 152 layers respectively, which are used for feature extraction of pictures and output feature vectors of pictures at different scales.

[0032] The slab number character region detection module calls the trained slab number character region detection model to detect the collected slab picture and segment out the character region sub-picture containing the slab number, which is used by the character detection module. The module belongs to online application. The slab number character region detection module is a program code and model file installed on an online running server, and the online running server can use GPU or CPU server.

[0033] The character detection module detects the character region sub-picture by using the character detection and recognition model, segments out each character picture, and outputs each character region image for the character recognition module.

[0034] The character recognition module recognizes the character picture by using the character detection and recognition model, and outputs each character picture.

[0035] The character detection module and the character recognition module are a program code and model file installed on an online running server, which are generally installed on the same server as the slab number character region detection module.

[0036] The character splicing module splices each output character into a slab number according to the character order, wherein the character order is determined according to the picture coordinates of each character.

[0037] Based on the above recognition system, the slab number high-precision recognition method based on two-level hierarchical detection model in a complex scene is as shown in the following steps: Figure 2 The specific steps are as follows:

[0038] Step 1: Collect the slab number picture by using the slab number picture collection module; if the industrial camera is used for the slab number picture collection module, the collected video signal needs to be converted into a picture. The collected picture is applied in two stages. In the model training stage, the picture is transmitted to the offline server for storage; in the running stage, the picture is transmitted to the online server for use by each detection module.

[0039] The picture collection should be in different scenes, and different angle pictures of the slab number should be collected. Specifically, the horizontal and vertical angles of the slab number are 90 degrees, and then the left and right rotation and the up and down rotation are used to form the horizontal direction 45 degrees, 75 degrees, 90 degrees, 115 degrees, 135 degrees, and the vertical direction 45 degrees, 75 degrees, 90 degrees, 115 degrees, 135 degrees. For example, after the slab cutting and marking, the industrial camera is installed beside the continuous casting exit roller, the industrial camera is installed beside the roller before entering the heating furnace, and the handheld shooting device is used in the slab yard. When collecting, at least 100 pictures should be collected at each angle.

[0040] Step 2: Label 1000-3000 slab number pictures in different scenes on the offline server, and ensure that the number of pictures in each scene is not less than 100. When labeling, the slab number character area is circled by a rectangular frame, and each slab number character is not labeled at this time.

[0041] Step 3: Randomly select 70% of the pictures labeled with the slab number character area as the training set, 20% as the validation set, and 10% as the test set. Then start training the slab number character area detection model through the slab number character area detection model training module. The training algorithm uses ResNet50+fcos full convolution structure, and the training environment uses pytorch as an example. The training model parameters are set as follows: the gradient update algorithm uses SGD, the learning rate is 0.006, the objective function uses cross-entropy loss function, and the feature fusion method uses FPN method. After training, save the model as a pth format file.

[0042] Step 4: Cut down the slab number character area part from the 1000-3000 pictures labeled with the slab number character area in step 2, save it as a slab number character area picture, and label each character area and character value in the slab number character area picture. When labeling, each character is circled by a rectangular frame. The labeled pictures ensure that the number of pictures in each scene is not less than 100.

[0043] Step 5: Randomly select 70% of the character area pictures labeled with the character area and character value as the training set, 20% as the validation set, and 10% as the test set. The character detection and recognition model training module starts to train the character detection and recognition model. The training algorithm uses ResNeXt101+fcos full convolution structure, and the training environment uses pytorch as an example. The training model parameters are set as follows: the gradient update algorithm uses SGD, the initial value of the dynamic learning rate is 0.009, the decay coefficient is 0.0001, the objective function uses cross-entropy loss function, the feature fusion method uses FPN method, the classification loss function uses FocalLoss, gamma=2, and alpha=0.25. After training, save the model as a pth format file.

[0044] Step 6: Copy the slab number character region detection model file and the character detection and recognition model file to the online server. The online server reads the slab number picture sent by the slab number picture acquisition module, standardizes the picture, that is, subtracts the average value from the pixel value in the picture and divides by the annotation difference, to facilitate neural network processing. Then the processed picture is input into the slab number character region detection model, and the rectangular frame coordinates of the slab number character region are obtained after the slab number character region detection model processing, including the coordinates of the four corners of the rectangular frame. Finally, the above rectangular frame coordinates and the slab number character region image circled by the rectangular frame are cut out.

[0045] Step 7: The character detection module inputs the character region image cut out in step 6 into the character detection and recognition model, detects the characters contained in the slab number character region image through the character detection and recognition model, obtains the coordinates of the detection frame of each character and the data of the detection frame, and determines whether to output the character recognition result according to whether the number of detection frames is equal to the preset number of characters of the slab number; if the numbers are equal, proceed to step 9; if the numbers are not equal, it means that the detection is wrong, and proceed to step 8.

[0046] Step 8: For the case where the number of detection frames is not equal to the preset number of characters of the slab number, it can be judged whether there is omission or overlap according to whether the number of output detection frames is less than or greater than the preset number of characters. If the number of detection frames is greater than the preset value, reduce the intersection over union (IOU) of the detection frame output by the detection model, re-output the retained detection frame coordinates, and return to the judgment process in step 7; if the number of detection frames is less than the preset value, increase the intersection over union (IOU) of the detection frame output by the detection model, re-output the retained detection frame coordinates, and return to the judgment process in step 7.

[0047] Step 9: When the number of detection frames is equal to the preset number of characters of the slab number, the character recognition module calls the character detection and recognition model to perform character recognition on the image in each detection frame, and outputs the recognized characters.

[0048] Step 10: The character concatenation module arranges the finally output characters according to the order of the characters appearing in the picture, and arranges the characters in the order of the left upper corner coordinates of each character from small to large in the horizontal direction, to form the final slab number.

Claims

1. A method for high-precision identification of slab number in complex scenes based on a two-level hierarchical detection model, characterized by the following steps: Step 1: Collecting slab number pictures containing slab numbers using a slab number picture collection module and uploading and saving them in a server; Step 2: Labeling the slab number character area in the server pictures, using a rectangular frame to frame the picture area containing the slab number, and forming a character area sample library; Step 3: Training a slab number character area detection model using the training samples obtained in Step 2, the model adopting a two-level structure based on a deep convolutional neural network, the first level using a deep residual neural network of ResNet34 or ResNet50 for feature extraction of the picture, and the second level using the fcos algorithm to regress the position coordinates of the target detection frame based on the feature picture generated by the first level; Step 4: Cutting down the slab number character area part in the pictures labeled with the slab number character area in Step 2, saving it as a slab number character area picture, and labeling each character area and character value in the slab number character area picture, and labeling each character by using a rectangular frame, forming a character detection and recognition sample library; Step 5: Training a character detection and recognition model using the training samples obtained in Step 4, the model also adopting a two-level structure based on a deep convolutional neural network, the first level using a deep residual neural network of ResNet101 or ResNet152 for feature extraction of the picture, and the second level containing two models, the first one using the fcos algorithm to regress the position coordinates of the character detection frame based on the feature picture generated by the first level, and the second one using a fully connected neural network trained by the feature generated by the first level to recognize each character; Step 6: Copying the slab number character area detection model file and the character detection and recognition model file to an online server; The online server reads the slab number picture sent by the slab number picture collection module, and standardizes the picture; Then inputting the processed picture into the slab number character area detection model, obtaining the rectangular frame coordinates of the slab number character area after the slab number character area detection model processing, including the coordinates of the four corners of the rectangular frame; Finally, cutting out the slab number character area image framed by the above rectangular frame coordinates and the rectangular frame, forming a character area sub-picture; Step 7: Inputting the character area sub-picture cut out in Step 6 into the character detection and recognition model, detecting the characters contained in the character area image by the character detection and recognition model, obtaining the coordinates of the detection frame of each character and the data of the detection frame, and determining whether to output the character recognition result according to whether the number of detection frames is equal to the preset number of characters; If the numbers are equal, proceed to Step 9; If the numbers are not equal, it means that the detection is wrong, and proceed to Step 8; Step 8: Judging whether there is omission or overlap according to whether the number of output detection frames is less than or greater than the preset number of characters; If the number of detection frames is greater than the preset value, reducing the intersection over union of the detection frame output by the detection model to re-output the detection frame coordinates, and returning to the judgment process in Step 7; ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ If the number of detection boxes is less than the preset value, the intersection over union of the detection boxes output by the detection model is increased, the retained detection box coordinates are re-output, and the judgment process in step 7 is returned to execute. Step 9: The character detection and recognition model is called by the character recognition module to perform character recognition on the images in each detection box, and the recognized characters are output. In step 1, the picture acquisition needs to be performed in different scenarios, and pictures of the slab number at different angles need to be captured. Specifically, the slab number is captured at an angle of 90 degrees horizontally and vertically, and then the angle is rotated left and right and up and down to form pictures of the slab number at angles of 45 degrees, 75 degrees, 90 degrees, 115 degrees, and 135 degrees horizontally, and at angles of 45 degrees, 75 degrees, 90 degrees, 115 degrees, and 135 degrees vertically.

2. The method of claim 1, wherein the method comprises: The system includes a slab number picture acquisition module, a slab number character region detection model training module, a character detection and recognition model training module, a slab number character region detection module, a character detection module, a character recognition module, and a character splicing module.

3. The slab number high-precision identification system based on the two-stage hierarchical detection model for complex scenes according to claim 1, characterized in that: The slab number picture acquisition module is composed of a picture acquisition device and a network transmission system. The picture acquisition device is used to take pictures of the slab with the number printed on the side and upload the pictures to the server. The slab number picture acquisition module is deployed in an industrial site and transmits the pictures to an offline server and an online server through an optical fiber network. The offline server is a GPU server and has a slab number character region detection model training module and a character detection and recognition model training module, which are used to train the slab number character region detection model and the character detection and recognition model. The slab number character region detection model training module trains a target detection model using slab pictures. The model only detects and does not recognize, detects the target containing the slab number character region in the picture, and mainly separates the character region sub-picture. The trained model is used by the slab number character region detection module. The character detection and recognition model training module trains a character detection and recognition model using slab number character region sub-pictures. The model includes detection and recognition parts and is used for character segmentation and character recognition, respectively. The trained model is used by the character detection module and the character recognition module. The slab number character region detection module calls the trained slab number character region detection model to detect the collected slab pictures and separate the slab number character region sub-pictures, which are used by the character detection module. The character detection module detects the slab number character region sub-pictures using the character detection and recognition model, separates each character picture, and outputs each character region image, which is used by the character recognition module. The character recognition module recognizes the character pictures using the character detection and recognition model and outputs the characters of each picture. The character splicing module splices each output character into a slab number according to the character order, which is determined according to the picture coordinates of each character. ​ 4. The system according to claim 3, wherein the system comprises: a first detection model for detecting the slab number in the first image; a second detection model for detecting the slab number in the second image; and a third detection model for detecting the slab number in the third image. The character detection and recognition model adopts ResNet101 or ResNet152 as a feature extraction network; The training process includes two steps of character labeling and model training.

5. The system according to claim 3, wherein the system comprises: a first detection model for detecting the slab number in the first image; a second detection model for detecting the slab number in the second image; and a third detection model for detecting the slab number in the third image. The character detection model adopts ResNet34 or ResNet50 as a feature extraction network; The training process includes two steps of labeling the character area of the slab number and training the model.

Images