An engineering drawing text detection and recognition method and system based on improved YOLOv5

Abstract

The application provides an engineering drawing text detection and recognition method and system based on an improved YOLOv5s, which comprises the following steps: using a pre-trained improved YOLOv5s detection model to detect the text of an engineering drawing image to be detected, and obtaining text coordinate information; wherein the improved YOLOv5s detection model uses multiple parallel convolution kernels of different sizes to extract features of different scales; the obtained text coordinate information is used to cut the corresponding text image of the engineering drawing to be detected, the text image is input into a trained PaddleOCR recognition model, and a text recognition result is obtained. Based on the width-height ratio characteristics of the text box in the engineering drawing, the YOLOv5s network is improved, the nonlinear expression capability of the model is increased without increasing the calculation complexity, and the detection accuracy of the model for the engineering drawing is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image detection, and particularly relates to an engineering drawing text detection and recognition method and system based on improved YOLOv5s. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] At present, with the continuous development of AI technology, the general OCR model has achieved very high accuracy in the accuracy of conventional document recognition. However, for the detection and recognition of text in engineering drawings, the general OCR model has poor overall accuracy in text detection and recognition in drawings, and there are problems such as misrecognition and missed recognition. The main reason for this problem is that the size of the drawing image is very large, the size of the text is very small, the text and the symbol frame are close, part of the device symbol is easily misrecognized as text, the text distribution is random, and the text direction is random, etc. Therefore, how to efficiently and accurately detect and recognize the text in the engineering drawing is a technical problem to be solved at present. SUMMARY

[0004] In order to overcome the shortcomings of the prior art, the present application provides an engineering drawing text detection and recognition method and system based on improved YOLOv5s, which improves the YOLOv5s network based on the aspect ratio of the text box in the engineering drawing, increases the non-linear expression ability of the model without increasing the computational complexity, and improves the detection accuracy of the model for engineering drawings.

[0005] To achieve the above purpose, the first aspect of the present application provides an engineering drawing text detection and recognition method based on improved YOLOv5s, comprising:

[0006] The improved YOLOv5s detection model is used to detect the text of the engineering drawing image to be detected, and the text coordinate information is obtained; wherein the YOLOv5s network is improved based on the aspect ratio of the text box in the engineering drawing, and the improved YOLOv5s detection model uses multiple parallel convolution kernels of different sizes to extract features of different scales; the improved YOLOv5s detection model is trained using a text detection data set, and the text detection data set includes a set of engineering drawings rotated in different directions and added with specific noise;

[0007] The obtained text coordinate information is used to crop the corresponding text image of the engineering drawing to be detected, and the text image is input into the trained PaddleOCR recognition model to obtain the text recognition result.

[0008] The second aspect of the present application provides an engineering drawing text detection and recognition system based on improved YOLOv5, comprising:

[0009] The text detection module: using a pre-trained improved YOLOv5s detection model to detect the text of the engineering drawing image to be detected, and obtaining text coordinate information; wherein, based on the width-height ratio of the text box in the engineering drawing, the YOLOv5s network is improved, and the improved YOLOv5s detection model uses multiple parallel convolution kernels of different sizes to extract features of different scales; the improved YOLOv5s detection model is trained using a text detection dataset, and the text detection dataset includes a set of engineering drawings rotated in different directions and added with specific noise;

[0010] The text recognition module: cutting the text image corresponding to the engineering drawing to be detected from the obtained text coordinate information, and inputting the text image into the trained PaddleOCR recognition model to obtain a text recognition result.

[0011] The third aspect of the present application provides a computer device, comprising a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the computer device is running, the processor and the memory communicate through the bus, and the machine readable instructions are executed by the processor to perform an engineering drawing text detection and recognition method based on improved YOLOv5s.

[0012] The fourth aspect of the present application provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform an engineering drawing text detection and recognition method based on improved YOLOv5s.

[0013] The above one or more technical solutions have the following beneficial effects:

[0014] In the present application, based on the characteristics of the width-height ratio of the text box in the engineering drawing, the YOLOv5s network is improved, the backbone network of the YOLOv5s network is changed to a form of multiple parallel convolution kernels of different sizes to learn features of different scales, and then the features of different scales are fused for detection, without increasing the computational complexity, the nonlinear expression ability of the model is increased, and the detection accuracy of the model for the engineering drawing is improved.

[0015] In the application, the text detection data set is constructed by rotating the engineering drawing and adding specific noise such as blank box, device symbol and straight line, etc., the improved YOLOv5s detection model is trained by using the constructed text detection data set, the influence of the text direction in the drawing on the text missed detection is reduced by rotating the engineering drawing in the text detection data set, the text detection accuracy is improved by adding specific noise, and the probability of misrecognizing these symbols as text is reduced.

[0016] Advantages of the additional aspects of the application will be partially given in the following description, partially will become obvious from the following description, or will be understood by the practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0017] The drawings accompanying the specification of this application form a part thereof, serve to provide further understanding of the application, and together with the description of the exemplary embodiments of the application given below, serve to explain the application, and do not constitute an improper limitation of the application.

[0018] Figure 1 A flow chart of the engineering drawing text detection and recognition method based on the improved YOLOv5s in the embodiment one of the application;

[0019] Figure 2 A cutting diagram of the drawing image in the embodiment one of the application;

[0020] Figure 3 A text detection data set augmentation process in the embodiment one of the application;

[0021] Figure 4 The original CBL structure in the existing YOLOv5s backbone network;

[0022] Figure 5 The IBL structure in the improved YOLOv5s backbone network in the embodiment one of the application;

[0023] Figure 6 A straight line removal process of the text box image in the embodiment one of the application. DETAILED DESCRIPTION

[0024] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application belongs.

[0025] It should be noted that the terms used herein are only for the purpose of describing the specific embodiments, and are not intended to limit the exemplary embodiments according to the application.

[0026] In the case of no conflict, the embodiments in the application and the features in the embodiments can be combined with each other.

[0027] Terminology explanation:

[0028] OCR: optical character recognition, optical character recognition.

[0029] opencv library: a popular open-source computer vision library that provides many functions and tools for image processing and computer vision.

[0030] LabelImg tool: a dataset annotation tool.

[0031] Example one

[0032] As shown in Figure 1 The embodiment discloses an engineering drawing text detection and recognition method based on improved YOLOv5, which includes:

[0033] The pre-trained improved YOLOv5s detection model is used to detect the text of the engineering drawing image to be detected, and the text coordinate information is obtained; wherein, based on the width-height ratio of the text box in the engineering drawing, the YOLOv5s network is improved, and the improved YOLOv5s detection model uses multiple parallel convolution kernels of different sizes to extract features of different scales; the improved YOLOv5s detection model is trained using a text detection dataset, which includes a set of engineering drawings rotated in different directions and added with specific noise.

[0034] The obtained text coordinate information is used to crop the corresponding text image of the engineering drawing to be detected, and the text image is input into the trained PaddleOCR recognition model to obtain the text recognition result.

[0035] In this embodiment, a variety of strategies are used to construct the drawing text detection dataset and the text recognition dataset. Text detection: mainly refers to detecting the area of text in the image. Text recognition: mainly refers to recognizing the text in the image into editable text.

[0036] The specific method for constructing the text detection dataset is as follows: after preprocessing the drawing image, a variety of strategies (cutting, rotating, blurring, scaling, background inversion) are used for dataset augmentation. At the same time, in order to further improve the generalization ability and anti-interference ability of the model in the dataset, specific noise (such as blank box, device symbol, straight line, etc.) is added in the blank part of part of the drawing image, and a background image with a proportion of 0.5% of the total number of the dataset is added in the dataset. In addition, in order to improve the detection accuracy of overlapping text, the proportion of the overlapping text dataset is increased.

[0037] Text detection dataset construction example: a large size drawing image, first divide the image size as a fixed cutting position, project in the vicinity of the fixed cutting position by 100 pixels, find the position with the most pixels as the final cutting position, to avoid dividing the complete symbol frame into two parts.

[0038] As Figure 2 shown, after cutting into 6 paper cutouts, the main purpose of cutting is to reduce the labeling workload, improve the detection effect of small size text, and increase the number of datasets. After cutting, the paper data is labeled using the LabelImg tool and saved in YOLO format. The above cutouts are rotated clockwise by 90°, 180°, and 270°, respectively. The main purpose of rotation is to reduce the influence of text direction on text detection. The dataset has a total of 24 paper cutouts. Gaussian blur is used on the above paper cutouts, and the dataset has a total of 48 paper cutouts. The above paper cutouts are respectively reduced by 0.8 times in width and height, and the edges around them are added to the same size as the original size to avoid yolo automatically scaling the image size back to the original size during training. The dataset has a total of 96 paper cutouts, and 9 of them are randomly selected and added with specific noise (such as blank boxes, device symbols, straight lines, etc.) in the blank area. The above paper cutouts are inverted to improve the text detection effect of the paper in the black background, and the dataset has a total of 192 paper cutouts. Finally, 1 cutout is randomly selected, and the paper text information is removed, and the background image is inverted. These two images are added to the dataset as background images to improve the accuracy of the detection model and reduce the error rate of detecting symbols as text. The dataset has a total of 194 paper cutouts. Using the above dataset augmentation method, 1 large size paper can be expanded to 194 cutouts for training set. The above dataset augmentation process is shown in Figure 3 .

[0039] This embodiment uses Gaussian blur to blur the image for augmentation to obtain a more realistic image blur. Adding specific noise to the blank area improves the accuracy of text detection and reduces the probability of misidentifying these symbols as text.

[0040] The specific method for constructing the text recognition dataset is as follows: In order to increase the recognition ability of the text recognition model on special characters, a paper text recognition dataset is constructed. The dataset uses a publicly available printed text dataset, and also includes special character data contained in the paper. When constructing special character data, a section of text containing special characters is randomly selected and fused into a background image (the background image mainly has white and black backgrounds) using the opencv library to construct a dataset containing special characters.

[0041] In this embodiment, the improved YOLOv5s model is used for text detection model training. Finally, the text detection model after training is used for text detection in the complete drawing image, and the position coordinates of all texts in the drawing image are obtained.

[0042] The width-height ratio of the text box in the engineering drawing is generally between 3:1-5:1, so the backbone layer of YOLOv5s is improved according to this feature to enhance the nonlinear expression ability of the model and improve the detection performance of the model. Specifically, the Inception structure is applied to the backbone layer of YOLOv5s, and 3x1 convolution kernel, 4x1 convolution kernel and 5x1 convolution kernel are connected in parallel to the convolution layer of YOLOv5s to form a connection mode similar to the residual network, and the edge padding method is used to ensure the consistency of the size of the convolution layer after convolution. As shown in 4 and Figure 5 , specifically, the CBL structure in the YOLOv5s backbone network is replaced by the innovative IBL structure, which uses the Inception structure to learn features of different scales, and then fuses these features to pass them to the subsequent convolution layer for processing. In this way, the nonlinear expression ability of the model can be improved without increasing the computational complexity, and the detection accuracy of the model can be improved.

[0043] Specifically, the IBL structure includes a parallel convolution module, a normalization layer and an activation function. The parallel convolution module includes four parallel branches for learning different scale features. Specifically, the first branch includes a convolution layer with a convolution kernel size of 1x1 and a convolution layer with a convolution kernel size of 1x3 connected in sequence, the second branch includes a convolution layer with a convolution kernel size of 1x1 and a convolution layer with a convolution kernel size of 1x4 connected in sequence, the third branch includes a convolution layer with a convolution kernel size of 1x1 and a convolution layer with a convolution kernel size of 1x5 connected in sequence, and the fourth branch includes a convolution layer with a convolution kernel size of 3x3.

[0044] In order to reduce the mis-detection of some symbols in the drawing as text, in the data set annotation, in addition to classifying the text box, the symbols that are often mis-detected as text are classified as a new category to reduce the text mis-detection rate. That is, the text is classified as class 0, and the symbols that are easily mis-detected as text are classified as other classes.

[0045] After setting the hyperparameters, the improved YOLOv5s model is used to train a drawing-specific text detection model. Using the text detection model to detect the input complete drawing will obtain the text position coordinates {[x1, y1, w1, h1],... [x n , y n , w n , h n ]}.

[0046] Since PaddleOCR is a general recognition model, it cannot recognize special characters in the drawing (such as ±, ℃, α, etc.), so it is necessary to fine-tune the PaddleOCR recognition model to increase the recognition ability of special characters. In the process of preparing the text recognition dataset, in addition to using the public printed text dataset, special characters in the drawing are also added as the dataset. After setting the hyperparameters, the fine-tuned PaddleOCR recognition model is obtained by training the text recognition dataset.

[0047] For the above detected text position coordinates {[x1, y1, w1, h1],... [x n ,y n ,w n ,h n ]}, the corresponding text image {I1...I n} is cut from the drawing image, and the text cut image is grayed and binarized, and the upper and lower edges of the image are intercepted. Then, the one-dimensional array is obtained by horizontally projecting the edge image. It is judged whether there is a value equal to the width of the text cut image in the array. If there is, it is considered that there is a straight line in the edge, and the edge is removed. Otherwise, the image edge is retained. The processing flow is shown in Figure 6 Similarly, the left and right edges of the image can also be removed by the above method. The purpose of removing the edge straight line is because part of the text is written into the symbol box and the text is close to the symbol box line. In order to avoid the influence of the straight line on the recognition, the straight line in the image edge needs to be removed. In this embodiment, 5 pixels high (empirical value) are cut according to the actual situation, so the actual operation is to directly remove the edge image containing the straight line.

[0048] Embodiment Two

[0049] The purpose of this embodiment is to provide an engineering drawing text detection and recognition system based on improved YOLOv5s, which comprises:

[0050] The text detection module uses a pre-trained improved YOLOv5s detection model to detect the text of the engineering drawing image to be detected and obtains text coordinate information; wherein, based on the width-height ratio of the text box in the engineering drawing, the YOLOv5s network is improved, and the improved YOLOv5s detection model uses multiple parallel convolution kernels of different sizes to extract features of different scales; the improved YOLOv5s detection model is trained using a text detection dataset, and the text detection dataset includes a set of engineering drawings rotated in different directions and added with specific noise;

[0051] The text recognition module: the obtained text coordinate information cuts the text image corresponding to the engineering drawing to be detected, and inputs the text image into the trained PaddleOCR recognition model to obtain a text recognition result.

[0052] Embodiment three

[0053] The purpose of this embodiment is to provide a computing device, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the above method when executing the program.

[0054] Embodiment four

[0055] The purpose of this embodiment is to provide a computer-readable storage medium.

[0056] A computer-readable storage medium having stored thereon a computer program, the program being executable by a processor to perform the steps of the above method.

[0057] The steps and methods involved in the above embodiments two, three and four correspond to embodiment one, and the specific embodiments can refer to the relevant description part of embodiment one. The term "computer-readable storage medium" should be understood to include a single medium or multiple media of one or more instruction sets; it should also be understood to include any medium capable of storing, encoding or carrying instruction sets for execution by a processor and causing the processor to perform any of the methods in the present application.

[0058] Those skilled in the art should understand that the above-mentioned modules or steps of the present application can be realized by a general computer device, alternatively, they can be realized by program codes executable by a computing device, so that they can be stored in a storage device for execution by a computing device, or they can be made into individual integrated circuit modules, or a plurality of modules or steps among them can be made into a single integrated circuit module to realize. The present application is not limited to any specific combination of hardware and software.

[0059] Although the specific embodiments of the present application are described above in combination with the accompanying drawings, it is not a limitation on the scope of protection of the present application, and those skilled in the art should understand that various modifications or variations made by those skilled in the art on the basis of the technical solutions of the present application without creative labor are still within the scope of protection of the present application.

Claims

1. An improved YOLOv5s-based engineering drawing text detection and recognition method, characterized in that, The application comprises the following steps: The improved YOLOv5s detection model is trained by using a text detection data set, and the text detection data set comprises engineering drawing sets rotated in different directions and added with specific noise. The specific process is as follows: the CBL structure in the YOLOv5s backbone network is replaced by an IBL structure, and the IBL structure comprises a parallel convolution module, a normalization layer and an activation function.

2. The engineering drawing text detection and recognition method based on improved YOLOv5 of claim 1, wherein, The parallel convolution module comprises four parallel branches for learning different scale features. Specifically, the first branch comprises a 1x1 convolution layer and a 1x3 convolution layer, the second branch comprises a 1x1 convolution layer and a 1x4 convolution layer, the third branch comprises a 1x1 convolution layer and a 1x5 convolution layer, and the fourth branch comprises a 3x3 convolution layer. The obtained text coordinate information is used to crop the corresponding text image of the engineering drawing to be detected, and the text image is input into the trained PaddleOCR recognition model to obtain a text recognition result. The specific process is as follows: the engineering drawing is cut into multiple parts according to the size, the cut engineering drawing images are labeled and rotated in different directions to obtain drawing cut images.

3. The improved YOLOv5s-based engineering drawing text detection and recognition method of claim 1, wherein, The width and height of the drawing cut images are reduced in proportion, and the edges consistent with the background pixels are added around the drawing cut images, and then the drawing cut images are expanded to the size before the reduction.

4. The improved YOLOv5s-based engineering drawing text detection and recognition method of claim 1, wherein, Specific noise is added to the blank part of the expanded drawing image, and then the background is inverted to obtain a text recognition data set.

5. The improved YOLOv5s-based engineering drawing text detection and recognition method of claim 1, wherein, The specific noise is a blank box, a device symbol and a straight line.

6. An engineering drawing text detection and recognition system based on improved YOLOv5, characterized in that, The PaddleOCR recognition model is trained by using a text recognition data set, and the specific process is as follows: special character data is added to the drawing, a section of text containing special characters is selected, and the opencv library is used to fuse the text into a background image to obtain a text recognition data set. Before the text image is input into the trained PaddleOCR recognition model, the following steps are further included: the text image is subjected to grayscale and binarization processing, the upper and lower edges of the text image are cut, the edge image is subjected to horizontal projection to obtain a one-dimensional array, it is judged whether there is a value with a length equal to the width of the text image in the one-dimensional array, if there is, a straight line exists in the edge image, the edge image is removed, otherwise, the edge image is retained. The application comprises the following steps: The character detection module: utilize the improved YOLOv5s detection model pre-trained to detect the engineering drawing image to be detected to obtain character coordinate information; wherein the improved YOLOv5s detection model is that the backbone network adopts multiple parallel convolution kernels of different sizes to extract features of different scales; the improved YOLOv5s detection model is trained by using a character detection data set, wherein the character detection data set includes a set of engineering drawings rotated in different directions and adding specific noise; the improved YOLOv5s detection model is specifically: the CBL structure in the YOLOv5s backbone network is replaced by an IBL structure, and the IBL structure includes a parallel convolution module, a normalization layer and an activation function; the parallel convolution module includes four parallel branches for learning different scale features; specifically, the first branch includes a 1x1 convolution layer and a 1x3 convolution layer, the second branch includes a 1x1 convolution layer and a 1x4 convolution layer, the third branch includes a 1x1 convolution layer and a 1x5 convolution layer, and the fourth branch includes a 3x3 convolution layer; The character recognition module: the obtained character coordinate information is used to cut the corresponding character image of the engineering drawing to be detected, and the character image is input into the trained PaddleOCR recognition model to obtain a character recognition result.

7. A computer device, comprising: It comprises: A processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the computer device is running, the processor and the memory communicate through the bus, the machine readable instructions are executed by the processor to execute the method of claim 1 to 5 based on the improved YOLOv5s engineering drawing character detection and recognition.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to execute the method of claim 1 to 5 based on the improved YOLOv5s engineering drawing character detection and recognition.

Images