A steel plate character detection method and device based on multi-level network fusion

Abstract

The present invention discloses a steel plate character detection method and device based on multi-level network fusion, the method comprising: collecting images of steel plate characters, and annotating the collected images to construct a steel plate character data set; using a GAN network to perform sample enhancement on the constructed steel plate character data set to expand the steel plate character data set and balance the sample distribution, and obtain a sample data set for model training and testing; constructing a steel plate character recognition algorithm model of multi-level network fusion; based on the sample data set, using a transfer learning method to train the steel plate character recognition algorithm model; using the trained steel plate character recognition algorithm model to realize online detection of steel plate characters. The present invention can meet the real-time and precision requirements of steel plate character detection in the actual production process of steel plates, can effectively improve production efficiency, reduce labor costs, and has great application prospects.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision, deep learning, target detection and metal surface detection, and in particular to a steel plate character detection method and device based on multi-level network fusion. Background Art

[0002] The steel plate character information of traditional steel production lines is issued by the MES system, and the operator manually confirms whether the steel plate character information is correct. The steel plate characters include surface spray and side spray characters; the steel plate number information and the quality of the marking need to be manually confirmed. At present, with the continuous advancement of intelligent manufacturing, the upgrade of production lines to unmanned operation is becoming increasingly urgent. Steel plate character recognition is an important part of the product quality control process. How to use image recognition technology to detect a variety of steel plate surface characters to replace manual recognition is one of the key difficulties in the intelligentization of steel production.

[0003] OCR (Optical Character Recognition) technology converts texts such as various bills, manuscripts, printed products, etc. into image information through optical input methods such as scanning, and then uses text recognition technology to convert image information into usable computer input technology. Using this technology, important data such as labels and texts can be directly extracted from images to generate the required new text, thereby replacing manual entry by humans. Therefore, it is widely used in the field of character recognition, and the characters on the surface of steel plates are mainly composed of Arabic numerals "0, 1, 2, 3, 4, 5, 6, 7, 8, 9" and 26 English letters "A, B, C, D, E, F, G, H, I, J, K, L, M, N, O, P, Q, R, S, T, U, V, W, X, Y, Z", which also belong to the OCR technology neighborhood. However, unlike other characters, the printed steel plate characters for industrial scenes are mainly manifested in three forms of printed information: printed characters on the steel plate surface, steel plate surface steel-printed characters, and side printed characters. If conventional OCR recognition technology is used, the following problems and technical difficulties will be faced: First, due to the large differences in the image features of the three forms of character information, the use of conventional OCR recognition technology requires the design of three specific feature extractors for character recognition, and the algorithm design is complex and has poor generalization; Second, the surface lighting and shadows, mirror reflections, and occlusion of debris on industrial products reduce the imaging quality of steel plate characters; Third, steel plate characters are blurred, missing, and have strokes that stick together, and conventional OCR recognition technology has low recognition accuracy; Fourth, steel plate spray characters are arranged in an arc shape and have uncertain directions, and conventional OCR recognition is prone to flip recognition; Fifth, in the steel plate production process, the spatial distribution structure and contextual semantic rules of steel plate characters have specific rules, and conventional OCR recognition technology ignores the valuable information of spatial distribution and contextual semantics.

[0004] In addition, the characteristics of image recognition by target detection are to identify the category of a target of interest in a digital image and locate the position of the target of interest in the digital image; at the same time, target detection technology can be used as the basic research of visual processing tasks such as instance segmentation and target tracking. At present, with the continuous development of deep learning technology in the field of image recognition, OCR technology based on target detection has more advantages than conventional OCR technology. Conventional OCR technology mainly relies on the method of designing complex artificial feature extractors. Compared with the traditional OCR technology of constructing artificial feature extractors, the target detection algorithm based on deep neural network realizes OCR, and its algorithm structure design is simpler, can automatically extract features, has high detection accuracy and better robustness. Therefore, with the continuous intelligence of steel production lines and the continuous improvement of production rhythm, it is necessary to perform steel plate character recognition based on deep learning and target detection technology in the actual steel production process. However, the use of character recognition technology based on deep learning for steel plate character detection also needs to solve the following two problems: First, in the actual steel plate production workshop environment, a large number of steel plate surface character samples in the actual production environment are difficult to collect. Therefore, insufficient sample quantity can easily lead to an imbalance in the number of samples of different types of defects, which can easily cause overfitting problems during training. Second, for the various forms of steel plate character information, how to use a set of detection algorithms to identify various forms of steel plate character information is the key to improving detection efficiency. If only a single classification network and target detection network are used to identify various types of steel plate spraying information, the actual detection accuracy of the model will be low. Therefore, it is necessary to design the algorithm model in a targeted manner according to the actual production process and the characteristics of the detection object. Summary of the invention

[0005] The present invention provides a steel plate character detection method and device based on multi-level network fusion, so as to solve the technical problems existing in the prior art, such as the imbalance in the number of defect samples of different categories due to insufficient number of samples, easy overfitting during training, and the unsuitability of the prior art for polymorphic steel plate character detection.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] On the one hand, the present invention provides a steel plate character detection method based on multi-level network fusion, comprising:

[0008] Collect images of steel plate characters and annotate the collected images to construct a steel plate character dataset;

[0009] The constructed steel plate character dataset is enhanced by using the GAN network to expand the steel plate character dataset and balance the sample distribution, thus obtaining a sample dataset for model training and testing.

[0010] Construct a steel plate character recognition algorithm model with multi-level network fusion;

[0011] Based on the sample data set, the steel plate character recognition algorithm model is trained using a transfer learning method;

[0012] The trained steel plate character recognition algorithm model is used to realize online detection of steel plate characters.

[0013] Furthermore, the collecting images of steel plate characters and labeling the collected images include:

[0014] Automatically collect images of steel plate characters using an image acquisition system;

[0015] The images of the steel plate characters collected by the image acquisition system are annotated using an annotation tool, and the annotation information is saved in an annotation text file; wherein the annotation information includes: the position of the steel plate character area, the segmentation information of the steel plate character line, and the position information of a single character.

[0016] Furthermore, the steel plate characters collected by the image acquisition system include any one or more combinations of characters spray-printed on the steel plate surface, characters steel-stamped on the steel plate surface, and characters spray-printed on the side of the steel plate.

[0017] Furthermore, the GAN network uses LSTM as a generator and CNN as a discriminator.

[0018] Furthermore, the steel plate character recognition algorithm model includes a first module, a second module and a third module; wherein,

[0019] The first module is a YOLOv3 network, which is used to detect the character area of ​​the steel plate surface image;

[0020] The second module is a DBNet network, which is used to segment the character area of ​​the steel plate surface image detected by the YOLOv3 network to obtain corresponding character rows;

[0021] The third module is an EfficientDet network, which is used to perform single character detection on the character row segmented by the YOLOv3 network to detect each steel plate character in the character row.

[0022] Furthermore, a feature extraction layer is added to each of the three scale feature extraction branches of the YOLOv3 backbone convolutional neural network Darknet53 to increase the receptive field of prediction targets of different scales; wherein the kernel size of the feature extraction layer is 3*3, the strides is 1, and the padding is 1;

[0023] At the same time, the input size of the Darknet53 network is set to 448*448.

[0024] Furthermore, when the DBNet network is used to segment the character area of ​​the steel plate surface image detected by the YOLOv3 network, the steel plate character detection method further includes:

[0025] Expanding the position of the character area of ​​the steel plate surface image detected by the YOLOv3 network;

[0026] The expanded character region is segmented using the DBNet network.

[0027] Furthermore, when using the EfficientDet network to perform single character detection on the character line segmented by the YOLOv3 network, the steel plate character detection method also includes:

[0028] Positioning and arrangement are performed based on the position information of individual steel plate characters obtained by detection.

[0029] Furthermore, based on the sample data set, the steel plate character recognition algorithm model is trained using the transfer learning method, including:

[0030] The weights obtained by training on the ImageNet dataset are used as the initial weights for backbone network training, and pre-training is performed on a public similar character dataset to obtain pre-trained weights;

[0031] The sample data set is divided into a training set, a validation set, and a test set in a ratio of 7:1:2;

[0032] Freeze the shallow pre-trained weights of the backbone convolutional neural network, use the Adam optimizer, and set the learning rate to 1e-3 for network model pre-training;

[0033] Unfreeze the shallow pre-trained weights of the backbone convolutional neural network and reduce the learning rate to 1e-4 for training again.

[0034] On the other hand, the present invention also provides a steel plate character detection device based on multi-level network fusion, the steel plate character detection device based on multi-level network fusion comprises:

[0035] A steel plate character dataset construction module is used to collect images of steel plate characters and annotate the collected images to construct a steel plate character dataset;

[0036] A data enhancement module, used for performing sample enhancement on the steel plate character dataset constructed by the steel plate character dataset construction module using a GAN network, so as to expand the steel plate character dataset and balance the sample distribution, thereby obtaining a sample dataset for model training and testing;

[0037] Model building module, used to build a steel plate character recognition algorithm model with multi-level network fusion;

[0038] A model training module, used to train the steel plate character recognition algorithm model constructed by the model construction module using a transfer learning method based on the sample data set obtained by the data enhancement module;

[0039] The steel plate character online detection module is used to realize the online detection of steel plate characters by using the steel plate character recognition algorithm model trained by the model training module.

[0040] On the other hand, the present invention further provides an electronic device, comprising a processor and a memory; wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the above method.

[0041] In yet another aspect, the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, and the instruction is loaded and executed by a processor to implement the above method.

[0042] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0043] 1. In the actual steel plate production process, the present invention solves the problem of unbalanced collection of steel plate character data sets, and can effectively achieve sample enhancement, thereby improving the generalization ability and detection accuracy of the algorithm network model;

[0044] 2. The present invention designs a detection model composed of three-level network modules, which can ensure both the accuracy of recognition and the robustness of the algorithm, and realizes the effect of effectively recognizing various types of steel plate characters (printed characters on the surface of steel plates, steel-stamped characters on the surface of steel plates, and printed characters on the sides of steel plates);

[0045] 3. In the training process of the algorithm model, the present invention uses the training method of transfer learning to effectively avoid the problem of overfitting in training and improve the network training effect;

[0046] 4. The steel plate character detection method based on multi-level network fusion provided by the present invention is applied to steel plate character detection in the actual production process of steel plates, which can meet the real-time and accuracy requirements of steel plate character detection, and can effectively improve production efficiency and reduce labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1It is a flow chart of a steel plate character detection method based on multi-level network fusion provided by an embodiment of the present invention;

[0049] Figure 2 It is a schematic diagram of the GAN network;

[0050] Figure 3 is a schematic diagram of a primary network module provided by an embodiment of the present invention;

[0051] Figure 4 is a schematic diagram of a secondary network module provided by an embodiment of the present invention;

[0052] Figure 5 is a schematic diagram of a three-level network module provided by an embodiment of the present invention;

[0053] Figure 6 It is a schematic diagram of the transfer learning training network model. DETAILED DESCRIPTION

[0054] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0055] First embodiment

[0056] This embodiment provides a steel plate character detection method based on multi-level network fusion, which can be implemented by an electronic device. The method first uses an image acquisition system to automatically collect images of steel plate characters (steel plate surface spray-printed characters, steel plate surface steel-stamped characters, steel plate side spray-printed characters, hereinafter collectively referred to as steel plate characters, all of which include these three types of characters), and uses a labeling tool to annotate them to form a data set; secondly, in view of the imbalanced distribution of steel plate character images of various categories in the data set, a generative adversarial network (GAN) is used to perform data enhancement to obtain a data set for training and testing; then, in view of the limited feature extraction capability of a single network module and the difficulty in meeting the accuracy requirements of steel plate character detection of multiple categories, a steel plate character detection algorithm model with multi-level network fusion is designed using deep learning technology; finally, the algorithm model training method of transfer learning is used to train the steel plate character detection algorithm model based on multi-level network fusion, and the entire algorithm is deployed in a server for detecting steel plate characters, realizing steel plate character detection with industry-leading recognition accuracy.

[0057] Specifically, the execution process of this method is as follows Figure 1 As shown, the following steps are included:

[0058] S1, collect images of steel plate characters and annotate the collected images to build a steel plate character dataset;

[0059] Specifically, in this embodiment, the implementation process of the above S1 is as follows:

[0060] S11, designing an image acquisition system, and automatically collecting images of steel plate characters by using the image acquisition system behind the spraying machine of the actual steel plate production; wherein the collected steel plate characters include: any one or more combinations of characters sprayed on the steel plate surface, characters steel-stamped on the steel plate surface, and characters sprayed on the side of the steel plate;

[0061] S12, using LabImg and Labme annotation tools to annotate the images of the steel plate characters collected by the image acquisition system, and save the annotation information in an annotation text file; wherein the annotation information includes: the position of the steel plate character area, the segmentation information of the steel plate character line, and the position information of a single character;

[0062] S13, a steel plate character data set is composed of a variety of steel plate character images and corresponding annotation text files.

[0063] S2, using the GAN network to perform sample enhancement on the constructed steel plate character dataset to expand the steel plate character dataset and balance the sample distribution, thereby obtaining a sample dataset for model training and testing;

[0064] It should be noted that in this embodiment, the GAN network used uses LSTM as the generator and CNN as the discriminator; increasing the number of samples can solve the problem of imbalance in the number of samples of different categories of defects caused by insufficient sample quantity, which helps to improve the generalization ability and detection accuracy of the algorithm network model. Figure 2 As shown in the figure, Random noise is random noise, Generator is the generator, Discriminator is the discriminator, Trainset is the dataset image data and the fake Image output by the generator are used as the input of the discriminator, and Real and Fake are the results of the discriminator's judgment. The goal of the generator is to generate images that the discriminator cannot recognize as Real or Fake, even if the generated images are as realistic as possible; GAN generates image data from the perspective of data distribution, and achieves a balance in the confrontation process between the generator and the discriminator, generating similar image data close to the real image data to achieve image sample enhancement.

[0065] S3, building a steel plate character recognition algorithm model with multi-level network fusion;

[0066] It should be noted that the steel plate character recognition algorithm model constructed in this embodiment can locate the steel plate character area, segment the character line, recognize a single character, and compose an ordered string of single character recognition results according to the position information, thereby realizing high-precision, strong robustness, multi-type, multi-scale, and multi-form steel plate character effective quality detection. Specifically, the steel plate character recognition algorithm model design process is as follows:

[0067] S31, by improving the YOLOv3 backbone network Darknet53, construct Figure 3 The primary network module shown detects the character area of ​​the steel plate surface image and makes a quality judgment on the detected area; in order to improve the feature extraction capability of YOLOv3 and improve the receptive field of multi-scale prediction, the present embodiment improves the backbone convolutional neural network Darknet53 of YOLOv3, and the improvement includes: adding a feature extraction layer (the kernel size of the feature extraction layer is 3*3, the strides is 1, and the padding is 1) at the three scale feature extraction branches of the Darknet53 network respectively to increase the receptive field of the prediction targets of different scales y1, y2, and y3 respectively; at the same time, the input size of the network is adjusted from 416*416 to 448*448 to extract more feature information and improve the network's detection accuracy for the steel plate character area;

[0068] S32, based on DBNet semantic segmentation network design Figure 4 The secondary network module shown in the figure segments the surface printing character area and the steel stamp character area detected by the primary network module to obtain a character line image; wherein, the network module uses a feature pyramid to upsample to the same size based on the features extracted by the backbone convolutional neural network, and performs feature cascading to obtain a feature map; the feature map is used to predict a probability map and a threshold map, and an approximate binary map is obtained by fusion calculation of the probability map and the threshold map, and the steel plate character line image is predicted and segmented;

[0069] Furthermore, in order to reduce the error in steel plate character line segmentation caused by the detected steel plate character area being too close together, this embodiment expands the position of the character area detected by S31 before segmenting the character area.

[0070] S33, using the lightweight EfficientDet target detection network as Figure 5 The three-level network module shown in the figure performs single character detection, detects each steel plate character based on the steel plate surface character line segmented by the second-level network module, and positions and arranges the single steel plate character according to the position information of the detected single steel plate character to achieve orderly quality detection of the character string. Specifically, the backbone network of the network module is a weighted bidirectional feature pyramid network, which allows simple and fast multi-scale feature fusion, and scales the resolution, depth and width of all backbone networks, feature networks and prediction networks through a composite scale expansion method, so that the lightweight object detection network can ensure fast and high-precision recognition of single steel plate characters.

[0071] S4, based on the sample data set, uses the transfer learning method to train the steel plate character recognition algorithm model;

[0072] It should be noted that this embodiment adopts Figure 6 The model training method of pre-training fine-tuning transfer learning shown in the figure can improve the ability of the algorithm model to learn target features on the steel plate character dataset by learning similar features on the public similar character dataset, solving the problem of insufficient feature extraction to a certain extent, and improving the generalization ability of the model. It can solve the problem of overfitting that is easy to cause during the training process, further improve the network training effect, and thus realize end-to-end, robust, and high-precision online steel plate character detection.

[0073] Specifically, the training process of the steel plate character recognition algorithm model using the transfer learning method is as follows:

[0074] S41, when using transfer learning to train a multi-level network fusion algorithm model, first use the weights obtained by training on the ImageNet dataset as the initial weights for backbone network training, and perform pre-training on a public similar character dataset to obtain pre-trained weights;

[0075] S42, divide the steel plate character dataset into training set, validation set, and test set in a ratio of 7:1:2. The training of the steel plate character dataset is divided into two steps. The first step is to freeze the shallow pre-trained weights of the backbone convolutional neural network, use the Adam optimizer, set the learning rate to 1e-3, the batchsize to 20, and the number of iterations to 200 epochs; perform network model pre-training. The second step is to unfreeze the shallow pre-trained weights of the backbone convolutional neural network, reduce the learning rate to 1e-4, set the batchsize to 5, and train again for 300 epochs.

[0076] Through training, the network model can recognize and locate various forms of characters on the surface of steel plates.

[0077] S5, using the trained steel plate character recognition algorithm model to achieve online detection of steel plate characters.

[0078] In summary, this embodiment uses deep learning technology to design an algorithm model, and its algorithm can realize end-to-end online steel plate character detection of steel plate characters in various forms (printed characters on the steel plate surface, steel plate surface steel stamped characters, steel plate side sprayed characters, etc.); at the same time, the multi-level network is integrated and designed to realize steel plate character recognition in various forms (printed characters on the steel plate surface, steel plate surface steel stamped characters, steel plate side sprayed characters), which improves the algorithm detection accuracy and robustness; and when establishing a data set, the sample is enhanced by the generative adversarial network (GAN), and when training the network model, the entire multi-level network model algorithm is trained by transfer learning, which solves the problem that the number of samples of different categories of defects is unbalanced due to insufficient number of samples and overfitting is easily caused during the training process. The method of this embodiment is applied to the steel plate character detection of multiple actual steel plate hot rolling production lines. It is found that compared with the existing detection method, the method of this embodiment has good robustness for steel plate character detection of multiple categories and scales, and the comprehensive detection accuracy is greater than 98%, which can effectively improve production efficiency and reduce labor costs. Therefore, it has a large application prospect and is in the leading position in the industry.

[0079] Second embodiment

[0080] This embodiment provides a steel plate character detection device based on multi-level network fusion, including:

[0081] A steel plate character dataset construction module is used to collect images of steel plate characters and annotate the collected images to construct a steel plate character dataset;

[0082] A data enhancement module, used for performing sample enhancement on the steel plate character dataset constructed by the steel plate character dataset construction module using a GAN network, so as to expand the steel plate character dataset and balance the sample distribution, thereby obtaining a sample dataset for model training and testing;

[0083] Model building module, used to build a steel plate character recognition algorithm model with multi-level network fusion;

[0084] A model training module, used to train the steel plate character recognition algorithm model constructed by the model construction module using a transfer learning method based on the sample data set obtained by the data enhancement module;

[0085] The steel plate character online detection module is used to realize the online detection of steel plate characters by using the steel plate character recognition algorithm model trained by the model training module.

[0086] The steel plate character detection device based on multi-level network fusion of this embodiment corresponds to the steel plate character detection method based on multi-level network fusion of the above-mentioned first embodiment; wherein, the functions implemented by each functional module in the steel plate character detection device based on multi-level network fusion of this embodiment correspond one by one to each process step in the steel plate character detection method based on multi-level network fusion of the first embodiment; therefore, they will not be repeated here.

[0087] Third embodiment

[0088] This embodiment provides an electronic device, which includes a processor and a memory; wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the method of the first embodiment.

[0089] The electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) and one or more memories, wherein the memory stores at least one instruction, and the instruction is loaded by the processor to execute the above method.

[0090] Fourth embodiment

[0091] This embodiment provides a computer-readable storage medium, which stores at least one instruction, and the instruction is loaded and executed by a processor to implement the method of the first embodiment. The computer-readable storage medium may be a ROM, a random access memory, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc. The instructions stored therein may be loaded by a processor in a terminal to execute the method.

[0092] In addition, it should be noted that the present invention can be provided as a method, an apparatus or a computer program product. Therefore, the embodiments of the present invention can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the embodiments of the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0093] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0094] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0095] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0096] Finally, it should be noted that the above is a preferred embodiment of the present invention. It should be pointed out that although the preferred embodiment of the present invention has been described, for those skilled in the art, once the basic creative concept of the present invention is known, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be regarded as the protection scope of the present invention. Therefore, the attached claims are intended to be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the embodiments of the present invention.

Claims

1. A steel plate character detection method based on multi-level network fusion, characterized in that: include: Collect images of steel plate characters and annotate the collected images to construct a steel plate character dataset; The constructed steel plate character dataset is enhanced by using the GAN network to expand the steel plate character dataset and balance the sample distribution, thus obtaining a sample dataset for model training and testing. Construct a steel plate character recognition algorithm model with multi-level network fusion; Based on the sample data set, the steel plate character recognition algorithm model is trained using a transfer learning method; Use the trained steel plate character recognition algorithm model to realize online detection of steel plate characters; The steel plate character recognition algorithm model includes a first module, a second module and a third module; wherein, The first module is a YOLOv3 network, which is used to detect the character area of ​​the steel plate surface image; The second module is a DBNet network, which is used to segment the character area of ​​the steel plate surface image detected by the YOLOv3 network to obtain corresponding character rows; The third module is an EfficientDet network, which is used to perform single character detection on the character line segmented by the DBNet network to detect each steel plate character in the character line; A feature extraction layer is added to each of the three scale feature extraction branches of the backbone convolutional neural network Darknet53 of the YOLOv3 network to increase the receptive field of prediction targets of different scales; wherein the kernelsize of the feature extraction layer is 3*3, the strides is 1, and the padding is 1; At the same time, the input size of the Darknet53 network is set to 448*448; When the DBNet network is used to segment the character area of ​​the steel plate surface image detected by the YOLOv3 network, the steel plate character detection method further includes: Expanding the position of the character area of ​​the steel plate surface image detected by the YOLOv3 network; Using the DBNet network to segment the expanded character area; When using the EfficientDet network to perform single character detection on the character line segmented by the DBNet network, the steel plate character detection method further includes: Positioning and arrangement are performed based on the position information of individual steel plate characters obtained by detection.

2. The steel plate character detection method based on multi-level network fusion as claimed in claim 1, characterized in that: The collecting of images of steel plate characters and labeling of the collected images include: Automatically collect images of steel plate characters using an image acquisition system; The images of the steel plate characters collected by the image acquisition system are annotated using an annotation tool, and the annotation information is saved in an annotation text file; wherein the annotation information includes: the position of the steel plate character area, the segmentation information of the steel plate character line, and the position information of a single character.

3. The steel plate character detection method based on multi-level network fusion as claimed in claim 2, characterized in that: The steel plate characters collected by the image acquisition system include any one or more combinations of characters spray-printed on the steel plate surface, characters steel-stamped on the steel plate surface, and characters spray-printed on the side of the steel plate.

4. The steel plate character detection method based on multi-level network fusion as claimed in claim 1, characterized in that: The GAN network uses LSTM as a generator and CNN as a discriminator.

5. The steel plate character detection method based on multi-level network fusion as claimed in claim 1, characterized in that: Based on the sample data set, the steel plate character recognition algorithm model is trained using the transfer learning method, including: The weights obtained by training on the ImageNet dataset are used as the initial weights for backbone network training, and pre-training is performed on a public similar character dataset to obtain pre-trained weights; The sample data set is divided into a training set, a validation set, and a test set in a ratio of 7:1:2; Freeze the shallow pre-trained weights of the backbone convolutional neural network, use the Adam optimizer, and set the learning rate to 1e-3 for network model pre-training; Unfreeze the shallow pre-trained weights of the backbone convolutional neural network and reduce the learning rate to 1e-4 for training again.

6. A steel plate character detection device based on multi-level network fusion, characterized in that: include: A steel plate character dataset construction module is used to collect images of steel plate characters and annotate the collected images to construct a steel plate character dataset; A data enhancement module, used for performing sample enhancement on the steel plate character dataset constructed by the steel plate character dataset construction module using a GAN network, so as to expand the steel plate character dataset and balance the sample distribution, thereby obtaining a sample dataset for model training and testing; Model building module, used to build a steel plate character recognition algorithm model with multi-level network fusion; A model training module, used to train the steel plate character recognition algorithm model constructed by the model construction module using a transfer learning method based on the sample data set obtained by the data enhancement module; A steel plate character online detection module, used for realizing online detection of steel plate characters by using the steel plate character recognition algorithm model trained by the model training module; The steel plate character recognition algorithm model includes a first module, a second module and a third module; wherein, The first module is a YOLOv3 network, which is used to detect the character area of ​​the steel plate surface image; The second module is a DBNet network, which is used to segment the character area of ​​the steel plate surface image detected by the YOLOv3 network to obtain corresponding character rows; The third module is an EfficientDet network, which is used to perform single character detection on the character line segmented by the DBNet network to detect each steel plate character in the character line; A feature extraction layer is added to each of the three scale feature extraction branches of the backbone convolutional neural network Darknet53 of the YOLOv3 network to increase the receptive field of prediction targets of different scales; wherein the kernelsize of the feature extraction layer is 3*3, the strides is 1, and the padding is 1; At the same time, the input size of the Darknet53 network is set to 448*448; When the DBNet network is used to segment the character area of ​​the steel plate surface image detected by the YOLOv3 network, the steel plate character detection method further includes: Expanding the position of the character area of ​​the steel plate surface image detected by the YOLOv3 network; Using the DBNet network to segment the expanded character area; When using the EfficientDet network to perform single character detection on the character line segmented by the DBNet network, the steel plate character detection method further includes: Positioning and arrangement are performed based on the position information of individual steel plate characters obtained by detection.

Images