Converter image incremental automatic machine learning system and its establishment and training method

By using deep learning image incremental automatic machine learning system during converter steelmaking, real-time identification and early warning of converter abnormal scenes, the problem of abnormal phenomenon recognition in converter production is solved, and an efficient and stable production environment is achieved.

CN115035328BActive Publication Date: 2024-11-15SHANGHAI UNIV
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202210437673.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-25
Publication Date
2024-11-15
Estimated Expiration
2042-04-25

AI Technical Summary

Technical Problem

During the converter steelmaking process, there are abnormal phenomena such as splashing, oxygen gun ignition failure, and oxygen gun water leakage, which leads to unstable equipment and increases steel consumption and production costs. It is difficult for the existing technology to achieve real-time identification and early warning.

Method used

Design a converter image incremental automatic machine learning system based on deep learning, adopting deep residual network as the backbone network, and through the combination of data layer, service layer and user layer, it realizes end-to-end solution from image acquisition to recognition and early warning The solution is to use the camera to monitor the converter scene, perform data preprocessing and feature extraction, build an abnormal warning model, identify and issue alarms in real time.

Benefits of technology

Accurate identification and early warning of converter abnormal scenes is achieved, the recognition accuracy rate exceeds 95%, the system communication function is stable, the model can be self-learned and updated, and structured data can be stored for a long time, which significantly improves the stability and efficiency of converter production.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115035328B_ABST
    Figure CN115035328B_ABST
Patent Text Reader

Abstract

The present invention relates to an incremental automatic machine learning system for converter images and a method for establishing and training the system, forming an end-to-end solution for converter images from perception to recognition and analysis, in order to cope with abnormal phenomena such as splashing, oxygen lance ignition failure, oxygen lance leakage, and post-pull blowing that occur in steel mills during converter steelmaking. The present invention can be deployed in edge devices such as factories, overcoming the difficulties of insufficient equipment computing power and real-time detection in industrial machine vision tasks. Compared with the existing image recognition methods for smart factories at this stage, the incremental automatic machine learning system for converter images proposed in the present invention has obvious advantages in practical applications.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to an image processing and application technology, and in particular to a converter image incremental automatic machine learning system and an establishment and training method thereof. Background Art

[0002] With the maturity and application of the field of artificial intelligence, the strategic position of the manufacturing industry has received more and more attention. In recent years, many countries have formulated and launched their own manufacturing development strategies, and smart factories have become the main direction of industrial development. At present, steel mills are using converter steelmaking process, there are abnormal phenomena such as splashing, oxygen lance ignition failure, oxygen lance leakage, and post-pull blowing, which are not conducive to the efficient and stable production of converters. For example, splashing is inevitable in the process of converter steelmaking. Splashing not only interferes with the normal operation of converter steelmaking, but also contains a large amount of metal in its carryover, thereby reducing the amount of steel produced and increasing the consumption of steel. In order to detect abnormal conditions in converter steelmaking as soon as possible and take preventive measures in time, it is urgent to develop a converter scene intelligent recognition and analysis system to realize intelligent analysis of converter abnormal scenes.

[0003] Today, deep learning and convolutional neural networks are widely used in the field of computer vision. In 2017, the concept of lightweight networks was proposed, which takes into account the speed and accuracy of network training so that they can be deployed on edge terminal devices. Lightweight networks are widely used in industrial factories, unmanned driving, IoT edge computing and other scenarios. The method based on deep learning and convolutional neural network representation learning can automatically extract features and self-learn, and has made great breakthroughs in image classification. Summary of the invention

[0004] In view of the contradiction between the real-time detection requirements and the insufficient computing power of equipment in industrial machine vision tasks, an incremental automatic machine learning system for converter images and its establishment and training method are proposed. Based on deep learning, converter anomaly recognition is formed to form an end-to-end solution for converter images from perception to recognition and judgment. In the design of image recognition algorithm, it is proposed to train based on deep residual network as the backbone network, which effectively solves the problem of vanishing gradient of neural network and reduces or compensates for a lot of information loss.

[0005] The technical solution of the present invention is: a converter image incremental automatic machine learning system, which includes a data layer, a business layer and a user layer from bottom to top;

[0006] Data layer: used to process data, collect real-time converter video data, pre-process image data, and image recognition label data driven by the business layer to form an unstructured database system and classify and store typical converter abnormality scene data;

[0007] Business layer: It is used as the core driver for user layer image increment and customer needs, and is also the data driver for the lower data layer; it includes user module and abnormal warning module. The abnormal warning module is responsible for model construction and prediction. The system reads abnormal data of the data layer by calling the abnormal warning module, and returns the processed results to the user module, including the parameters and abnormal conditions of each model; the user module returns the results required by the user to the user layer;

[0008] User layer: used to provide users with an interface, online incremental images, required task input, and result display.

[0009] Preferably, the camera accurately monitors the color and shape of the converter scene. For four typical scenes, namely, splashing, oxygen lance ignition failure, oxygen lance leakage, and post-pull blowing, the camera outputs the converter video data and sends it to the data layer for conversion into converter data set storage and high-quality image data output. The high-quality image data is sent to the business layer for image feature extraction and recognition, and the classified abnormal images are output to the user layer.

[0010] Preferably, the abnormal warning module of the business layer uses an image feature extraction algorithm to extract key features of the converter image, converts the unstructured image data into structured feature vectors in the feature space, fuses the features through vector splicing, and then converts the unstructured converter image data into typical converter scene recognition through the converter scene image feature extraction algorithm. Finally, the abnormal warning module identifies the abnormal converter scene and issues an alarm message, and stores the abnormal warning model parameter information and abnormal result data in the database.

[0011] A method for establishing a data layer of a converter image incremental automatic machine learning system, which uses video summarization technology to filter, preprocess and store the collected converter video data to form a complete converter scene training data set, a verification data set and a test data set for the business layer to classify and identify four types of typical scenes; the converter video data is frame-processed, and the image data is preprocessed using digital image processing technology, and the preprocessed high-quality image data is imported into an unstructured database system to form a converter image database; the valid data of the training data set, the verification data set and the test data set are annotated by domain experts and used for the training, verification and testing of the business layer model respectively.

[0012] Furthermore, the image data is preprocessed using an image normalization configuration to normalize the input image, and the stored and read images are in BGR channel order by default; the preprocessing data flow is as follows: read the image, randomly scale and cut out the image, randomly flip the image horizontally with a probability of 0.4 and then normalize it, and finally convert the image format into a tensor; the four typical scene labels in the configuration file are also converted into tensors by one-hot encoding.

[0013] A business layer establishment method for a converter image incremental automatic machine learning system, which establishes a complete model tool chain including model training, model verification, and model deployment;

[0014] The backbone network depth of the model training is 18 layers, and the loss function is set to cross entropy loss. First, the image and label data preprocessed by the data layer are read. The training optimization strategy: the optimizer selects stochastic gradient descent, the optimizer's learning rate is initially set, and the momentum, weight decay coefficient and training rounds are set.

[0015] A converter image incremental automatic machine learning system training method, including system model process construction, automatic machine learning model network determination method and algorithm model evaluation index establishment;

[0016] The model process of the system is constructed as follows:

[0017] 1) To create a model on the system, you need to determine the model name and select the functions to be implemented by the model;

[0018] 2) Upload data. After the user uploads the data, the data is labeled according to the data requirements of different model types. If there is locally labeled data, it can be uploaded directly.

[0019] 3) Train the model and verify the effect, select the algorithm, configure the training data and other task-related parameters to start the training task;

[0020] 4) After the model training is completed, it supports visual viewing of the model evaluation report, and identifies four abnormal conditions of the converter image through the model;

[0021] 5) Finally, publish the model and deploy the satisfactory model on the server or local device;

[0022] The automatic machine learning model network determination method: for selecting a backbone network, the parameter training and model structure search are decoupled through the neural network architecture search and network parameter determination method in the automatic machine learning, the backbone network model is first determined, and then the parameters are trained based on the determined network structure, so that the searched neural network is highly consistent with the task scenario of actual converter image recognition;

[0023] Model evaluation metrics for the algorithm:

[0024] Accuracy: a measurement indicator of the image classification model, the ratio of the number of correctly classified samples to the total number of samples. The closer the value is to 1, the better the model effect.

[0025] F1-score, for a certain category, is the harmonic mean of precision and recall. For image recognition classification models, the higher the index, the better the effect;

[0026] Precision, for a certain category, is the ratio of the number of samples correctly predicted as that category to the total number of samples predicted as that category;

[0027] Recall rate, for a certain category, is the ratio of the number of samples correctly predicted as that category to the total number of samples in that category; MAP is an indicator for measuring the effectiveness of the object detection algorithm. For object detection tasks, the precision and recall rate of each type of object can be calculated. After multiple calculations / experiments at different thresholds, a PR curve can be obtained for each class, and the area under the curve is the average.

[0028] The beneficial effects of the present invention are: the converter image incremental automatic machine learning system and its establishment and training method of the present invention have stable system communication functions, and the structured data formed after recognition can be stored for more than 6 months. The model has a self-learning function and can realize the function of automatically updating the calculation model. The model can accurately identify the abnormal situation of converter splashing and issue an alarm message, and the recognition accuracy rate exceeds 95%. Compared with the existing intelligent manufacturing methods at this stage, the present invention has obvious advantages in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a schematic diagram of the structure of the converter image incremental automatic machine learning system for smart factories of the present invention;

[0030] Figure 2 A schematic diagram of the interactive learning process between the data layer and the business layer of the present invention;

[0031] Figure 3 This is a flowchart of the incremental automatic machine learning anomaly recognition of converter images for smart factories according to the present invention. DETAILED DESCRIPTION

[0032] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0033] like Figure 1 The figure shows the schematic diagram of the structure of the incremental automatic machine learning system for converter images for smart factories. The system includes the data layer, business layer and user layer from bottom to top.

[0034] Data layer: The data layer mainly processes data, such as converter video data, pre-processed high-quality image data, and label data, so that the data can be input into the built-in algorithm for training later;

[0035] Business layer: The business layer is the core driver of the image incremental automatic machine learning system of the upper user layer, and is also the data driver of the lower data layer; it includes the user module and the abnormal warning module. The abnormal warning module is the core module of the business layer, responsible for the construction and prediction of the model. The system reads the abnormal data of the data layer by calling the abnormal warning module, and returns the processed results to the user module, including the parameters and abnormal conditions of each model; the user module returns the results required by the user to the user layer;

[0036] User layer: The main purpose of the user layer is to provide an interface for the staff of the steel plant, online incremental images, demand task input and result display. The user layer directly faces the user, inputs real-time monitoring data, and outputs abnormal situations and abnormal video playback.

[0037] This system aims to form an end-to-end solution for converter images from perception to recognition and analysis. When an abnormal scene appears in the monitoring video, the end-to-end solution will immediately output structured warning data, and the system will automatically generate records or alarm data.

[0038] like Figure 2 The diagram shows the interactive learning process between the data layer and the business layer. The core of the bottom data layer is data processing and storage. First, the research on converter image data collection and recognition is carried out. ImageNet (imaging network) is constructed, and industrial cameras are installed to collect converter images. The specific installation method of the camera is as follows: the camera is placed at the furnace mouth, furnace body and furnace bottom of the converter respectively; the camera at the furnace mouth monitors the abnormal conditions of oxygen lance leakage and oxygen lance ignition failure; the camera at the furnace body monitors the abnormal conditions of pull-back blowing; the camera at the furnace bottom is aimed at the ground directly below the converter to monitor the abnormal conditions of converter splashing. The installed camera can accurately monitor the color and shape of the converter scene, and identify and judge the four typical scenes of splashing, oxygen lance ignition failure, oxygen lance leakage, and pull-back blowing. Secondly, the converter image preprocessing (datapreparing) step is carried out. The collected converter video data is filtered, preprocessed and stored through video summarization technology to form a complete converter scene training data set (training set), verification data set (validation set) and test data set (testing set) for classification and recognition of four types of typical scenes. Among them, the converter video is framed, and the framed converter images are processed using digital image processing technology to form a high-quality image data set, which is then imported into an unstructured database system to classify and store the four types of typical scene data.

[0039] The data layer also includes the image annotation stage. In the early stage of system operation, the model does not have strong image recognition capabilities and domain data. It is necessary for domain experts to annotate the valid data of the training data set, verification data set, and test data set for subsequent model training, verification, and testing. As the number of data sets increases, the image recognition ability of the model will also increase. The type format of images stored in the data layer is ImageNet format. The preprocessing of image data uses image normalization configuration to normalize the input image. The stored and read images default to BGR channel order. Preprocessing data flow: read the image, randomly scale and cut out the image, randomly flip the image horizontally with a probability of 0.4 and then normalize it, and finally convert the image format into a tensor. The labels in the configuration file (four types of converter abnormal scenarios) are also converted into tensors through one-hot encoding.

[0040] The business layer is the core driver of the image incremental automatic machine learning system of the upper user layer, and is also the data driver of the lower data layer. It is the core layer of the entire converter image incremental automatic machine learning system. The abnormal warning module is the core module of the business layer, which issues timely warnings for abnormal situations such as splashing and oxygen lance leakage that may occur during the converter steelmaking process. The abnormal warning module includes abnormal warning, real-time monitoring and video playback. The abnormal warning module includes model training, model verification, model deployment and other processes. Only through a complete model tool chain can relatively satisfactory prediction results be obtained.

[0041] The system uses Python as the programming language, and an NVIDIA GeForceRTX2080Ti graphics card is used in the model training phase. The backbone network of the pre-trained model has a depth of 18 layers, and the loss function is set to cross entropy loss. First, read the image and label data preprocessed by the data layer to determine which keys in the data should be passed to the detector. Training optimization strategy: The optimizer selects stochastic gradient descent (SGD), and the learning rate of the optimizer is initially set to 0.01, the momentum is 0.9, and the weight decay coefficient is 0.0001. A total of 20 epochs are trained, that is, the data set is scanned 20 times in total, and the learning rate is decayed at epochs 5, 12, and 15 respectively.

[0042] like Figure 3As shown in the figure, an image feature extraction algorithm is built into the abnormal warning module. The image feature extraction algorithm is used to extract the key features of the converter image, thereby converting the unstructured image data into a structured feature vector in the feature space. The features are fused by vector splicing, so that the model can obtain more comprehensive data feature information. Then, through the research and development of efficient, accurate and robust converter scene image feature extraction algorithms, the unstructured converter image data is converted into features with stronger expressive power, and then the typical converter scene recognition and judgment are carried out. Ultimately, the module is required to accurately identify the converter abnormal scene and issue an alarm message, and store the model parameter information and abnormal result data in the database.

[0043] The principles of image feature extraction are as follows: 1) pre-train the network on ImageNet; 2) fine-tune the network on the converter image dataset; 3) use the neural network architecture search (NAS) method to search on the trained network, draw on the idea of ​​the DARTS algorithm, reasonably search the topological space, and achieve model compression; 4) based on the idea of ​​incremental learning, save the currently learned model parameters, penalty coefficients and sample playback pool.

[0044] User layer: When an abnormal typical scene appears in the surveillance video, the end-to-end solution immediately outputs structured warning data, and the system automatically generates records or alarm data. Based on the results of the algorithm output, a structured abnormal event record is generated and stored in the form of logs, databases, etc., which is provided for subsequent data analysis and process optimization.

[0045] The model building process of the user operation system is as follows: 1) To create a model on the system, you need to determine the model name and select the functions implemented by the model. 2) Upload data. After the user uploads the data, the data can be labeled according to the data requirements of different model types. If there is local labeled data, it can also be uploaded directly. 3) Train the model and verify the effect, select the algorithm, configure the training data and other task-related parameters to complete the start of the training task. 4) After the model training is completed, it supports visual viewing of the model evaluation report, and uses the model to identify four abnormal conditions of the converter image. 5) Finally, publish the model and deploy the satisfactory model on the server or local device.

[0046] The system establishment and training method mainly includes the system model process construction, the automatic machine learning model network determination method and the model evaluation index establishment of the algorithm;

[0047] The model process construction of the system: First, create a model on the system. You need to determine the model name and select the functions that the model implements. The second step is to upload the data. After the user uploads the data, the data can be labeled according to the data requirements of different model types. If there is locally labeled data, you can also upload it directly. The third step is to train the model and verify the effect. Select the algorithm, configure the training data and other task-related parameters to complete the training task. After the model training is completed, it supports visual viewing of the model evaluation report, and uploads data online through the model verification function to test the model effect. Finally, publish the model and deploy the satisfactory model on the server or local device;

[0048] Method for determining the network of the automatic machine learning model: Most image recognition methods are directly completed using the backbone network. Since this system is an incremental automatic learning system for converter images for smart factories, for the problem of object target detection, each picture needs to pay attention not only to the category of the target, but also to the specific location of the target, such as the coordinates and length and width of the object in the picture. Therefore, directly selecting the backbone network will lead to a decrease in model performance. Through the method of neural network architecture search and network parameter determination in automatic machine learning, parameter training and model structure search are decoupled, the backbone network model is determined first, and then the parameters are trained based on the determined network structure, so that the searched neural network is highly consistent with the actual converter image recognition task scenario, and a higher model evaluation index can be obtained.

[0049] Model evaluation indicators of the algorithm: Accuracy, a measurement indicator of the image classification model, is the ratio of the number of correctly classified samples to the total number of samples. The closer the value is to 1, the better the model effect. F1-score is the harmonic mean of precision and recall for a certain category. For classification models such as image recognition, the higher the indicator, the better the effect. Precision, for a certain category, is the ratio of the number of samples correctly predicted as that category to the total number of samples predicted as that category. Recall, for a certain category, is the ratio of the number of samples correctly predicted as that category to the total number of samples in that category. MAP (mean average precision) is an indicator for measuring the effectiveness of the algorithm in the object detection algorithm. For object detection tasks, the precision and recall of each class of object can be calculated. After multiple calculations / experiments at different thresholds, a PR curve can be obtained for each class, and the area under the curve is the average.

[0050] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A converter image incremental automatic machine learning system, characterized in that: From bottom to top, it includes data layer, business layer and user layer; Data layer: used to process data, collect real-time converter video data, pre-process image data, and image recognition label data driven by the business layer to form an unstructured database system and classify and store typical converter abnormality scene data; Business layer: It is used as the core driver for user layer image increment and customer needs, and is also the data driver for the lower data layer; it includes user module and abnormal warning module. The abnormal warning module is responsible for model construction and prediction. The system reads abnormal data of the data layer by calling the abnormal warning module, and returns the processed results to the user module, including the parameters and abnormal conditions of each model; The user module returns the results required by the user to the user layer; User layer: used to provide users with an interface, online incremental images, required task input, and result display; The camera accurately monitors the color and shape of the converter scene. For four typical scenes, such as splashing, oxygen lance ignition failure, oxygen lance leakage, and post-pull blowing, the camera outputs the converter video data and sends it to the data layer for conversion into converter data set storage and high-quality image data output. The high-quality image data is sent to the business layer for image feature extraction and recognition, and the classified abnormal images are output to the user layer. The abnormal warning module of the business layer uses an image feature extraction algorithm to extract key features of the converter image, converts unstructured image data into structured feature vectors in the feature space, fuses the features through vector splicing, and then converts the unstructured converter image data into typical converter scene recognition through the converter scene image feature extraction algorithm. Finally, the abnormal warning module identifies the abnormal converter scene and issues an alarm message, and stores the abnormal warning model parameter information and abnormal result data in the database.

2. A method for establishing a data layer for the converter image incremental automatic machine learning system according to claim 1, characterized in that: The collected converter video data is filtered, preprocessed and stored through video summarization technology to form a complete converter scene training data set, verification data set and test data set, which are used for the business layer to classify and identify four types of typical scenes; The converter video data is processed by frame division, and the image data is preprocessed by using digital image processing technology, and the preprocessed high-quality image data is imported into the unstructured database system to form a converter image database; Domain experts annotate the valid data of the training data set, validation data set, and test data set, which are used for training, validation, and testing of business layer models, respectively.

3. A data layer establishment method for the converter image incremental automatic machine learning system according to claim 2, characterized in that: The image data preprocessing uses image normalization configuration to normalize the input image, and the stored and read images default to the BGR channel order; Preprocessing data flow: read the image, randomly scale and cut out the image, randomly flip the image horizontally with a probability of 0.4, and then normalize it. Finally, convert the image format into a tensor; the four typical scene labels in the configuration file are also converted into tensors through one-hot encoding.

4. A method for establishing a business layer for the converter image incremental automatic machine learning system according to claim 1, characterized in that: Establish a complete model tool chain including model training, model verification, and model deployment; The backbone network depth of the model training is 18 layers, and the loss function is set to cross entropy loss. First, the image and label data preprocessed by the data layer are read. The training optimization strategy: the optimizer selects stochastic gradient descent, the optimizer's learning rate is initially set, and the momentum, weight decay coefficient and training rounds are set.

5. A converter image incremental automatic machine learning system training method, characterized in that: This includes the construction of a systematic model process, the method of determining the network of an automatic machine learning model, and the establishment of model evaluation indicators for the algorithm; The model process of the system is constructed as follows: 1) To create a model on the system, you need to determine the model name and select the functions to be implemented by the model; 2) Upload data. After the user uploads the data, the data is labeled according to the data requirements of different model types. If there is locally labeled data, upload it directly; 3) Train the model and verify the effect, select the algorithm, configure the training data and other task-related parameters to start the training task; 4) After the model training is completed, it supports visual viewing of the model evaluation report, and identifies four abnormal conditions of the converter image through the model; 5) Finally, publish the model and deploy the satisfactory model on the server or local device; The automatic machine learning model network determination method: for selecting a backbone network, the parameter training and model structure search are decoupled through the neural network architecture search and network parameter determination method in the automatic machine learning, the backbone network model is first determined, and then the parameters are trained based on the determined network structure, so that the searched neural network is highly consistent with the task scenario of actual converter image recognition; Model evaluation metrics for the algorithm: Accuracy: a measurement indicator of the image classification model, the ratio of the number of correctly classified samples to the total number of samples. The closer the value is to 1, the better the model effect. F1-score, for a certain category, is the harmonic mean of precision and recall. For image recognition classification models, the higher the index, the better the effect; Precision, for a certain category, is the ratio of the number of samples correctly predicted as that category to the total number of samples predicted as that category; Recall rate, for a certain category, is the ratio of the number of samples correctly predicted as that category to the total number of samples in that category; MAP is an indicator for measuring the effectiveness of the object detection algorithm. For object detection tasks, the precision and recall rate of each type of object can be calculated. After multiple calculations / experiments at different thresholds, a PR curve can be obtained for each class, and the area under the curve is the average.

Citation Information

Patent Citations

  • Image incremental learning method based on data protection

    CN112115967A

  • Method for storing medical image and information interaction method and apparatus

    WO2015127795A1

  • System and method of operation for remotely operated vehicles leveraging synthetic data to train machine learning models

    WO2020030950A1

  • Application development method and apparatus based on machine learning model, and electronic device

    WO2021136365A1