A steel production data real-time processing method based on Kafka

Abstract

The application belongs to the technical field of steel molten iron production, and discloses a steel production data real-time processing method based on Kafka, wherein the real-time processing method is assisted by advanced real-time data processing technology, i.e., Kafka real-time processing, realizes the publishing and subscribing mode, theme partition, stream computing, message storage and the like of steel industry data, and realizes the real-time processing and intelligent grading of steel data in combination with data preprocessing and improved clustering analysis methods, so as to help improve the real-time dynamic adjustment capability of production plans and make the steel system more flexible and resilient.

Description

Technical Field

[0001] This invention belongs to the field of steel production technology, specifically a real-time steel production data processing method based on Kafka. Background Technology

[0002] The steel production system currently faces enormous data challenges. The volume of data generated by the system is massive, with daily production and consumption of molten iron reaching thousands of tons. Traditional data transmission methods struggle to handle such a large data flow, posing a significant challenge to conventional methods. Traditional approaches are ill-equipped to effectively process and transmit such massive data streams; their processing speed and transmission efficiency cannot meet the production process's requirements for data immediacy and accuracy. This challenge of large-scale data processing poses a major obstacle to maintaining the efficiency and accuracy of the production process, as timely and accurate data is crucial for production decisions and the stable operation of the production process.

[0003] Meanwhile, the steelmaking process involves a large amount of real-time data, including real-time monitoring of multiple parameters such as temperature, pressure, and safe production cycle volume. The immediacy of this data is crucial for ensuring the stability and safety of the production process. However, traditional batch processing systems cannot meet the requirements for this immediacy. Especially when processing large-scale, multi-parameter real-time data, traditional batch processing systems face a series of problems, such as transmission delays, data loss, and the inability to meet real-time monitoring needs. This high demand for real-time performance makes data processing in steelmaking systems even more challenging, as accurate and timely data acquisition is essential for the stable operation of the production process.

[0004] Furthermore, the steel system exhibits high uncertainty and instability in actual production. Therefore, maintaining the online molten iron circulation volume to ensure the normal operation of the entire system is particularly important. The uncertainty and instability of the steel system mean that an efficient emergency response mechanism is required. In the event of sudden abnormalities, operators must promptly obtain information about the current production anomalies and assign an anomaly level based on the specific circumstances of the anomaly, such as the degree of deviation between the actual production volume and the static plan, the degree of impact on the production process, and the severity of the anomaly. In traditional handling methods, manual paging is a common means, but its low efficiency and long response time limit the real-time adjustment of the production plan and are not conducive to the real-time dynamic adjustment of the production plan. Summary of the Invention

[0005] Based on the existing problems, this invention utilizes advanced real-time data processing technology, namely Kafka real-time processing, to realize the publish-subscribe model, topic partitioning, streaming computing, and message storage of data in the steel industry. Combined with data preprocessing and improved clustering analysis methods, it enables real-time processing and intelligent classification of steel data, which helps to improve the real-time dynamic adjustment capability of production plans and make the steel system more resilient and adaptable.

[0006] The core problem addressed by this invention is to provide a real-time steel production data processing method based on Kafka. This method can not only acquire system data in real time, but also provide feedback on the current operation of the steel system based on the acquired data. If the current situation is not conducive to the stable operation of the process, upstream production can make timely adjustments according to the specific circumstances.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0008] This invention is a real-time data processing method for steel production based on Kafka, such as... Figure 1 As shown, the method includes the following steps:

[0009] The upstream data source is obtained from the upstream production system and classified and archived according to the timestamp to build a data center. The classified and archived upstream data source serves as the molten iron data source. The data center is used for data query and processing. It should be noted that the upstream production system refers to the production system consisting of the blast furnace end and the converter end of the ironmaking and steelmaking production workshop, and the upstream data refers to all data generated by the upstream production system during the production process.

[0010] The data source classification interface divides the molten iron data source into blast furnace end data and converter end data. The blast furnace end data contains information related to molten iron production, such as temperature, pressure, and molten iron output. The converter end data contains information related to molten iron consumption, such as the demand for molten iron, its quality, and the steel grades produced.

[0011] The categorized molten iron data sources undergo preprocessing, which includes cleaning, missing data filling, standardization, and dimensionality reduction. The cleaning and missing data filling steps involve using linear interpolation to handle outliers and missing values ​​in the molten iron data to ensure data quality. After cleaning and missing data filling, features closely related to production quality and stability are selected from the molten iron data, and these features, such as temperature and molten iron composition, are standardized to ensure that each feature has the same numerical scale. The standardized molten iron data then undergoes dimensionality reduction, which uses an adaptive weighted K-means clustering algorithm with multi-dimensional features to reduce the complexity of the production data into fewer features, thereby improving data processing efficiency. This adaptive weighted K-means clustering algorithm with multi-dimensional features introduces multi-dimensional feature weights on top of the traditional K-means algorithm, taking each feature weight into account. In each iteration, cluster analysis is performed on the data samples based on the feature weights, and the cluster centers are updated based on the distance between the data samples and each feature. Then, based on the principle of forming clusters based on similar information, the molten iron data flowing into Kafka is preprocessed in batches.

[0012] Kafka is used to receive and optimize the preprocessed molten iron data. Kafka is a distributed platform for building real-time data channels and data stream processing. A typical Kafka module includes: a producer module, a consumer module, a topic partitioning module, a Kafka server module, and a Zookeeper distributed coordination service module. Specifically, the producer module sends data to the Kafka topic and includes a data capture unit for capturing data from various data sources, a data creation and serialization unit responsible for converting captured data into Kafka processing objects, a partitioner unit for processing data partitions, a batch processing unit for packaging data into batches, a data sending unit responsible for sending data to specified partitions, and a compression unit for compressing the data. The consumer module reads data from the Kafka topic, the topic partitioning module is a logical unit for data storage, and the Kafka server module stores data and provides data read / write services, as well as manages the Kafka cluster. The Zookeeper distributed coordination service module is used for collaborative scheduling of Kafka operations in a distributed environment. The steps for receiving and optimizing the pre-processed molten iron data using Kafka include: first, unifying the data access interface, defining and implementing a unified data format, classifying and storing data according to data source and timestamp, storing multiple copies of the data to improve system fault tolerance, and storing a replica on another server; capturing the data through a capture unit; then creating Kafka topics and partitions, creating multiple topics based on data source and production shift; next, improving the data processing unit by cleaning, filtering, feature engineering, and model prediction of the received messages to ensure data quality, and feeding the results back to the production system and database; finally, performing data content conversion and correction to ensure data accuracy and consistency; after the above Kafka receiving and optimization steps, preliminary Kafka-optimized data is formed. Through these improvements, the system can flexibly adjust performance optimization strategies, improve data capture and processing efficiency, and achieve the optimization goal of production data.

[0013] The Kafka Stream component performs in-depth analysis on the initially optimized Kafka data and feeds the results back to the upstream production system, allowing the upstream system to adjust its production plans accordingly. This real-time data monitoring and feedback module enables real-time monitoring of the production process, including the operating status, output, quality, and overall system operation of blast furnaces and converters. The system can quickly detect and respond to any potential anomalies, ensuring the efficiency, quality, and safety of the production process. The specific steps for performing in-depth analysis on the initially optimized Kafka data and feeding the results back to the upstream production system include: initializing the Kafka Stream application, configuring and defining the processing topology; creating a streaming processing topology, including the data source, processing nodes, and destination; performing logical aggregation analysis of data within a time window using window operations; further data transformation and processing using the Kafka Stream API; and finally, resending the processing results to the specified Kafka topic and feeding them back to the upstream production system, providing immediate insights and feedback for the production process.

[0014] A further improvement of the present invention is that the calculation formula for the feature weights is as follows:

[0015]

[0016] Among them, w k (t+1) x is the updated value of the weight of the k-th feature in the (t+1)-th iteration. ik c is the value of sample i on the k-th feature. jk It is the value of the j-th cluster center on the k-th feature, where n represents the number of samples and m is the number of features.

[0017] A further improvement of the present invention is that the formula for updating the cluster center based on the distance between data samples on each feature is as follows:

[0018]

[0019] Among them, w i It is the weight of feature i, x ik c is the value of sample i on the k-th feature. jk is the value of the j-th cluster center on the k-th feature, and m is the number of features.

[0020] A further improvement of this invention lies in the following: the database tool for constructing the data center is preferably HBase as the backend database. HBase features low latency, horizontal scalability, suitability for wide-table data models, and flexibility. HBase's low latency ensures rapid response to real-time data updates from the steel system. HBase employs a wide-table data model, which is ideally suited for storing the complex and diverse data generated in the steel system. Through HBase's column family structure, multidimensional data related to steel production can be efficiently stored and queried, including various indicators such as temperature, humidity, and pressure, thereby providing the system with more comprehensive real-time monitoring and analysis.

[0021] A further improvement of this invention is that the data source classification interface is preferably Flume, which is embedded in the data source classification stage. Flume is configured with data collectors from multiple data sources, including Flume agents for the blast furnace end and the converter end. These agents are responsible for collecting raw molten iron data in real time from different data sources, such as temperature, humidity, and pressure. By classifying the data input from Flume in a differentiated manner, the system categorizes the data from the blast furnace end and the converter end separately, providing a basis for subsequent differential processing.

[0022] A further improvement of this invention lies in the fact that although Kafka itself possesses certain data capture capabilities, its effectiveness is limited in practical applications, especially when processing large amounts of data, where it lacks flexibility and suffers from high latency and low throughput. Against this backdrop, this invention improves the data capture unit in the producer module by changing its capacity to an adjustable packet buffer capacity. This not only enables efficient data capture and maintenance, allowing the system to flexibly adjust performance optimization strategies based on changes in upstream data traffic, but also allows Kafka to better capture upstream traffic data and maintain the captured upstream data, ensuring both efficiency and integrity in data capture. Furthermore, this invention's data... The capture unit also includes a data prediction function, which dynamically optimizes the Kafka producer module based on traffic monitoring and ARIMA time-series prediction. Accordingly, the capture steps through the capture unit are optimized as follows: first, the data stream is rate monitored and sampled, and then the future traffic is predicted using the ARIMA time-series prediction model to pre-set a better packet buffer capacity value. On the other hand, to avoid the additional overhead caused by rate sampling and thus reduce application performance, an intelligent mechanism is designed: each sampling point is compared with the previous prediction value. When the difference between the prediction value and the sampled value is not large, the system will correspondingly shorten the sampling time and reduce the sampling frequency to achieve more efficient use of resources and performance optimization.

[0023] Beneficial effects: Over a certain period, Kafka, as the core engine for data stream processing, has accumulated a large amount of steel production data. This invention provides a real-time steel production data processing method based on Kafka. Utilizing Kafka Stream functionality, this method enables real-time, streaming analysis of this data. Kafka Stream can process data streams quickly and accurately, identifying potential patterns, trends, or anomalies using built-in analytical algorithms and real-time processing capabilities. The analysis results are then fed back to the upstream production end, providing real-time insights and feedback information for the production process. This real-time data analysis and feedback mechanism enables the upstream production end to adjust its production plan in a timely manner based on the feedback results and quickly handle abnormal situations, thereby improving the efficiency, quality, and reliability of the entire steel production system. Furthermore, this invention also classifies and preprocesses the data sources based on their characteristics, and introduces an improved K-means clustering algorithm during the dimensionality reduction process of preprocessing. In the traditional K-means algorithm, each data feature has the same impact on the clustering analysis. However, in this application, multi-dimensional data feature weights are introduced into the traditional K-means algorithm based on different data sources. The feature weights of the blast furnace end data and the converter end data reflect the relative importance of each feature in the sample clustering. By considering the feature weights, the impact of different features on the clustering results can be better reflected. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of a real-time steel production data processing method based on Kafka, as described in this application.

[0025] Figure 2 This is a structural diagram of the real-time acquisition, processing, and feedback of Kafka data;

[0026] Figure 3 This is a schematic diagram of the adaptive weighted K-means algorithm based on multi-dimensional features. Detailed Implementation

[0027] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described and explained in detail below with reference to the accompanying drawings.

[0028] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, the invention may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit and principles of the invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the scope of the claims of the invention.

[0029] The real-time steel production data processing method based on Kafka proposed in this invention mainly includes two aspects in its implementation: a preliminary data processing flow and a real-time data acquisition, processing, and feedback flow. A schematic diagram of the Kafka real-time data acquisition, processing, and feedback flow is shown below. Figure 2 As shown.

[0030] The implementation process of the preliminary data processing scheme proposed in this invention mainly includes the following steps:

[0031] Step S1: The primary consideration is to separate the database from the existing relational database and create a new database dedicated to real-time data processing. This invention selects HBase as the optimal solution and stores the data on a server. HBase is installed and deployed on the server, and a data table structure is constructed based on the relationships between molten iron data. Tables are created for blast furnace molten iron data generation, converter molten iron consumption, steel grade information, equipment status, energy consumption statistics, generated reports, and detailed molten iron information. These tables are used to comprehensively model the relevant data. This database architecture design aims to optimize real-time data processing, providing more efficient data storage and management to meet the complex data management and processing needs of the steel production system.

[0032] Step S2: Identifying the data sources from the blast furnace and converter ends. The system first extracts data items from the data sources to identify whether the data stream originates from the blast furnace or converter end. In subsequent data processing, it marks the data with furnace IDs, modifying them according to the blast furnace number of the ironworks or the converter number of the steelworks, for further source location modification and association with the data stream, embedding the IDs into the data as markers. This marking method not only ensures the traceability of data sources but also enables the system to distinguish data generated by different devices in subsequent processing and analysis. Then, a timestamp is assigned to each data item to ensure data order. The introduction of timestamps allows data to be more accurately routed to different Kafka topic partitions. For example, each topic partition represents data from a specific data source in different time periods. Based on the timestamp of each data item, the data is divided by hourly granularity. Each partition stores blast furnace or converter data for a specific hour of the day. By adding furnace IDs to distinguish between blast furnace and converter data and marking with timestamps, the system ensures that data is accurately routed to the corresponding Kafka partition for storage, thus laying a solid foundation for subsequent data processing and real-time feedback.

[0033] Step S3: Preprocess and optimize the blast furnace end and converter end data after step S2 using an improved K-means clustering algorithm;

[0034] Step S31: First, select the data timestamp attribute as the clustering feature, and process the selected timestamp data using the K-means clustering algorithm. Before this, clarify the expected number of clusters to ensure that the algorithm divides the timestamp data into clusters corresponding to the hours of the production shift. The K-means algorithm assigns data points to different clusters based on the numerical proximity of the timestamps, ensuring that data with similar time features are grouped reasonably. In the traditional K-means algorithm, each data feature has the same impact on clustering analysis. However, in this application, depending on the data source, the improved K-means clustering algorithm introduces multi-dimensional data feature weights into the traditional K-means algorithm, mainly divided into blast furnace end data and converter end data. These feature weights reflect the relative importance of each feature in the sample clustering. By considering the feature weights, the impact of different features on the clustering results can be better reflected. The learning of the feature weights is designed for the respective features of the blast furnace end data and converter end data, such as... Figure 3 As shown, the preprocessing and optimization of the blast furnace and converter end data after step S2 using the improved K-means clustering algorithm includes four steps: Step S311 Initial weight setting, which sets an initial weight value for each data feature; Step S312 Sample weighting, in each iteration, the samples are weighted according to the current feature weights, meaning that for each sample, its feature value will be weighted according to the corresponding weight to obtain a weighted sample representation; Step S313 Weight update, which dynamically adjusts the weights according to the importance of the features; Step S314 Iterative optimization, by repeatedly performing the above steps until the algorithm converges or reaches the set upper limit of the number of iterations, in each iteration, the samples are weighted according to the current feature weights. The weight calculation formula of this method is as follows, where w k (t+1) x is the updated value of the weight of the k-th feature in the (t+1)-th iteration. ik c is the value of sample i on the k-th feature. jk It is the value of the j-th cluster center on the k-th feature, where n represents the number of samples and m is the number of features.

[0035]

[0036] Based on the weights of each feature, cluster analysis is performed on the samples according to the weights of each feature in each iteration. The algorithm is dynamically adjusted through feature weight adjustments, cluster center updates, and iterative optimization, making it more adaptable to different data. These improvements enhance the accuracy of data acquired during steel production. The formula for calculating the distance between samples in this method is as follows, where w... i It is the weight of feature i, x ikc is the value of sample i on the k-th feature. jk is the value of the j-th cluster center on the k-th feature, and m is the number of features.

[0037]

[0038] Step S32: After completing the clustering, evaluate and interpret the clustering results. Combined with visualization analysis, by plotting scatter plots of timestamp data and time series plots within clusters, we can gain a deeper understanding of the temporal distribution among different clusters. This analysis helps to initially understand the temporal distribution characteristics of data points within different time clusters and provides explanation and labeling for subsequent optimization.

[0039] Step S33: The pre-processed and optimized molten iron data is then transmitted to the Kafka real-time processing module.

[0040] Step S4: After the preliminary data processing scheme proposed in this invention is implemented, the real-time data acquisition, processing, and feedback process begins. This process mainly includes the following steps:

[0041] Step S41: Unify the data access interface, define the interface and the format of the received data. Each data source and processing function needs to implement the corresponding interface and data format. Data is classified and stored in different topics according to different data sources and different data timestamps to facilitate data management and feedback by the Kafka real-time module. At the same time, to improve system fault tolerance, the data is stored in multiple copies on another server to prevent the loss of critical data. The data access here uses an improved capture unit to achieve efficient capture, ensuring that the system can flexibly adjust the performance optimization strategy according to the changes in upstream data traffic. Based on the data prediction function, that is, using ARIMA time series prediction, the future data flow is predicted to set a better packet cache capacity value and improve data processing efficiency.

[0042] Step S42: Create Kafka topics and partitions. Based on the data sources at the blast furnace end and the converter end, and combined with the production shift, each corresponds to a Kafka topic. There are three shifts per workday, which means that six topics will be created each workday for data processing. Considering that the duration of each shift is eight hours, each topic is partitioned by hour and backed up twice on different hosts to improve the efficiency of parallel data processing and the fault tolerance of data.

[0043] Step S43: Improve the data processing unit. The data processing unit is responsible for in-depth processing and analysis of the messages received by Kafka. After receiving the message, since the data may change or be abnormal in the actual production process, such as abnormal high-end temperature or abnormal molten iron composition, the data is first cleaned and filtered to remove abnormal data or unnecessary information to ensure the quality and accuracy of the data. Secondly, feature engineering is performed according to the analysis objectives, mainly including feature extraction, transformation and selection, to extract the effective information in the data. After that, a model prediction is built through data analysis and aggregation. A prediction model is built based on the data stream in Kafka to perform data prediction. Then, the processed result data is sent to the upstream production system, and the analysis results are stored in the database.

[0044] Step S44: After the initial processing, there may be data content transformation, that is, the data is transformed or corrected to avoid the existence of abnormal data, so as to ensure the accuracy and consistency of the data and make the data more suitable for subsequent processing and storage. Through these processing steps, this module can effectively process the data, make the production data more valuable, and achieve the goal of production optimization.

[0045] Step S45: Implement the second phase of data processing, using Kafka Streams for real-time data processing. Compared to existing processing methods, Kafka Streams provides a smoother and more direct way to process real-time data streams. Window operations are used to perform aggregation analysis on the data within a time window, and the APIs provided by Kafka Streams are used to transform the data and generate new data results. This step is further divided into the following sub-steps:

[0046] Step S451: Initialize the Kafka Stream application, set the application configuration, specify input and output topics, and define the required processing topology;

[0047] Step S452: Build a streaming topology using the high-level API provided by Kafka Streams, including the source of the data stream, processing nodes, and the final destination;

[0048] Step S453: Use window operations to aggregate the data within a time window. Based on a fixed window length per hour, within each time window, the aggregation function will be applied to the data in the data stream and generate the aggregation result.

[0049] Step S454: Based on the aggregated data within the time window, use the API methods provided by Kafka Streams to perform further data transformation and processing, including calculating statistical data, performing complex computational logic, and generating new data structures.

[0050] Step S6: After completing the second stage of data processing, resend the data to the designated Kafka topic based on the processing results for use in subsequent steps or by the system. Simultaneously, the analysis results are fed back to the upstream production end, providing real-time insights and feedback for the production process. This streaming processing mechanism allows the system to process data more flexibly and efficiently, fully utilizing real-time data streams to achieve deeper data analysis and mining.

[0051] Based on the specific production scenario of a steel company in Nanjing, a practical case is provided. The production system includes 5 ironworks and 5 steelworks. The 5 ironworks involve 9 blast furnaces, and the 5 steelworks involve 5 converters. To facilitate the identification of subsequent data sources, these 14 devices are pre-assigned unique ID numbers for subsequent data source identification. The data generated by these data sources will additionally include their device ID number and a timestamp sequence in hourly units, with as many as 24 timestamps per day.

[0052] Then, the data type generated by each device is defined. Using a clustering algorithm, the data is preprocessed and cleaned by using timestamp attributes as clustering features. This removes dirty data to ensure that the data flowing into Kafka is correct. Taking the blast furnace in an ironworks as an example, the data it generates should include: iron consumption, molten iron temperature, molten iron composition, molten iron tonnage, fuel consumption, etc. It is necessary to ensure that these data are within a reasonable range. For example, outliers such as an iron consumption value of 5.8 should be removed.

[0053] Subsequently, within the Kafka topic and partition, the data is paired according to device ID and timestamp, and then partitioned based on different characteristics of the data source and timestamp, generating corresponding data streams. These data streams will then be sent to the Kafka topic and partition.

[0054] Data in Kafka is processed in real time using Kafka Streams to meet the real-time requirements of the production process. Analysis reports are generated based on the results, enabling data analysis and mining. Real-time analysis of this data allows for the timely detection of anomalies or potential problems in the production process. For example, if the temperature of molten iron in a blast furnace suddenly rises abnormally, it may indicate an operational malfunction or equipment failure, requiring prompt adjustments and interventions to avoid impacting production.

Claims

1. A method for real-time processing of steel production data based on Kafka, characterized in that: The real-time processing method includes the following steps: The upstream data source is obtained from the upstream production system, and the upstream data source is classified and archived according to the timestamp to build a data center. The classified and archived upstream data source serves as the molten iron data source, and the data center is used for data query and processing needs. The data source classification interface divides the molten iron data source into blast furnace end data and converter end data. The blast furnace end data is related to molten iron production, including temperature, pressure, and molten iron output. The converter end data is related to molten iron consumption, including the demand for molten iron, its quality, and the steel grades produced. The classified molten iron data source undergoes preprocessing, which includes cleaning, missing data filling, standardization, and dimensionality reduction. The cleaning and missing data filling steps involve using linear interpolation to handle outliers and missing values ​​in the molten iron data to ensure data quality. After cleaning and missing data filling, features closely related to production quality and stability are selected from the molten iron data, and these features are standardized to ensure that each feature has the same numerical scale. The standardized molten iron data then undergoes dimensionality reduction, which uses an adaptive weighted K-means clustering algorithm with multi-dimensional features to reduce the complexity of the production data into fewer features. This adaptive weighted K-means clustering algorithm with multi-dimensional features introduces multi-dimensional feature weights on top of the traditional K-means algorithm, considering each feature weight. In each iteration, cluster analysis is performed on the data samples based on the feature weights, and the cluster centers are updated based on the distance of the data samples on each feature. Subsequently, based on the principle of forming clusters based on similar information, the molten iron data flowing into Kafka is preprocessed in batches. The preprocessed molten iron data is received and optimized using Kafka. Kafka includes a producer module, a consumer module, a topic partitioning module, a Kafka server module, and a Zookeeper distributed coordination service module. The producer module sends data to the Kafka topic and includes a data capture unit for capturing data from various data sources. After the above Kafka receiving and optimization steps, pre-optimized Kafka data is formed. The Kafka Stream component performs in-depth analysis on the data from the initial Kafka optimization and feeds the analysis results back to the upstream production system, which can then adjust its production plan accordingly.

2. The method for real-time processing of steel production data based on Kafka as described in claim 1, characterized in that: The formula for calculating the feature weights is as follows: Among them, w k (t+1) x is the updated value of the weight of the k-th feature in the (t+1)-th iteration. ik c is the value of sample i on the k-th feature. jk It is the value of the j-th cluster center on the k-th feature, where n represents the number of samples and m is the number of features.

3. The real-time steel production data processing method based on Kafka as described in claim 1, characterized in that: The formula for updating cluster centers based on the distances between data samples on various features, i.e., calculating the distances between samples, is as follows: Among them, w i It is the weight of feature i, x ik c is the value of sample i on the k-th feature. jk is the value of the j-th cluster center on the k-th feature, and m is the number of features.

4. The method for real-time processing of steel production data based on Kafka as described in claim 1, characterized in that: The database backend of the database tool used to build the data center is HBase.

5. The method for real-time processing of steel production data based on Kafka as described in claim 1, characterized in that: The data source classification interface is Flume.

6. The method for real-time processing of steel production data based on Kafka as described in claim 1, characterized in that: The capacity of the data capture unit has been changed to an adjustable packet cache capacity, and the data capture unit also includes a data prediction function, that is, to dynamically optimize the Kafka producer based on traffic monitoring and ARIMA time series prediction.

7. The method for real-time processing of steel production data based on Kafka as described in claim 6, characterized in that: The data capture unit performs the following steps: first, it monitors and samples the data stream rate; then, it uses the ARIMA time-series prediction model to predict future traffic in order to pre-set a better packet buffer capacity value; and each sampling point is compared with the previous prediction value. When the difference between the prediction value and the sampled value is not significant, the system will correspondingly shorten the sampling time and reduce the sampling frequency.

Images