A metallurgical furnace condition recognition method based on a streaming multi-modal graph neural network

Abstract

The application discloses a metallurgical furnace condition recognition method based on a streaming multimodal graph neural network, which extracts visible light image features of a furnace material surface by using a convolutional neural network, extracts time series data features by using a multilayer Transformer encoder, deeply fuses double-mode features through a bidirectional cross-attention mechanism, constructs a dynamic memory bank, stores historical sample features by using a first-in-first-out strategy, takes current sample features as a query vector, filters neighbor nodes based on double-mode similarity, constructs a dynamic topology graph through an adaptive edge weight fusion module, and outputs a furnace condition recognition result by using a lightweight graph neural network to weight, aggregate and update node features. The application adopts a semi-supervised learning strategy, combines cross-entropy loss, Laplace loss and consistency loss for joint optimization, realizes high-precision furnace condition recognition under the condition of only a small amount of labeled samples, has the advantages of lightweight, low latency and support for streaming incremental reasoning, and is suitable for edge deployment in a metallurgical industrial field.

Description

Technical Field

[0001] This invention belongs to the field of metallurgical automation and intelligent detection technology, specifically relating to a method for identifying the condition of metallurgical furnaces based on a streaming multimodal graph neural network. Background Technology

[0002] Metallurgical furnaces are core thermal equipment in the iron and steel and non-ferrous metallurgical processes, and their conditions affect product quality, energy consumption, production safety, and continuous operation. These equipment typically exhibit characteristics of high temperature and strong interference, dynamic and time-varying conditions, and multiple coupled factors. During operation, they are susceptible to fluctuations in raw material composition, adjustments to operating parameters, and equipment aging and degradation, leading to frequent abnormal furnace conditions. These abnormalities not only result in decreased production efficiency and lower product qualification rates but can also cause serious malfunctions such as furnace slagging and equipment burn-out, and even trigger safety accidents, severely impacting production continuity and system stability.

[0003] Currently, methods for identifying the condition of metallurgical furnaces mainly include traditional empirical rule methods, single-modal analysis methods, and multimodal fusion identification methods. Traditional methods based on threshold judgment and empirical rules are simple to deploy, have low computational overhead, and can quickly respond to basic judgment needs, making them practical in standardized scenarios. However, they are difficult to adapt to non-steady-state scenarios with dynamic fluctuations in furnace conditions. Single-modal methods have significant shortcomings: methods based on time-series sensor data rely on process parameter sequences for global quantitative analysis, offering significant advantages in real-time performance and coverage, but lack visually intuitive features; methods based on visible light images can capture visual features such as flame morphology and furnace status, providing high recognition of typical furnace conditions, but lack quantitative assistance from process parameters. Both types of methods suffer from a lack of information dimensions. Multimodal fusion methods can leverage the complementary nature of data, resulting in higher identification accuracy and reliability, and better model convergence and interpretability. However, they heavily rely on large-scale, high-quality labeled data, while the high cost of on-site labeling and the scarcity and uneven distribution of fault samples limit their practical application.

[0004] In summary, bimodal data can complementarily represent furnace conditions, and feature fusion is key to effective furnace condition identification. Mainstream methods rely on high-quality labeled data, but labeling in industrial settings is difficult, fault samples are scarce and unbalanced, severely limiting their engineering applicability. Addressing the current situation of limited labels and heterogeneous multimodal semantics in metallurgical furnaces, this invention proposes a multimodal semi-supervised identification method based on streaming graph neural networks. Utilizing visible light images of furnace material surfaces and time-series data of the furnace production process, a technical solution is designed that incorporates bimodal feature extraction, deep fusion with cross-attention, adaptive edge-weighted graph construction, lightweight graph neural network streaming inference, and semi-supervised multi-loss function joint iteration. This overcomes the limitations of insufficient single-modal representation and solves the problem of unlabeled data being difficult to utilize, significantly improving the real-time performance and generalization ability of furnace condition identification, providing reliable technical support for stable furnace operation.

[0005] A search revealed that Chinese invention patent CN119646395A proposes a method for identifying and controlling blast furnace conditions. This method includes: acquiring historical blast furnace data and performing cleaning and preprocessing to obtain historical time-series data; capturing blast furnace adjustment and identifying states and events from the historical time-series data to generate result time-series data; constructing a time-series-to-series deep learning model based on the historical and result time-series data, and training the model; inputting real-time collected data into the trained model, and outputting blast furnace condition identification results and operational suggestions.

[0006] The method employed in the aforementioned patent integrates the changes in operating conditions and manual adjustment data from the historical operation data of the blast furnace, fully exploring the value of the full-dimensional data of blast furnace production. However, in this patent, image data is only treated as ordinary time-series input data and processed in a homogenized manner. The core features of its visual dimension are not extracted and utilized separately, resulting in the value of image data not being specifically explored and the expressive power of the model being limited.

[0007] Chinese invention patent CN112819802B proposes a method for monitoring and predicting blast furnace condition anomalies based on deep learning of tuyere information. The method includes: extracting and preprocessing blast furnace tuyere images according to feature frequencies; employing adaptive median filtering and gamma transform for denoising, and grayscale binarization and upscaling to enhance contrast; automatically clustering the preprocessed images using unsupervised spectral clustering to generate labels and constructing a tuyere image sample database; establishing a TI-ResNet model to extract image features and training it; and inputting real-time offline tuyere monitoring data into the trained model to achieve monitoring and prediction of blast furnace condition anomalies, while incrementally updating the dataset and model parameters at a specified frequency.

[0008] The method employed in the aforementioned patent fully leverages the visual feature value of tuyere image data through automatic image preprocessing, unsupervised label generation, and closed-loop incremental training, thereby improving the automation and intelligence level of furnace anomaly identification. However, this patent relies solely on single-modal image data and does not integrate time-series numerical data of the furnace, resulting in the inability to utilize multimodal complementary information. Consequently, the method has limitations in recognition accuracy and generalization under complex scenarios. Summary of the Invention

[0009] This invention proposes a method for identifying the condition of metallurgical furnaces based on a streaming multimodal graph neural network. This method innovatively integrates multimodal information such as time-series data and visible light images, and fully explores the potential value of a large amount of unlabeled data by relying on semi-supervised learning. While significantly reducing labeling costs, it overcomes the limitations of human experience and achieves accurate identification of complex furnace conditions. Furthermore, it achieves lightweight deployment of the model through multiple lightweight network designs, adapting to the low latency and low resource requirements of industrial edge computing scenarios. This solution aims to construct a semi-supervised intelligent identification method, transforming traditional human-dependent experience-based judgments into data-driven conclusions from a semi-supervised model, thereby realizing intelligent online identification of the condition of metallurgical furnaces.

[0010] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0011] A method for identifying the furnace condition of a metallurgical furnace based on a streaming multimodal graph neural network includes the following steps:

[0012] S1. Use a convolutional neural network to extract image features of the furnace material surface, and use a multi-layer Transformer encoder to extract time-series data features of the furnace, forming image feature vectors and time-series feature vectors;

[0013] S2. Perform bidirectional cross-attention fusion on the image feature vector and the temporal feature vector to generate a unified fused feature;

[0014] S3. Construct a model memory library to store the extracted image feature vectors and temporal feature vectors;

[0015] S4. Using the bimodal features of the current sample as the query vector, calculate the bimodal similarity with the features of all historical samples in the memory bank, select the TopK neighbor nodes based on the average similarity, complete the edge weight fusion through modal weight coefficients, and construct a dynamic topology graph.

[0016] S5. Lightweight graph neural networks are used to update the fusion features and identify the conditions of each furnace.

[0017] Further, step S1 specifically includes:

[0018] Image features of the furnace material surface are extracted using a deep separable convolutional neural network, and the corresponding formula is:

[0019]

[0020] in, Given an input image matrix, The image features are extracted by a depthwise separable convolutional neural network, with dimension D being 64.

[0021] By extracting the long-short-term dependencies of time series parameters through multiple Transformers, the characteristics of time series data are obtained. The corresponding formula is:

[0022]

[0023] in, This is a time-series data vector with 35 dimensions. The extracted time-series data features have a dimension D of 64.

[0024] Further, step S2 specifically includes:

[0025] S21. Let the visible light image features output by the encoder for the i-th sample be... The time series characteristics are Both dimensions are aligned to the same semantic space;

[0026] S22. Cross-attention guided by temporal features uses temporal features as queries and image features as keys and values. It mines the attention relationships between temporal features and image features through a multi-head attention mechanism. The calculation formula is:

[0027]

[0028] The query, key, and value are obtained by linear projection of their corresponding features:

[0029]

[0030] , , Let be the time-series query matrix, image key matrix, and image value matrix of the i-th sample, respectively. , , Let be the learnable projection matrix of the i-th sample. , , For bias terms, Temporal enhancement features that fuse image information under the guidance of temporal features;

[0031] S23. Image feature-guided cross-attention uses image features as queries and temporal features as keys and values ​​to reverse-engineer the attention relationship between image features and temporal features. The calculation formula is as follows:

[0032]

[0033] Among them, Query, Key, and Value are obtained by linear projection of the corresponding features:

[0034]

[0035] , , The image query, time series key matrix, and time series value matrix for the i-th sample are respectively. , , Let be the learnable projection matrix of the i-th sample. , , For bias terms, Image enhancement features that integrate temporal information and are guided by image features;

[0036] S24. Attention output is processed via residual connection and layer normalization:

[0037]

[0038] in, For the initial fusion of image features, Features of the time series data for initial fusion;

[0039] S25. Adaptive fusion is then achieved through a gating mechanism. The gating weight calculation formula is as follows:

[0040] Image features, temporal features, and fused temporal features of the furnace are concatenated and input into a fully connected layer. ReLU activation is applied, and finally, the layer is normalized using the Softmax function to obtain a weight vector of temporal data and visible light images. ;

[0041] S26. The two vectors obtained from the initial fusion are weighted and added together, and then concatenated with the fusion vector obtained by directly adding the two initial vectors. After passing through a fully connected layer, a unified fusion feature that can comprehensively represent the furnace condition is finally generated. :

[0042] .

[0043] Further, step S4 specifically includes:

[0044] S41. First, construct a graph for classifying the current furnace samples, and calculate the cosine similarity between the image and the temporal modality:

[0045]

[0046] in, The cosine similarity of the embeddings of each graph in the memory bank. The cosine similarity of the embeddings for each time-series data in the memory bank. The image feature vector of the center node, The time-series data feature vector of the central node. The image feature vectors of the nodes in the memory bank. This represents the temporal data feature vector of the nodes in the memory bank;

[0047] S42. For the current sample, use the average similarity between the two modalities as the retrieval score, select the top K neighbor indices with the highest scores from the memory bank, and connect the samples corresponding to these indices to form a KNN graph.

[0048] S43. After the candidate neighbors are determined, the adaptive edge weight fusion module is entered: for each edge, the image similarity and temporal similarity between the center node and its neighbors are extracted simultaneously, and then they are used as inputs to automatically learn the gate control coefficients through MLP, and the fused similarity is output by combining the product enhancement term.

[0049] S44. Softmax normalization is applied to the fusion similarity to obtain the final edge weights:

[0050]

[0051] in, The temperature coefficient is taken as 1.0. For bimodal overall similarity, This is the final side weight.

[0052] Furthermore, the method for updating the fused features using a lightweight graph neural network in step S5 is as follows:

[0053] The lightweight graph neural network contains only a single fully connected layer and uses the GELU activation function. The features of neighbor nodes are represented by simple averaging. The features of the center node are concatenated with the aggregated messages of its neighbors and then updated by the fully connected layer.

[0054] Furthermore, the identification method employs a semi-supervised learning strategy, constructing a loss function that includes cross-entropy loss, Laplace loss, and consistency loss. The model update batch size is 64, and the AdamW optimizer is used for parameter updates.

[0055] Compared with the prior art, the present invention has the following significant advantages:

[0056] 1. This invention constructs a multi-level and multi-dimensional feature aggregation module, deeply explores the intrinsic correlation and complementarity between the time-series numerical features of the furnace production process and the visual image features of the furnace material surface, strengthens the correlation and integrity of multimodal information through bidirectional cross attention, effectively enhances the overall feature expression capability, and provides high-quality feature vectors for subsequent models to accurately complete the task of identifying the furnace condition of complex metallurgical furnaces.

[0057] 2. This invention designs a dedicated streaming memory adapted to the actual situation of continuous production and continuous data generation in furnaces and kilns. It ensures that the memory retains the latest and most representative historical dual-modal data, establishes a data foundation for dynamic graph topology construction and streaming inference, and meets the industrial application requirements for uninterrupted and long-term stable monitoring of furnaces and kilns.

[0058] 3. This invention designs a topology modeling method that combines dual-modal adaptive edge weight graph construction with a lightweight graph inference network. This method dynamically integrates the node similarity between the visible light visual features of the furnace material surface and the time-series data features of the furnace production process through an adaptive gating mechanism to construct a graph topology that fits the real furnace conditions. It also uses a lightweight graph inference network with only a single fully connected layer to complete efficient node association feature learning, so as to accurately characterize the relationship between nodes in the furnace conditions with lightweight computing power.

[0059] 4. This invention constructs a semi-supervised learning architecture with joint constraints of multiple losses. It performs weighted summation of three types of loss functions: cross-entropy loss, Laplace loss, and consistency loss. It completes model iteration by relying on a small number of manually labeled samples combined with massive amounts of unlabeled field data, which can significantly reduce the cost of furnace and kiln data labeling and effectively improve the model's generalization ability in real production scenarios.

[0060] 5. This invention addresses the furnace condition identification needs of continuous production in metallurgical furnaces and kilns. It employs a multi-dimensional feature fusion mechanism to deeply mine the complementary characteristics of multi-source data, constructs a dynamic memory adapted to continuous industrial data streams, proposes a lightweight inference model using an adaptive edge weight graph construction strategy, and introduces a semi-supervised learning method for multi-objective joint optimization, ultimately achieving intelligent identification of complex furnace conditions. Attached Figure Description

[0061] Figure 1 This is a flowchart of the method of the present invention;

[0062] Figure 2 This is the confusion matrix diagram of the test set in Example 2. Detailed Implementation

[0063] The present invention will now be described in further detail with reference to the accompanying drawings:

[0064] Example 1:

[0065] like Figure 1 As shown, this invention provides a method for identifying the furnace condition of metallurgical furnaces based on a streaming multimodal graph neural network, the specific steps of which include:

[0066] S1. Extract features from visible light images and time-series data.

[0067] To extract visible light image features from the furnace material surface, this invention utilizes a deep separable convolutional neural network to extract these features. This significantly reduces the number of model parameters and improves training efficiency. The corresponding formula is as follows:

[0068]

[0069] in, Given an input image matrix, The image features are extracted by a depthwise separable convolutional neural network, with dimension D being 64.

[0070] To extract time-series data features during the furnace production process, this invention uses a multi-layer Transformer to extract the long-term and short-term dependencies of time-series parameters. The corresponding formula is as follows:

[0071]

[0072] in, This is a time-series data vector with 35 dimensions. The extracted time-series data features have a dimension D of 64.

[0073] This invention employs a dual-path feature extraction scheme: CNN extracts spatial features from visible light images of furnaces, while Transformer mines the global dependencies of time-series data. This approach preserves the local structural accuracy of visual information while capturing the dynamic changes in time-series data. After feature extraction for each modality, the output feature vectors of both are uniformly mapped to the same 64-dimensional semantic space, achieving standardized alignment of feature dimensions between modalities. This design effectively eliminates the dimensional heterogeneity problem of features from different modalities, allowing heterogeneous features to form deep associations and information complementarity within a shared semantic framework, thus laying a solid feature foundation for subsequent fusion modeling and downstream task inference.

[0074] S2. Perform bidirectional cross-attention fusion of visible light feature vectors and temporal feature vectors.

[0075] This invention achieves bidirectional interaction between furnace images and temporal features through a bidirectional cross-attention module, and completes adaptive fusion by combining a gating mechanism. Let the visible light image features output by the encoder for the i-th sample be... The time series characteristics are Both dimensions are aligned to the same semantic space.

[0076] The temporal feature-guided cross-attention uses temporal features as the query and image features as the key and value. It mines the attention relationship between temporal features and image features through a multi-head attention mechanism. The calculation formula is as follows:

[0077]

[0078] Among them, Query, Key, and Value are obtained by linear projection of the corresponding features:

[0079]

[0080] , , Let be the time-series query matrix, image key matrix, and image value matrix of the i-th sample, respectively. , , Let be the learnable projection matrix of the i-th sample. , , For bias terms, This is a temporal enhancement feature that fuses image information under the guidance of temporal features.

[0081] Image feature-guided cross-attention uses image features as the query, temporal features as the key and value, and reverse-engineers the attention relationship between image features and temporal features. The calculation formula is as follows:

[0082]

[0083] Among them, Query, Key, and Value are obtained by linear projection of the corresponding features:

[0084]

[0085] , , The image query, time series key matrix, and time series value matrix for the i-th sample are respectively. , , Let be the learnable projection matrix of the i-th sample. , , For bias terms, Image enhancement features are derived from image features and incorporate temporal information.

[0086] Attention output is processed via residual connections and layer normalization:

[0087]

[0088] in, For the initial fusion of image features, The characteristics of the time-series data are initially integrated.

[0089] Then, adaptive fusion is achieved through a gating mechanism. The gating weight calculation formula is as follows:

[0090] Image features, temporal features, and fused temporal features of the furnace are concatenated and input into a fully connected layer. ReLU activation is applied, and finally, the layer is normalized using the Softmax function to obtain a weight vector of temporal data and visible light images. .

[0091] Since the feature vectors extracted from visible light images and time-series data have the same dimension and reside in the same semantic space, they can be directly added together to obtain a fused vector. The two initially fused vectors are weighted and added together, then concatenated with the fused vector obtained by directly adding the two initial vectors. After passing through a fully connected layer, a unified fused feature capable of comprehensively representing the furnace condition is finally generated.

[0092]

[0093] The final fusion vector will serve as the primary basis for subsequent furnace condition classification.

[0094] S3. Construction of the memory bank

[0095] The memory bank is the core feature storage carrier of the streaming multimodal graph neural network model of this invention. It is used to centrally store historical furnace condition sample data to serve the continuous and dynamic furnace condition classification task of metallurgical furnaces. The memory bank independently stores the visible light image embedding features of each input sample output by the dual encoder. With temporal embedding features .

[0096] The memory strictly employs a first-in, first-out (FIFO) dynamic capacity management mechanism, with a preset maximum storage capacity threshold N=30000, thereby limiting memory usage and meeting the requirements of lightweight operation. Let the current number of samples stored in the memory be... The number of samples to be stored in a single batch is B, when the following conditions are met: At that time, redundant historical data that was first entered into the queue is automatically removed. Through this iterative update logic, the present invention can always retain the latest time-series and most representative dual-modal feature set of furnace conditions within a fixed storage space.

[0097] Overall, the memory not only provides massive and reliable historical prior support for bimodal similarity calculation, adaptive edge weight construction, and lightweight GNN neighborhood message passing, but also controls memory consumption based on a limited first-in-first-out caching strategy, adapting to the practical application needs of lightweight deployment and 24 / 7 uninterrupted streaming monitoring in the metallurgical industry.

[0098] S4. Construction of Flow Topology Graph

[0099] During the topology graph construction phase, the current furnace sample to be classified is taken as the central node, and its bimodal features are used as the query vector. The bimodal adaptive similarity is calculated with the features of all historical samples in the memory.

[0100] First, construct a graph for classifying the current furnace samples, and calculate the cosine similarity between the image and the temporal modality:

[0101]

[0102] in, The cosine similarity of the embeddings of each graph in the memory bank. The cosine similarity of the embeddings for each time-series data in the memory bank. The image feature vector of the center node, The time-series data feature vector of the central node. The image feature vectors of the nodes in the memory bank. This represents the temporal data feature vector of a node in the memory bank.

[0103] For the current sample, in order to quickly determine candidate neighbors, this invention uses the average similarity between the two modalities as the retrieval score, and selects the top K neighbor indices with the highest scores from the memory. In this invention, top K is set to 50. The samples corresponding to these indices are connected to them to form a KNN graph.

[0104] After candidate neighbors are determined, this invention does not directly use single-modal similarity as edge weights. Instead, it enters an adaptive edge weight fusion module: for each edge, it simultaneously extracts the image similarity and temporal similarity between the center node and its neighbors, and then uses these as inputs to automatically learn gate control coefficients through a small MLP, adaptively balancing the contributions of the two modalities. Simultaneously, a product enhancement term is added to encourage neighbors that are "similar in both modalities" to obtain higher scores, ensuring that if a single modality considers two samples similar, the edge weight connecting the two samples is low; if both modalities consider two samples similar, the edge weight connecting the two samples is high. Finally, a softmax operation is performed on the fused scores to obtain normalized edge weights, ensuring that the sum of the weights of the K neighbors of the same node is 1.

[0105] The formula for the adaptive edge weight strategy is as follows:

[0106]

[0107] in, These are the modal weighting coefficients. For bimodal overall similarity, This is the gain coefficient for the product term, taken as 0.5.

[0108] Then, the fusion similarity is normalized using Softmax to obtain the final edge weights:

[0109]

[0110] in, The temperature coefficient is set to 1.0. This is the final side weight.

[0111] In the training phase, this invention adopts a batch-based streaming graph construction mode. After completing the graph construction and classification prediction for each furnace condition node in the current batch, the dual-modal features of the entire batch of nodes are stored in a dedicated streaming memory in batches, continuously providing the latest historical features to support the lightweight GNN dynamic topology construction of subsequent batch nodes.

[0112] In the testing phase, this invention adopts a sample-by-sample real-time streaming graph construction mode. For each set of dual-modal monitoring data acquired in actual production, furnace condition classification and prediction are immediately completed, and the dual-modal features of the sample are stored one by one in the streaming memory. The furnace condition node association topology is dynamically updated, ensuring low latency in furnace condition identification throughout the process, which meets the actual application requirements of online real-time monitoring in the metallurgical industry.

[0113] S5, Feature Update of Graph Neural Networks

[0114] Based on the furnace condition association graph obtained by bimodal adaptive edge weighting graph construction, this invention uses a lightweight GNN layer to perform weighted aggregation and iterative updates on the features of neighboring nodes retrieved from the memory bank, allowing the current furnace condition sample to fully absorb prior knowledge of similar historical furnace conditions, thus significantly improving the discrimination accuracy of various furnace conditions.

[0115] When updating the graph neural network, each sample of the furnace condition to be tested is treated as a node to be predicted. This node serves as the central node. Its initial features are the fused features obtained through a bidirectional cross-attention mechanism. After obtaining the neighbor index and edge weight, the two-modal features of the neighbors are retrieved from the memory bank, and a simple average is used to form the neighbor representation. This avoids repeatedly performing cross-attention on each neighbor, saving memory and computation.

[0116] Fusion characteristics of neighboring nodes The calculation formula is:

[0117]

[0118] The aggregation of the center node and its neighbor nodes occurs in the lightweight GNN layer: for each node, the final weights obtained through an adaptive edge weighting strategy are... As attention weights, the aggregated message is obtained by weighted summation of the neighbor features.

[0119] The neighbor aggregation formula for the GNN layer is:

[0120]

[0121] in, This is aggregated information for all neighboring nodes.

[0122] In this invention, the feature update of the GNN layer uses the GELU function, which is smoother and has a more stable gradient than activation functions such as ReLU. It can effectively alleviate the gradient vanishing problem in deep networks and balance expressive power and computational efficiency.

[0123] The formula for GELU is:

[0124]

[0125] in, This is the cumulative distribution function of the standard normal distribution.

[0126] The node representation to be predicted is concatenated with the aggregated messages of its neighbors and then fed into a small fully connected layer for updating. After layer normalization and activation by the GELU function, the updated node representation is obtained.

[0127] The feature update formula for the GNN layer is:

[0128]

[0129] Unlike graph neural networks with complex structures such as graph attention networks, the graph neural network layer of this invention completes the aggregation and updating of node features through only a single fully connected layer, which effectively simplifies the model parameter scale and greatly improves the training efficiency and inference speed of the model. This is the feature vector of the central node after being updated by the graph neural network. This vector is used to perform the final furnace condition classification prediction of the central node.

[0130] Predict various furnace conditions:

[0131]

[0132] in, For logarithmic odds, The predicted probability for each furnace condition. The final classification result is obtained by taking the updated feature vector of the central node of the graph neural network as input and passing it through two fully connected layers.

[0133] In the semi-supervised training phase, a loss function is constructed by combining supervised loss, Laplace smoothing loss, and neighbor consistency loss for backpropagation to update parameters, fully leveraging the value of the large amount of unlabeled furnace condition data in the memory. The supervised loss function is as follows:

[0134]

[0135] Here, S represents the number of labeled training samples. This loss function is the classic cross-entropy loss in neural network classification problems, making full use of labeled data information for calculation. Cross-entropy loss plays a central role in supervised learning frameworks, serving as the main driving force for the model to learn target category information. In node classification tasks, cross-entropy loss directly measures the difference between the model's prediction and the true label, guiding the optimization direction of model parameters by minimizing this difference.

[0136] The Laplace loss function is:

[0137]

[0138] Here, N represents the number of unlabeled training samples. This loss function utilizes a large amount of unlabeled data. In graph neural networks, a fundamental assumption is the homogeneity assumption, meaning that connected nodes tend to have similar attributes or category labels. The main function of the Laplacian loss is to implement smoothness constraints through the graph Laplacian operator, forcing connected nodes to have similar feature representations, which helps preserve the local structural information of the graph.

[0139] Graph structures can be viewed as manifold representations of data in high-dimensional space, and Laplace loss can capture the smoothness of such manifold structures. By forcing feature similarity among adjacent nodes, the model can learn the inherent geometric structure of the data.

[0140] By using the Laplace function, the model can deeply explore the spatial distribution of unlabeled data and make full use of the information hidden in a large amount of unlabeled data.

[0141] The consistency loss function is:

[0142]

[0143] in, Let i be the classification probability distribution of node i itself. Let be the classification probability distribution of neighbor sample j. The design of the consistency loss function is based on the consistency regularization theory, which has become a standard technique in the field of semi-supervised learning. When labeled data is scarce, the model can utilize a large amount of unlabeled data to learn better feature representations through consistency constraints. The core idea of ​​consistency regularization in this invention is that the model should produce similar outputs for similar inputs; that is, after injecting noise into the input data or performing transformations, the model output should remain unchanged.

[0144] Laplace loss focuses on the structural characteristics and detailed information of the data, while consistency loss focuses on the stability of the model output. Together, they deeply explore the information hidden in unlabeled data.

[0145] The model's total loss function is:

[0146]

[0147] in, For the total loss, For cross-entropy loss, For Laplace's loss, This results in a loss of consistency. The weights of the Laplace loss, The weights for consistency loss are all set to 0.1.

[0148] The total loss is calculated by weighted summation of three loss functions, and the model update batch size is set to 64. Gradients are propagated back from the total loss layer by layer to the classifier, GNN information aggregation layer, cross-attention fusion module, and lightweight CNN and temporal Transformer encoder. After gradient calculation, gradient norm is constrained by gradient clipping to prevent gradient explosion, and then gradient descent is performed to update parameters by decaying the AdamW optimizer. During training, supervised signals are the primary driver, and unsupervised graph structure regularization is used as an auxiliary method to achieve optimization of multimodal deep network parameters and convergence of streaming learning.

[0149] As the memory bank continuously accumulates furnace condition samples, the furnace condition characterization and prediction capabilities of this invention continue to evolve, ultimately achieving real-time and stable identification of metallurgical furnace conditions in industrial scenarios.

[0150] The streaming multimodal graph neural network recognition method constructed in this embodiment has achieved excellent results in the task of metallurgical furnace condition recognition, and is suitable for the actual scenarios of high temperature interference, scarce data labeling, and streaming incremental acquisition in the metallurgical industry.

[0151] This embodiment relies on a semi-supervised learning framework, maintaining high furnace condition recognition accuracy even under stringent conditions where only 20% of the samples are labeled and 80% are unlabeled data. It employs a lightweight network architecture, significantly reducing memory usage and enabling faster inference speeds, allowing direct deployment on edge devices in metallurgical fields. Supporting streaming incremental inference, the memory uses a first-in-first-out (FIFO) strategy for dynamic updates, enabling the recognition of newly added furnace condition samples. This ensures recognition accuracy while meeting the engineering requirements for real-time furnace condition assessment in metallurgical furnaces.

[0152] In summary, this embodiment achieves high accuracy and low resource consumption for identifying the condition of metallurgical furnaces in scenarios with a small number of tags.

[0153] Example 2:

[0154] This embodiment combines the methods described in the previous embodiments, focusing on a 2650m³ blast furnace in a domestic ironmaking plant. The data sample used includes time-series variable data and furnace top burden images within the same time period. After data preprocessing, the shape of the time-series variable data is (38987, 35, 1); the corresponding burden image data shape is (38987, 256, 256). There are eight categories in total, with category labels 0-7 representing normal furnace conditions, pipe phenomena, suspended burden, slipping burden, collapse burden, data unavailable, ore distribution, and coke distribution, respectively. Abnormal furnace conditions in industry are categorized as 1-4. This invention uses 80% of the data as the training set, of which only 20% has classification labels (the training set contains 31189 data entries, of which 6237 are labeled). The training set is input into the proposed metallurgical furnace condition identification method based on streaming multimodal graph neural networks to conduct furnace condition classification and identification experiments. Figure 2 The results demonstrate the recognition performance of this invention on the test set. The confusion matrix shows that the average accuracy rate for classifying the eight furnace conditions is 79.60%. This invention is extremely accurate in predicting the four abnormal furnace conditions (1-4), demonstrating significant industrial value and providing reliable guidance for blast furnace operation.

[0155] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any modifications or equivalent changes made based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A method for identifying the condition of metallurgical furnaces and kilns based on streaming multimodal graph neural networks, characterized in that, Includes the following steps: S1. Use a convolutional neural network to extract image features of the furnace material surface, and use a multi-layer Transformer encoder to extract time-series data features of the furnace, forming image feature vectors and time-series feature vectors; S2. Perform bidirectional cross-attention fusion on the image feature vector and the temporal feature vector to generate a unified fused feature; S3. Construct a model memory library to store the extracted image feature vectors and temporal feature vectors; S4. Using the bimodal features of the current sample as the query vector, calculate the bimodal similarity with the features of all historical samples in the memory bank, select the TopK neighbor nodes based on the average similarity, complete the edge weight fusion through modal weight coefficients, and construct a dynamic topology graph. S5. Lightweight graph neural networks are used to update the fusion features and identify the conditions of each furnace.

2. The method for identifying the furnace condition of a metallurgical furnace based on a streaming multimodal graph neural network according to claim 1, characterized in that, Step S1 specifically includes: Image features of the furnace material surface are extracted using a deep separable convolutional neural network, and the corresponding formula is: in, Given an input image matrix, The image features are extracted by a depthwise separable convolutional neural network, with dimension D being 64. By extracting the long-short-term dependencies of time series parameters through multiple Transformers, the characteristics of time series data are obtained. The corresponding formula is: in, This is a time-series data vector with 35 dimensions. The extracted time-series data features have a dimension D of 64.

3. The method for identifying the furnace condition of a metallurgical furnace based on a streaming multimodal graph neural network according to claim 1, characterized in that, Step S2 specifically includes: S21. Let the visible light image features output by the encoder for the i-th sample be... The time series characteristics are Both dimensions are aligned to the same semantic space; S22. Cross-attention guided by temporal features uses temporal features as queries and image features as keys and values. It mines the attention relationships between temporal features and image features through a multi-head attention mechanism. The calculation formula is: The query, key, and value are obtained by linear projection of their corresponding features: , , Let be the time-series query matrix, image key matrix, and image value matrix of the i-th sample, respectively. , , Let be the learnable projection matrix of the i-th sample. , , For bias terms, Temporal enhancement features that fuse image information under the guidance of temporal features; S23. Image feature-guided cross-attention uses image features as queries and temporal features as keys and values ​​to reverse-engineer the attention relationship between image features and temporal features. The calculation formula is as follows: Among them, Query, Key, and Value are obtained by linear projection of the corresponding features: , , The image query, time series key matrix, and time series value matrix for the i-th sample are respectively. , , Let be the learnable projection matrix of the i-th sample. , , For bias terms, Image enhancement features that integrate temporal information and are guided by image features; S24. Attention output is processed via residual connection and layer normalization: in, For the initial fusion of image features, Features of the time series data for initial fusion; S25. Adaptive fusion is then achieved through a gating mechanism. The gating weight calculation formula is as follows: Image features, temporal features, and fused temporal features of the furnace are concatenated and input into a fully connected layer. ReLU activation is applied, and finally, the layer is normalized using the Softmax function to obtain a weight vector of temporal data and visible light images. ; S26. The two vectors obtained from the initial fusion are weighted and added together, and then concatenated with the fusion vector obtained by directly adding the two initial vectors. After passing through a fully connected layer, a unified fusion feature that can comprehensively represent the furnace condition is finally generated. : 。 4. The method for identifying the furnace condition of a metallurgical furnace based on a streaming multimodal graph neural network according to claim 1, characterized in that, Step S4 specifically includes: S41. First, construct a graph for classifying the current furnace samples, and calculate the cosine similarity between the image and the temporal modality: in, The cosine similarity of the embeddings of each graph in the memory bank. The cosine similarity of the embeddings for each time-series data in the memory bank. The image feature vector of the center node, The time-series data feature vector of the central node. The image feature vectors of the nodes in the memory bank. This represents the temporal data feature vector of the nodes in the memory bank; S42. For the current sample, use the average similarity between the two modalities as the retrieval score, select the top K neighbor indices with the highest scores from the memory bank, and connect the samples corresponding to these indices to form a KNN graph. S43. After the candidate neighbors are determined, the adaptive edge weight fusion module is entered: for each edge, the image similarity and temporal similarity between the center node and its neighbors are extracted simultaneously, and then they are used as inputs to automatically learn the gate control coefficients through MLP, and the fused similarity is output by combining the product enhancement term. S44. Softmax normalization is applied to the fusion similarity to obtain the final edge weights: in, The temperature coefficient is taken as 1.

0. For bimodal overall similarity, This is the final side weight.

5. The method for identifying the furnace condition of a metallurgical furnace based on a streaming multimodal graph neural network according to claim 1, characterized in that, The method for updating the fused features using a lightweight graph neural network in step S5 is as follows: The lightweight graph neural network contains only a single fully connected layer and uses the GELU activation function. The features of neighbor nodes are represented by simple averaging. The features of the center node are concatenated with the aggregated messages of its neighbors and then updated by the fully connected layer.

6. The method for identifying the furnace condition of a metallurgical furnace based on a streaming multimodal graph neural network according to claim 1, characterized in that, The identification method adopts a semi-supervised learning strategy, constructs a loss function that includes cross-entropy loss, Laplacian loss and consistency loss, the model update batch size is 64, and the AdamW optimizer is used for parameter update.

Images