Metallurgical furnace working condition intelligent identification method fusing multi-modal data

Abstract

The invention provides a metallurgical furnace working condition intelligent identification method fusing multi-modal data. A double-branch multi-level residual structure feature encoder is designed, and metallurgical furnace process variable data features and image data features are fully extracted. Aggregation and separation among modal features are carried out through a comparative learning method, feature semantic alignment is realized, and feature distribution is optimized. Under the constraint of orthogonal loss, a feature decomposer is utilized to effectively separate out inter-modal correlation features and modal private features, and an information bottleneck theory is utilized to promote redundancy removal of key correlation features of process variable data and image data. A multi-level and multi-dimensional feature fusion module is constructed, single-level feature interaction fusion and inter-level feature fusion are sequentially realized under the action of bidirectional cross attention, and fusion features rich in information content are obtained. And finally, the information is sent to the classifier, so that intelligent and accurate identification of the metallurgical furnace working condition under multi-modal data input is realized.

Description

Technical Field

[0001] The present invention relates to a metallurgical furnace, and in particular to a method for intelligently identifying working conditions of a metallurgical furnace by integrating multimodal data. Background Art

[0002] Metallurgical furnaces are core thermal equipment in process industries like steel and nonferrous metals. Their operating status directly impacts product quality, energy efficiency, and production safety. In actual production, furnace systems often exhibit complex characteristics such as strong coupling and nonlinearity. Their operation is susceptible to multiple factors, including raw material fluctuations, operating conditions, and equipment status. This leads to frequent fluctuations in furnace operating conditions, which in turn can trigger a variety of abnormal operating conditions. These abnormal conditions not only reduce output efficiency and product quality but can also cause safety incidents, seriously threatening production continuity and system stability.

[0003] Currently, methods for identifying metallurgical furnace operating conditions fall into two main categories: those based on process variable data and those based on image data. The former relies on time-series process data such as temperature, pressure, and flow collected by sensors at various locations within the furnace. These methods extract abnormal patterns through feature engineering, statistical modeling, or machine learning algorithms. While these methods offer advantages such as high real-time performance and wide coverage, they remain challenging to accurately identify due to nonlinear coupling between variables, similar data distributions, and data noise. The latter, on the other hand, utilizes in-furnace monitoring image data to intuitively reflect furnace condition changes. These methods are suitable for identifying abnormal conditions with distinct visual characteristics. While these methods have achieved some success, supported by deep learning image recognition technology, they also face challenges such as limited image viewing angles and poor image quality. Furthermore, some metallurgical furnace operating condition identification methods also involve the fusion of multi-source heterogeneous data, but these methods typically rely on simple concatenation or processing of multimodal features. While simple and efficient, these methods fail to fully exploit the underlying correlations and complementary mechanisms between modalities, resulting in limited fusion effectiveness and limited improvement in model performance and generalization.

[0004] Taking into account the advantages and disadvantages of the aforementioned methods for identifying metallurgical furnace operating conditions, it can be seen that process variable data and image data are naturally complementary in describing furnace operating status: the former reflects the global dynamic changes in operating conditions, while the latter focuses on the intuitive presentation of local spatial structure and physical form. Therefore, it is crucial to establish a multimodal fusion mechanism with a reasonable structure, semantic alignment, and information complementarity, fully exploit the synergistic relationship between different modal features, and achieve dynamic interactive fusion of the two, in order to improve the accuracy of operating condition identification.

[0005] Therefore, the present invention comprehensively considers the characteristics of multi-source heterogeneous data during the operation of metallurgical furnaces and proposes a method for intelligently identifying metallurgical furnace operating conditions by integrating multimodal data. This method effectively combines metallurgical furnace process variable data with image data. To address the semantic differences and redundant information issues between heterogeneous modalities, a systematic feature extraction, semantic alignment, redundancy suppression, and interactive fusion mechanism are designed. This method breaks through the limitations of traditional single-modal methods in information utilization and expression capabilities, and overcomes the problems of existing multimodal fusion methods that are overly simple and lack deep collaborative interaction, effectively improving the accuracy of intelligent identification of metallurgical furnace operating conditions.

[0006] 1. After searching, the Chinese invention patent with publication number CN112597904A proposes a method for identifying and classifying blast furnace charge surface images, which includes using a denoising convolutional neural network algorithm to denoise the charge surface data set images; dividing the denoised charge surface data set into a training set image and a test set image, both of which contain 7 categories; using a directional gradient histogram algorithm to extract features from each category of the training set image, and at the same time using a directional gradient histogram algorithm to extract features from each category of the test set image; using a multi-classification support vector machine algorithm to design a classifier based on the training set image features to obtain a classifier belonging to the charge surface data set image; adding the test set image features to the trained classifier; classifying the test set images and calculating the recognition accuracy; the method for identifying and classifying blast furnace charge surface images of the present invention has a good removal effect on the stripe noise in the charge surface image, and has the advantages of simple calculation and high speed, and can quickly and accurately realize the recognition and classification of the furnace conditions in the blast furnace charge surface image.

[0007] The above patent uses a denoising convolutional neural network to pre-process the blast furnace charge surface image, extracts image features through the histogram of oriented gradients (HOG), and uses a support vector machine (SVM) to classify and identify seven typical furnace conditions. This method has certain effects in image denoising and feature extraction, and can improve image recognition accuracy and efficiency. However, this invention only classifies working conditions based on static image data, does not integrate multi-source information such as process variables, lacks an information interaction mechanism between modalities, and has limitations in the model's expressive power and working condition discrimination accuracy.

[0008] 2. A Chinese invention patent with publication number CN117932479A proposes a blast furnace fault diagnosis method based on a trace distance partitioning decision tree, which relates to the technical field of blast furnace fault diagnosis. The present invention obtains the required data set from the blast furnace production process data acquisition system, obtains historical production process data from the enterprise server, and uses the data to establish a blast furnace fault diagnosis model based on a trace distance partitioning decision tree; then, the real-time sampling value at each sampling point is obtained from the blast furnace production process data acquisition system to obtain the blast furnace online production process data, and the online data is passed to the blast furnace fault diagnosis model, thereby outputting the online diagnosis results of the current blast furnace condition to the on-site operator; in addition, the blast furnace production process data acquisition system will store all newly sampled samples in the enterprise server. When the number of new samples meets the requirements, the blast furnace fault diagnosis model will be retrained in combination with the historical data and the new data to ensure that the model can track and diagnose the latest operating conditions of the blast furnace.

[0009] The aforementioned patent utilizes historical process variable data from blast furnaces to construct a decision tree model based on the trace distance partitioning criterion for fault diagnosis of blast furnace operating conditions. By defining a trace distance function for feature partitioning, the model partitioning efficiency and diagnostic accuracy are improved, and the model can be updated in conjunction with new samples. However, this invention relies solely on process variable data for modeling and lacks the use of external sensory information such as blast furnace images. Furthermore, the model is a static structure that makes it difficult to capture deep semantic associations between multi-source information, and its ability to identify complex dynamic operating conditions is limited. Summary of the Invention

[0010] In order to solve the above technical problems, the present invention proposes an intelligent identification method for metallurgical furnace operating conditions by integrating multimodal data, which can effectively realize semantic alignment, information redundancy removal and feature interactive fusion between metallurgical furnace process variable data and image data, thereby realizing accurate operating condition identification. From the perspective of multimodal heterogeneous feature extraction and fusion, it provides new ideas for the research on metallurgical furnace operating condition identification methods under multi-source heterogeneous data.

[0011] To achieve the above object, the technical solution adopted by the present invention is:

[0012] A method for intelligently identifying metallurgical furnace operating conditions by integrating multimodal data, characterized in that it comprises the following steps:

[0013] S1. Construct a dual-branch multi-level residual structure feature encoder to extract features and transfer semantic information at different semantic levels for process variable data and image data in the metallurgical furnace process.

[0014] S2. Design a multi-level semantic alignment module based on contrastive learning to aggregate and separate process variable data and image data features in the feature space, align multi-level multimodal features in the semantic space, and optimize feature distribution;

[0015] S3. Design a decomposer network and information redundancy removal module to decouple the two types of modal features, respectively decomposing inter-modal correlation features and modality-private features to achieve more discriminative feature expression. Then, redundancy is removed from the obtained inter-modal correlation features to further obtain effective features.

[0016] S4. Construct a multi-level and multi-dimensional feature fusion module to collaboratively fuse the inter-modal correlation features after redundancy removal with the private features of each modality within the same layer and across layers, fully exploring the complementarity of information at different levels and from different sources.

[0017] As the preferred technical solution of the present invention:

[0018] In step S1, during the metallurgical furnace production process, the furnace operation status is reflected by multi-source heterogeneous data. For two types of heterogeneous modal data: process variable data and image data, the process variable data is filtered to obtain variable data with low data distribution repetition. Then, the multi-level initial semantic representation of the process variable data is realized through the TrasnformerEncoder network with a multi-level residual structure. The image data is extracted from features at two scales and multiple levels using CrossVIT.

[0019] The specific steps include:

[0020] S11: Let the process variable data, image data and corresponding working condition labels at the same time be a data sample:

[0021] X={x 1 ,x 2 ,…,x k}={[v 1 ,i 1 ,l 1 ],[v 2 ,i 2 ,l 2 ],…,[v k ,i k ,l k ]} (1)

[0022] Where k = 1, 2, ..., N is the sample number, x = [v, i, l] represents a sample set, d is the number of variables contained in the process variable data, h×w is the image size of the material surface, Indicates the label value of the corresponding working condition;

[0023] S12: For any process variable data v in a data sample, v = [v 1 ,v 2 ,…,vk ] T =[v1,v2,…v p ,…,v q ,…,v d ], first calculate the Spearman rank correlation matrix between variables

[0024] R pq =ρ s (v p ,v q ) (2)

[0025] where ρ s Indicates Spearman correlation coefficient calculation, v p and v q Represent all data of corresponding variables in process variable data, set the correlation threshold τ, and construct a model that satisfies the correlation between any variable pair p,q∈{1,2,…,d} does not exceed τ:

[0026] |ρ s (v p ,v q )|≤τ (3)

[0027] According to the heuristic greedy strategy, p is retained and q is eliminated to obtain the maximum variable subset S = {1, 2, ..., s} that satisfies formula (3). Variables with too high data distribution similarity are eliminated one by one. The process variable data after elimination is

[0028] S13: The process variable data after elimination After nonlinear transformation projection, we get Then it is sent to a multi-level residual structure network composed of TransformerEncoder, and the residual connection is the input of this level. With output Make connections as input to the next level of hierarchy

[0029]

[0030] Where TE represents a single-level TransformerEncoder block, j = 1, 2, 3, represents three levels of features, Residual connections only exist in the last two levels (j=2,3):

[0031]

[0032] in Finally, we get a multi-level feature

[0033] S14: For image data i k , and divide them into two pre-set patch sizes p s ×p s and p l ×p l After splitting, the two-sized patch sets are sent to the multi-level CrossVIT for feature extraction:

[0034]

[0035] in The residual connection structure contained in this branch is:

[0036]

[0037] in Finally, a multi-level feature of the material surface image is obtained

[0038] As the preferred technical solution of the present invention:

[0039] In step S2, a multi-level semantic alignment module based on contrastive learning is designed to construct a new sample pair partitioning method. Under the constraint of adaptive weight contrast loss, consistent semantic relationship mapping in the shared feature space is achieved, as follows:

[0040] S21: First, the process variable data features and image data features obtained by the pre-order multi-level residual structure feature encoder are described as follows: where k v 、k i Indicates the sample number, the corresponding label is The sample pairs are divided into positive sample pairs, pseudo positive sample pairs and negative sample pairs, as follows:

[0041]

[0042] From the perspective of similarity, the similarities of positive sample pairs, weak positive sample pairs, and negative sample pairs should decrease step by step;

[0043] S22: For input features First calculate the normalized similarity matrix:

[0044]

[0045] Where p∈{1,2,…,b}, q∈{1,2,…,b} represents the sample number, b is the batch size during training, and <·> represents the similarity calculation.

[0046] The indexed similarity is:

[0047]

[0048] Where τ is the temperature coefficient, which is used to control the distinguishing ability of the similarity distribution.

[0049] The overall contrast loss function is:

[0050]

[0051] in, They represent the number of positive sample pairs, weak positive sample pairs, and negative sample pairs contained in a batch size, and the weights corresponding to the three sample losses are adaptively changed:

[0052] ω pos =1.5+0.5α t

[0053] ω weak =0.5+0.3α t

[0054] ω neg =0.2(1-α t )+ε

[0055]

[0056] Among them, t represents the current epoch of the training process, T represents the total epoch of the training process, and ε is a minimum value to avoid ω neg is zero.

[0057] As the preferred technical solution of the present invention:

[0058] In step S3, based on the semantic alignment of the two modal features, feature decomposition and redundancy removal are performed. The semantically aligned multi-level features are converted into multi-level inter-modal correlation features and modality-private features through a feature decomposer. The multi-level inter-modal correlation features achieve a more effective feature representation under the guidance of the information bottleneck theory, as follows:

[0059] S31: Different modal features contain important complementary information that other modalities do not have. The corresponding modal features are decomposed into inter-modal related features and modality-specific features:

[0060]

[0061] Where FD represents the feature decomposer, m=i,v represents the corresponding mode, represents the corresponding modal features of the i-th (i=1,2,3) level, θ de are the model parameters of the eigendecomposer, is the inter-modal correlation feature of the corresponding mode at the corresponding level, is the private feature of the corresponding modality at the corresponding level;

[0062] S32: In order to ensure the inter-modal correlation features decomposed by the multi-level dual-branch feature decomposer and There is high similarity between them, and similarity loss is introduced to guide the learning of inter-modal related features:

[0063]

[0064] Among them, b is the batch size during training,

[0065] In addition, in order to ensure that inter-modality related features and modality private features are strictly separated to achieve feature decoupling, an orthogonal loss is proposed:

[0066]

[0067] S33: The core goal of the information bottleneck theory is to maximize the correlation between the intermediate feature z and the output feature y, and minimize the redundant information between the input feature x and the intermediate feature z. The optimization goal is:

[0068]

[0069] Where θ is the Encoder-Decoder network structure parameter corresponding to the information bottleneck theory module, x is the inter-modal correlation feature obtained by the feature decomposer, z is the feature after de-redundancy, and y is the reconstructed feature used to measure the important information contained in z. The input feature x is:

[0070]

[0071] where x i represents the input features of the i-th level, and concat(·) represents the feature concatenation operation;

[0072] S34: Make a variational lower bound approximation for the IB objective. For the objective formula (16), I(x; z) can be minimized and I(y; z) can be maximized to achieve the optimization goal.

[0073] First, a learnable Gaussian mixture prior p(z) is constructed, and the KL divergence between p(z) and the encoder output q(z|x) is used to constrain the information bottleneck feature z and minimize the optimization target I(x,z).

[0074] For the reference prior distribution, a learnable Gaussian mixture prior p(z) is used, which contains K mixture components:

[0075]

[0076] where π k is the weight parameter of the Kth component, μ k 、 are the corresponding mean and variance,

[0077] For each sample, sample z from q(z|x) and use Gaussian distribution to calculate the corresponding conditional distribution density function element by element:

[0078]

[0079] Since the expectation of q(z|x) cannot be calculated analytically, a numerical method is used to estimate this expectation, and the Monte Carlo estimation is used to estimate the KL divergence:

[0080]

[0081] Where L is the number of samples, usually set to L = 1;

[0082] S35: For the optimization target maxI(y,z), the reconstruction loss To ensure that the model's intermediate features z retain enough information to reconstruct the original features x,

[0083]

[0084] In addition, in order to maximize the removal of redundant information, the information irrelevant to the task in the intermediate representation z is removed, and the z-oriented task loss is introduced, which indirectly ensures that z contains information related to the final working condition classification task.

[0085]

[0086] As the preferred technical solution of the present invention:

[0087] In step S4, through the dynamic interactive fusion method of the two modal features of process variable data and image data at different levels, under the action of the bidirectional cross-attention mechanism, the inter-modal related features and private features fully interact with each other, and the features of each level are also interactively fused between levels through the bidirectional attention mechanism. The final fusion feature is obtained through the adjustment of global attention and gated fusion units. The specific steps are as follows:

[0088] S41: Perform bidirectional cross-attention processing on the single-level process variable private features, image data private features, and inter-modal correlation features after redundancy removal. After stacking, residual connection, and normalization, preliminary feature interaction is achieved.

[0089]

[0090] Where BCA represents the bidirectional cross attention module;

[0091] S42: For multiple levels The multi-level preliminary fusion features are processed with two-way cross attention to perform cross-level interactive fusion:

[0092]

[0093] Then all Perform splicing and calculate global fusion features through multi-head attention to improve the global representation ability of fusion features:

[0094]

[0095] in e represents the embedding dimension hyperparameter, MHA represents the multi-head attention mechanism;

[0096] S43: For In the second dimension, we perform grading, copying, and splicing, and then input the corresponding gating network to obtain the corresponding gating fusion weights:

[0097]

[0098] in Then perform weighted summation and dimension transformation to obtain the final fusion feature with rich information content:

[0099]

[0100] in

[0101] S44: Under the constraints of the cross entropy loss function:

[0102]

[0103] where ω l is the weight coefficient of category l, p k,l is the probability of the true label of the model predictor, fusion feature Enter the MLP classifier with category-aware attention mechanism to achieve intelligent and accurate identification of metallurgical furnace operating conditions under multimodal data input.

[0104] Compared with the prior art, the present invention has the following beneficial effects:

[0105] In order to effectively extract features from process variable data and image data of metallurgical furnaces and preserve semantic information, a dual-branch multi-level residual structure feature encoder was designed to achieve effective extraction of semantic features from two modalities.

[0106] Aiming at the problem that the heterogeneity between process variable data and image data makes it difficult to model and align the semantic association between modal features, a multi-level semantic alignment method based on contrastive learning is used to align the two multi-level modal features obtained in the previous sequence in the semantic space dimension.

[0107] In response to the information redundancy phenomenon caused by the similar distribution of process variable data and the similar information between two modalities, this application constructs a decomposer network and a redundant information removal module to obtain effective inter-modal correlation features and rich modal private features.

[0108] In order to achieve fully effective and interactively correlated feature fusion, this application constructs a multi-level and multi-dimensional feature fusion module, and uses a bidirectional cross-attention mechanism to dynamically and interactively fuse features from different features and multi-level perspectives, thereby increasing the amount of information contained in the final fused features and providing subsequent classifiers with higher quality and more information-rich features.

[0109] The method presented in this paper can accurately identify five types of blast furnace operating conditions, resolving the issues of inaccurate blast furnace operating condition identification caused by information limitations in single-modal data, as well as unstable blast furnace operating condition identification caused by insufficient multimodal data fusion and information inefficiency. It provides accurate and reliable identification results for on-site production staff and offers new insights into multimodal fusion and operating condition identification in metallurgical furnace scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0110] Figure 1 It is a step diagram of the intelligent identification method of metallurgical furnace working conditions by integrating multimodal data;

[0111] Figure 2 It is the way to divide sample pairs in the contrastive learning module;

[0112] Figure 3 is the training loss graph;

[0113] Figure 4 It is the feature distribution map in t-SNE space;

[0114] Figure 5 It is the confusion matrix diagram of the recognition results. DETAILED DESCRIPTION

[0115] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0116] like Figure 1-2 As shown in FIG, the present invention proposes an intelligent identification method for metallurgical furnace working conditions by integrating multimodal data, comprising the following steps:

[0117] S1. Construct a dual-branch multi-level residual structure feature encoder to extract features and transfer semantic information at different semantic levels for process variable data and image data in the metallurgical furnace process.

[0118] In step S1, during the metallurgical furnace production process, the furnace operating status is often reflected by multi-source heterogeneous data. The present invention targets two types of heterogeneous modal data: process variable data and image data. The process variable data is filtered by a variable filter to obtain variable data with low data distribution repetition. Then, a multi-level residual structured TrasnformerEncoder network is used to achieve a multi-level initial semantic representation of the process variable data. The image data is extracted from features at multiple levels at two scales using CrossVIT. Specifically, the following steps are included:

[0119] S11: Let the process variable data, image data and corresponding working condition labels at the same time be a data sample:

[0120] X={x 1 ,x 2 ,…,x k}={[v 1 ,i 1 ,l 1 ],[v 2 ,i 2 ,l 2 ],…,[v k ,i k ,l k ]} (30)

[0121] Where k = 1, 2, ..., N is the sample number, x = [v, i, l] represents a sample set, d is the number of variables contained in the process variable data, h×w is the image size of the material surface, Indicates the label value of the corresponding working condition.

[0122] S12: For any process variable data v in a data sample, v = [v 1 ,v 2 ,…,v k ] T =[v1,v2,…v p ,…,v q ,…,v d ], first calculate the Spearman rank correlation matrix between variables

[0123] R pq =ρ s (v p ,v q ) (31)

[0124] where ρ s Indicates Spearman correlation coefficient calculation, v p and v q Represent all data of corresponding variables in process variable data, set the correlation threshold τ, and construct a model that satisfies the correlation between any variable pair p,q∈{1,2,…,d} does not exceed τ:

[0125] |ρ s (v p ,v q )|≤τ (32)

[0126] According to the heuristic greedy strategy, p is retained and q is eliminated to obtain the maximum variable subset S = {1, 2, ..., s} that satisfies formula (3). Variables with too high data distribution similarity are eliminated one by one. The process variable data after elimination is

[0127] S13: The process variable data after elimination After nonlinear transformation projection, we get Then it is sent to a multi-level residual structure network composed of TransformerEncoder, and the residual connection is the input of this level. With output Make connections as input to the next level of hierarchy

[0128]

[0129] Where TE represents a single-level TransformerEncoder block, j = 1, 2, 3, represents three levels of features, Residual connections only exist in the last two levels (j=2,3):

[0130]

[0131] in Finally, we get a multi-level feature

[0132] S14: For image data i k , and divide them into two pre-set patch sizes p s ×p s and p l ×p l After splitting, the two-sized patch sets are sent to the multi-level CrossVIT for feature extraction:

[0133]

[0134] in Similar to S13, the residual connection structure contained in this branch is:

[0135]

[0136] in Finally, a multi-level feature of the material surface image is obtained

[0137] S2. Design a multi-level semantic alignment module based on contrastive learning to aggregate and separate process variable data and image data features in the feature space, align multi-level multimodal features in the semantic space, and optimize feature distribution;

[0138] In step S2, before performing information redundancy removal and interactive fusion, the present invention addresses semantic confusion caused by the heterogeneity of process variable and image data. Specifically, the present invention designs a multi-level semantic alignment module based on contrastive learning and constructs a new sample pair partitioning method. Under the constraints of an adaptive weighted contrast loss, consistent semantic relationship mapping is achieved in the shared feature space.

[0139] S21: First, the process variable data features and image data features obtained by the pre-order multi-level residual structure feature encoder are described as follows: where k v 、k i Indicates the sample number, the corresponding label is In traditional contrastive learning, sample pairs are divided only into positive and negative pairs. This approach has certain limitations for learning discriminative features in industrial scenarios. This study considers the similarity of data within the same operating conditions in industrial scenarios and proposes a new sample pair partitioning strategy: dividing sample pairs into positive pairs, pseudo-positive pairs, and negative pairs.

[0140] Specifically, for the one-to-one corresponding process variable data features and image data features in the original feature pair, the present invention regards them as a positive sample pair. For the material surface image features and process variable features that are not strictly corresponding but have the same labels, and there is a certain similarity correlation between the two, the present invention regards them as weak positive sample pairs, and the rest are regarded as negative sample pairs.

[0141]

[0142] From the perspective of similarity, the similarities of positive sample pairs, weak positive sample pairs, and negative sample pairs should decrease step by step. Figure 2 shown.

[0143] S22: For input features First calculate the normalized similarity matrix:

[0144]

[0145] Where p∈{1,2,…,b} and q∈{1,2,…,b} represent the sample numbers, b is the batch size during training, and <·> represents the similarity calculation, which is generally the cosine similarity metric.

[0146] The indexed similarity is:

[0147]

[0148] Where τ is the temperature coefficient, which is used to control the distinguishing ability of the similarity distribution.

[0149] The overall contrast loss function is:

[0150]

[0151] in, They represent the number of positive sample pairs, weak positive sample pairs, and negative sample pairs contained in a batch size, respectively. The weights corresponding to the three sample losses are adaptive, making it convenient for the model to give different attention to sample pairs at different stages of training:

[0152] ω pos =1.5+0.5α t

[0153] ω weak =0.5+0.3α t

[0154] ω neg =0.2(1-α t )+ε

[0155]

[0156] Among them, t represents the current epoch of the training process, T represents the total epoch of the training process, and ε is a minimum value to avoid ω neg is zero.

[0157] S3. Design a decomposer network and information redundancy removal module to decouple the two types of modal features, respectively decomposing inter-modal correlation features and modality-private features to achieve more discriminative feature expression. Then, redundancy is removed from the obtained inter-modal correlation features to further obtain effective features.

[0158] In step S3, to effectively address the information redundancy issue between process variable data and image data, the present invention performs feature decomposition and redundancy removal based on semantic alignment of the two modal features. The semantically aligned multi-level features are transformed into multi-level inter-modality correlation features and modality-specific features through a feature decomposer. These multi-level inter-modality correlation features, guided by the information bottleneck theory, enable more effective feature representation.

[0159] S31: Different modal features contain important complementary information that other modalities do not have. The present invention decomposes the corresponding modal features into inter-modal related features and modality-specific features:

[0160]

[0161] Where FD represents the feature decomposer, m=i, v represents the corresponding mode, represents the corresponding modal features of the i-th (i=1,2,3) level, θ de are the model parameters of the eigendecomposer, is the inter-modal correlation feature of the corresponding mode at the corresponding level, It is the private feature of the corresponding level and mode.

[0162] S32: In order to ensure the inter-modal correlation features decomposed by the multi-level dual-branch feature decomposer and There is high similarity between them. This paper introduces similarity loss to guide the learning of inter-modal related features:

[0163]

[0164] Among them, b is the batch size during training.

[0165] In addition, in order to ensure that inter-modality related features and modality private features are strictly separated to achieve feature decoupling, the present invention proposes an orthogonal loss:

[0166]

[0167] S33: The core goal of the information bottleneck theory is to maximize the correlation between the intermediate feature z and the output feature y, and minimize the redundant information between the input feature x and the intermediate feature z. The optimization goal is:

[0168]

[0169] Where θ is the encoder-decoder network structure parameter corresponding to the information bottleneck theory module, x is the inter-modal correlation feature obtained by the feature decomposer in the present invention, z is the feature after de-redundancy, and y is the reconstructed feature used to measure the important information contained in z. In the present invention, the input feature x is:

[0170]

[0171] where x i represents the input features of the i-th level, and concat(·) represents the feature concatenation operation.

[0172] S34: In deep learning models, the distribution of the intermediate feature z is often unknown and cannot be explicitly expressed, making it difficult to integrate or differentiate it. Furthermore, high-dimensional density estimation is inherently unstable, and gradients cannot be effectively propagated. This makes direct optimization of mutual information infeasible in deep networks. A common approach is to perform a variational lower bound approximation on the IB objective. For the objective (16), the optimization goal can be achieved by minimizing I(x; z) and maximizing I(y; z).

[0173] For the optimization target minI(x,z), the complexity of the potential representation is limited by the KL (Kullback-Leibler) divergence, and randomness is introduced using variational inference to limit the flow of information. In this study, we face the problem of unknown prior distribution in KL divergence, and the use of standard normal distribution cannot well fit the feature distribution of intermediate features in multimodal recognition tasks, so the present invention attempts to use mixed Gaussian distribution to replace the standard normal distribution. Monte Carlo approximation is used to calculate KL divergence. First, a learnable Gaussian mixture prior p(z) is constructed, and the KL divergence between p(z) and the encoder output q(z|x) is used to constrain the information bottleneck feature z and minimize the optimization target I(x,z).

[0174] For the reference prior distribution, this study uses a learnable Gaussian mixture prior p(z) with K mixture components:

[0175]

[0176] where π k is the weight parameter of the Kth component, μ k 、 are the corresponding mean and variance respectively.

[0177] For each sample, sample z from q(z|x) and use Gaussian distribution to calculate the corresponding conditional distribution density function element by element:

[0178]

[0179] Since the expectation of q(z|x) cannot be calculated analytically (a closed-form solution cannot be found), a numerical method is used to estimate this expectation (sampling estimation), and the Monte Carlo estimation is used to estimate the KL divergence:

[0180]

[0181] Where L is the number of samples, usually set to L=1.

[0182] S35: For the optimization target maxI(y,z), similar to the optimization target minI(x,z), it is difficult to optimize it directly. In the present invention, the reconstruction loss To ensure that the model's intermediate features z retain enough information to reconstruct the original features x.

[0183]

[0184] In addition, in order to maximize the removal of redundant information, the information irrelevant to the task in the intermediate representation z is removed. The present invention introduces a z-oriented task loss, thereby indirectly ensuring that z contains information related to the final working condition classification task.

[0185]

[0186] S4. Construct a multi-level and multi-dimensional feature fusion module to collaboratively fuse the inter-modal correlation features after redundancy removal with the private features of each modality within the same layer and across layers, fully exploring the complementarity of information at different levels and from different sources.

[0187] In step S4, the present invention proposes a method for dynamic interactive fusion of two modal features, process variable data and image data, at different levels. Under the action of the bidirectional cross-attention mechanism, the inter-modal related features and private features fully interact with each other, and the features at each level are also interactively fused through the bidirectional attention mechanism. Through the adjustment of global attention and gated fusion units, the final fused features are obtained, providing information-rich and effective features for subsequent classifiers, thereby realizing intelligent and accurate identification of metallurgical furnace working conditions. The specific steps are as follows:

[0188] S41: The single-level process variable private features, image data private features, and inter-modal correlation features after redundancy removal are subjected to bidirectional cross-attention processing. After stacking, residual connection, and normalization, preliminary feature interaction is achieved.

[0189]

[0190] Where BCA stands for Bidirectional Cross Attention module.

[0191] S42: For multiple levels Similar to S41, the multi-level preliminary fusion features are subjected to bidirectional cross-attention processing to perform cross-level interactive fusion:

[0192]

[0193] Then all Perform splicing and calculate global fusion features through multi-head attention to improve the global representation ability of fusion features:

[0194]

[0195] in e represents the embedding dimension hyperparameter, and MHA represents the multi-head attention mechanism (MultiHeadAttention).

[0196] S43: For In the second dimension, we perform grading, copying, and splicing, and then input the corresponding gating network to obtain the corresponding gating fusion weights:

[0197]

[0198] in Then perform weighted summation and dimension transformation to obtain the final fusion feature with rich information content:

[0199]

[0200] in

[0201] S44: Under the constraints of the cross entropy loss function:

[0202]

[0203] where ω l is the weight coefficient of category l, p k,l is the probability of the model predictor's true label. Enter the MLP classifier with category-aware attention mechanism to achieve intelligent and accurate identification of metallurgical furnace operating conditions under multimodal data input.

[0204] Based on the above method,

[0205] The dual-branch multi-level residual structure feature encoder designed in this invention fully extracts the process variable data and image data features of the metallurgical furnace. The residual structure design also ensures the effective extraction of modal features and the preservation of semantic information.

[0206] The present invention designs a multi-level semantic alignment module based on contrastive learning. By proposing a new sample pair construction strategy and a customized contrastive loss function, it guides the model to learn modal feature representations with shared semantics in the cross-modal feature space, thereby achieving deep semantic alignment and enhanced discriminability of inter-modal features.

[0207] The present invention designs a decomposer network and an information redundancy removal module to decompose the modal features after semantic alignment into inter-modality related features and modality private features. Then, the redundant information of the two inter-modality related features is removed through the information bottleneck theory, thereby obtaining a more comprehensive and effective feature representation.

[0208] The present invention constructs a multi-level and multi-dimensional feature fusion module, which uses a bidirectional cross-attention mechanism to perform dynamic interactive fusion of features from different features and multi-level perspectives, thereby increasing the amount of information contained in the final fused features, providing high-quality features for subsequent classifiers, and realizing intelligent and accurate identification of metallurgical furnace working conditions.

[0209] The intelligent identification method for metallurgical furnace operating conditions by fusing multimodal data proposed in the present invention realizes the accurate identification of metallurgical furnace operating conditions from the perspective of multimodal heterogeneous feature extraction and fusion, providing a new technology for multimodal data fusion and intelligent identification of operating conditions in metallurgical furnace scenes.

[0210] Example:

[0211] This example is based on a domestic ironmaking plant with a 2650m 3 The blast furnace was expanded. The data samples used included process variable data and furnace top surface images within the same time period. After image enhancement and data supplementation, the process variable data dimension was 35, the total number of samples was 20,950, and the data shape was (20,950, 35); the corresponding surface image data shape was (20,950, 256, 256). This multi-source data was input into the proposed blast furnace operating condition identification method based on multi-source information collaborative interactive fusion, and operating condition classification and identification experiments were carried out. Figure 3 The loss change curve of the model during training is shown, indicating that the loss decreases rapidly, the model converges quickly, and can stably achieve better performance. Figure 4 The distribution of fused features in the t-SNE space is demonstrated. The features of different working condition categories are clearly separated and compactly clustered, which further verifies the effectiveness of the proposed multimodal feature fusion method in semantic alignment and discriminability. Figure 5The proposed method demonstrates its recognition performance on a test set. The confusion matrix shows that the average recognition accuracy for five typical blast furnace operating conditions (normal, suspended material, pipeline, sliding material, and collapsed material) reaches 96.35%, and the classification performance remains good even in the presence of class imbalance. In summary, the experimental results fully demonstrate the effectiveness of the proposed method in blast furnace operating condition recognition and provide reliable guidance for blast furnace operation.

[0212] The above description is merely a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent variation based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A method for intelligent identification of metallurgical furnace working conditions by integrating multimodal data, characterized in that: The process includes the following steps: S1. Construct a dual-branch multi-level residual structure feature encoder to extract features and transfer semantic information at different semantic levels for process variable data and image data in the metallurgical furnace process. S2. Design a multi-level semantic alignment module based on contrastive learning to aggregate and separate process variable data and image data features in the feature space, align multi-level multimodal features in the semantic space, and optimize feature distribution; S3. Design a decomposer network and information redundancy removal module to decouple the two types of modal features, respectively decomposing inter-modal correlation features and modality-private features to achieve more discriminative feature expression. Then, redundancy is removed from the obtained inter-modal correlation features to further obtain effective features. S4. Construct a multi-level and multi-dimensional feature fusion module to collaboratively fuse the inter-modal correlation features after redundancy removal with the private features of each modality within the same layer and across layers, fully exploring the complementarity of information at different levels and from different sources.

2. The method for intelligently identifying metallurgical furnace operating conditions by integrating multimodal data according to claim 1, characterized in that: In step S1, during the metallurgical furnace production process, the furnace operating status is reflected through multi-source heterogeneous data. For two types of heterogeneous modal data: process variable data and image data, the process variable data is filtered to obtain variable data with low data distribution duplication. Then, the multi-level initial semantic representation of the process variable data is achieved through the TrasnformerEncoder network with a multi-level residual structure. The image data is extracted from features at two scales and multiple levels using CrossVIT. The specific steps include the following: S11: Let the process variable data, image data and corresponding working condition labels at the same time be a data sample: X={x 1 ,x 2 ,…,x k }={[v 1 ,i 1 ,l 1 ],[v 2 ,i 2 ,l 2 ],…,[v k ,i k ,l k ]} (1) Where k = 1, 2, ..., N is the sample number, x = [v, i, l] represents a sample set, d is the number of variables contained in the process variable data, h×w is the image size of the material surface, Indicates the label value of the corresponding working condition; S12: For any process variable data v in a data sample, v = [v 1 ,v 2 ,…,v k ] T =[v1,v2,…v p ,…,v q ,…,v d ], first calculate the Spearman rank correlation matrix between variables R pq =ρ s (v p ,v q ) (2) where ρ s Indicates Spearman correlation coefficient calculation, v p and v q Represent all data of corresponding variables in process variable data, set the correlation threshold τ, and construct a model that satisfies the correlation between any variable pair p,q∈{1,2,…,d} does not exceed τ: |ρ s (v p ,v q )|≤τ (3) According to the heuristic greedy strategy, p is retained and q is eliminated to obtain the maximum variable subset S = {1, 2, ..., s} that satisfies formula (3). Variables with too high data distribution similarity are eliminated one by one. The process variable data after elimination is S13: The process variable data after elimination After nonlinear transformation projection, we get Then it is sent to a multi-level residual structure network composed of TransformerEncoder, and the residual connection is the input of this level. With output Make connections as input to the next level of hierarchy Where TE represents a single-level TransformerEncoder block, j = 1, 2, 3, representing three levels of features, Residual connections only exist in the last two levels (j=2,3): in Finally, we get a multi-level feature S14: For image data i k , and divide them into two pre-set patch sizes p s ×p s and p l ×p l After splitting, the two-sized patch sets are sent to the multi-level CrossVIT for feature extraction: in The residual connection structure contained in this branch is: in Finally, a multi-level feature of the material surface image is obtained 3. The method for intelligently identifying metallurgical furnace operating conditions by integrating multimodal data according to claim 1, characterized in that: In step S2, a multi-level semantic alignment module based on contrastive learning is designed to construct a new sample pair partitioning method. Under the constraint of adaptive weight contrast loss, consistent semantic relationship mapping in the shared feature space is achieved, as follows: S21: First, the process variable data features and image data features obtained by the pre-order multi-level residual structure feature encoder are described as follows: where k v 、k i Indicates the sample number, the corresponding label is The sample pairs are divided into positive sample pairs, pseudo positive sample pairs and negative sample pairs, as follows: From the perspective of similarity, the similarities of positive sample pairs, weak positive sample pairs, and negative sample pairs should decrease step by step; S22: For input features First calculate the normalized similarity matrix: Where p∈{1,2,…,b}, q∈{1,2,…,b} represents the sample number, b is the batch size during training, and <·> represents the similarity calculation. The indexed similarity is: Where τ is the temperature coefficient, which is used to control the distinguishing ability of the similarity distribution. The overall contrast loss function is: in, They represent the number of positive sample pairs, weak positive sample pairs, and negative sample pairs contained in a batch size, and the weights corresponding to the three sample losses are adaptively changed: oh pos =1.5+0.5a t oh weak =0.5+0.3a t oh neg =0.2(1-a t )+e Among them, t represents the current epoch of the training process, T represents the total epoch of the training process, and ε is a minimum value to avoid ω neg is zero.

4. The method for intelligently identifying metallurgical furnace operating conditions by integrating multimodal data according to claim 1, characterized in that: In step S3, based on the semantic alignment of the two modal features, feature decomposition and redundancy removal are performed. The semantically aligned multi-level features are converted into multi-level inter-modal correlation features and modality-private features through a feature decomposer. The multi-level inter-modal correlation features achieve a more effective feature representation under the guidance of the information bottleneck theory, as follows: S31: Different modal features contain important complementary information that other modalities do not have. The corresponding modal features are decomposed into inter-modal related features and modality-specific features: Where FD represents the feature decomposer, m=i,v represents the corresponding mode, represents the corresponding modal features of the i-th (i=1,2,3) level, θ de are the model parameters of the eigendecomposer, is the inter-modal correlation feature of the corresponding mode at the corresponding level, is the private feature of the corresponding modality at the corresponding level; S32: In order to ensure the inter-modal correlation features decomposed by the multi-level dual-branch feature decomposer and There is high similarity between them, and similarity loss is introduced to guide the learning of inter-modal related features: Among them, b is the batch size during training, In addition, in order to ensure that inter-modality related features and modality private features are strictly separated to achieve feature decoupling, an orthogonal loss is proposed: S33: The core goal of the information bottleneck theory is to maximize the correlation between the intermediate feature z and the output feature y, and minimize the redundant information between the input feature x and the intermediate feature z. The optimization goal is: Where θ is the Encoder-Decoder network structure parameter corresponding to the information bottleneck theory module, x is the inter-modal correlation feature obtained by the feature decomposer, z is the feature after de-redundancy, and y is the reconstructed feature used to measure the important information contained in z. The input feature x is: where x i represents the input features of the i-th level, and concat(·) represents the feature concatenation operation; S34: Make a variational lower bound approximation for the IB objective. For the objective formula (16), I(x; z) can be minimized and I(y; z) can be maximized to achieve the optimization goal. First, a learnable Gaussian mixture prior p(z) is constructed, and the KL divergence between p(z) and the encoder output q(z|x) is used to constrain the information bottleneck feature z and minimize the optimization target I(x,z). For the reference prior distribution, a learnable Gaussian mixture prior p(z) is used, which contains K mixture components: where π k is the weight parameter of the Kth component, μ k 、 are the corresponding mean and variance, For each sample, sample z from q(z|x) and use Gaussian distribution to calculate the corresponding conditional distribution density function element by element: Since the expectation of q(z|x) cannot be calculated analytically, a numerical method is used to estimate this expectation, and the Monte Carlo estimation is used to estimate the KL divergence: Where L is the number of samples, usually set to L = 1; S35: For the optimization target maxI(y,z), the reconstruction loss To ensure that the model's intermediate features z retain enough information to reconstruct the original features x, In addition, in order to maximize the removal of redundant information, the information irrelevant to the task in the intermediate representation z is removed, and the z-oriented task loss is introduced, which indirectly ensures that z contains information related to the final working condition classification task.

5. The method for intelligently identifying metallurgical furnace operating conditions by integrating multimodal data according to claim 1, characterized in that: In step S4, through the dynamic interactive fusion method of the two modal features of process variable data and image data at different levels, under the action of the bidirectional cross-attention mechanism, the inter-modal related features and private features fully interact with each other, and the features of each level are also interactively fused between levels through the bidirectional attention mechanism. The final fusion feature is obtained through the adjustment of global attention and gated fusion units. The specific steps are as follows: S41: Perform bidirectional cross-attention processing on the single-level process variable private features, image data private features, and inter-modal correlation features after redundancy removal. After stacking, residual connection, and normalization, preliminary feature interaction is achieved. Where BCA represents the bidirectional cross attention module; S42: For multiple levels The multi-level preliminary fusion features are processed with two-way cross attention to perform cross-level interactive fusion: Then all Perform splicing and calculate global fusion features through multi-head attention to improve the global representation ability of fusion features: in e represents the embedding dimension hyperparameter, MHA represents the multi-head attention mechanism; S43: For In the second dimension, we perform grading, copying, and splicing, and then input the corresponding gating network to obtain the corresponding gating fusion weights: in Then perform weighted summation and dimension transformation to obtain the final fusion feature with rich information content: in S44: Under the constraints of the cross entropy loss function: where ω l is the weight coefficient of category l, p k,l is the probability of the true label of the model predictor, fusion feature Enter the MLP classifier with category-aware attention mechanism to achieve intelligent and accurate identification of metallurgical furnace operating conditions under multimodal data input.

Images