A method and system for monitoring the condition of metallurgical blast furnace burden by integrating multimodal information

Abstract

This invention provides a method and system for monitoring the condition of blast furnace burden in metallurgical applications by integrating multimodal information. The method includes: inputting RGB time-series image sequences and multispectral time-series image sequences into a multimodal data fusion and digital representation module; performing multimodal data fusion frame-by-frame on images at each time point; extracting multi-scale image features using a combination of local and global methods; achieving deep fusion of multi-source images through cross-modal feature interaction to generate high-dimensional fused image features; and dynamically digitally representing the high-dimensional fused image features using a sensor parameter knowledge base and a hybrid expert model. The digital representation sequence and multi-sensor time-series data are then input into a multivariate time-series analysis module, which combines the two to construct a unified multivariate time-series analysis framework. This framework fully explores the time dependencies and interaction characteristics between different modal data to perform refined condition judgment of the burden surface and generate a burden surface condition analysis sequence. This invention can monitor the condition of the burden surface.

Description

Technical Field

[0001] This invention relates to the field of metallurgical blast furnace burden condition monitoring technology, and in particular to a method and system for monitoring the burden condition of metallurgical blast furnaces that integrates multimodal information. Background Technology

[0002] To optimize the blast furnace ironmaking process and achieve accurate prediction and optimized control of the blast furnace smelting process, numerous experts and scholars have conducted in-depth research and exploration, achieving certain results. Multiple production practices have shown that controlling the blast furnace gas flow distribution through top adjustment is crucial for blast furnace smelting, and the secondary and tertiary distribution of the gas flow is mainly determined by the shape of the burden surface in the furnace throat and the ore-coke ratio distribution. Therefore, reasonable burden adjustment is the core of optimizing the gas flow distribution, and burden surface condition monitoring becomes a key link. Although there are currently methods for burden surface condition monitoring, most of these methods have some problems. On the one hand, existing prediction models for blast furnace burden surface conditions are mainly based on vector machines, adaptive neural networks, and sequential extreme learning machines. These methods have good prediction effects under specific operating conditions, but their adaptability and robustness are relatively weak when facing complex, dynamic, and nonlinear changes in the blast furnace ironmaking process. Furthermore, these methods mostly rely on static data or single-objective optimization, making it difficult to respond dynamically to uncertainties in the production process in real time. On the other hand, existing research has achieved certain results in predicting single targets such as blast furnace permeability index, gas utilization rate and hearth thermal state, but lacks systematic exploration and modeling of the coupling relationship between the targets, which limits the all-round monitoring of blast furnace burden state. Summary of the Invention

[0003] To address the technical problems existing in the prior art, the present invention provides a method for monitoring the condition of metallurgical blast furnace burden by integrating multimodal information. The technical solution is as follows:

[0004] On the one hand, a method for monitoring the condition of metallurgical blast furnace burden surface that integrates multimodal information is provided, the method comprising:

[0005] S1. Acquire the RGB time-series image sequence and multispectral time-series image sequence of the blast furnace charge surface to be monitored;

[0006] S2. Input the RGB time-series image sequence and the multispectral time-series image sequence into the multimodal data fusion and digital representation module. The multimodal data fusion and digital representation module performs multimodal data fusion on each frame of the images at each time point in the RGB time-series image sequence and the multispectral time-series image sequence. It extracts multi-scale image features by combining local and global methods, and achieves deep fusion of multi-source images through cross-modal feature interaction to generate high-dimensional fused image features. Combined with the guidance of the sensor parameter knowledge base and the hybrid expert model, the high-dimensional fused image features are dynamically digitally represented. Key digital information is extracted for different scenario requirements to obtain the digital representation sequence of the hybrid image.

[0007] S3. Input the digital characterization sequence and multi-sensor time-series data into the multivariate time-series analysis module. The multivariate time-series analysis module combines the digital characterization sequence and multi-sensor time-series data to construct a unified multivariate time-series analysis framework, fully explore the time dependence and interaction characteristics between different modal data, perform refined state judgment of the material surface, and generate a material surface state analysis sequence.

[0008] Optionally, the multimodal data fusion and digital representation module includes a feature encoder, a cross-attention mechanism module, and a hybrid expert digital representation module;

[0009] The feature encoder uses a dual-branch network combining CNN and Transformer to extract multi-scale image features of RGB images and multi-scale image features of multispectral images.

[0010] The cross-attention mechanism module dynamically captures the interdependencies between the multi-scale image features of the RGB image and the multi-scale image features of the multispectral image, performs fine-grained cross-modal feature alignment and fusion, and generates high-dimensional fused image features.

[0011] The hybrid expert digital representation module interacts with the high-dimensional fused image features in a multimodal manner through the introduced sensor parameter knowledge base. With the guidance of sensor parameter knowledge, it optimizes the mapping process of high-dimensional image features to low-dimensional key digital parameters and performs dynamic digital representation for different scenarios.

[0012] Optionally, the feature encoder includes a Transformer encoder, a CNN encoder, and a deep feature fusion module;

[0013] The Transformer encoder is responsible for modeling global context information, which is used to capture long-range dependencies and global features of the image. It extracts high-dimensional features of the image through a 4-layer ViT architecture, and finally integrates the features through image patch embedding to generate global image features.

[0014] The CNN encoder is responsible for extracting local detail features of the image to enhance the model's ability to perceive targets of different sizes and complex scenes. It extracts high-dimensional features of the image by a 4-layer multi-scale CNN module and integrates the features through image patch embedding to generate local image features.

[0015] The deep feature fusion module utilizes a cross-attention mechanism to further interact and fuse the global and local image features after layer-normalized LN. The formula for the cross-attention mechanism is as follows:

[0016]

[0017] The output features of the Transformer encoder are used as the query. The output features of the CNN encoder are used as key K CNN With value V CNN , This represents the scaling factor, corresponding to K. CNN This fusion method uses global features as queries to guide the main information, while local features that enhance details are used as keys and values ​​to enrich feature expression and capture details, achieving efficient fusion of global and local information. In order to make global features dominant, query information is retained and skip connections are introduced to sum with the features obtained by cross attention, which more effectively fuses global and local features. In addition, in the subsequent integration process, layer normalization (LN), multilayer perceptron (MLP) and skip connections are introduced to further enhance the information interaction and fusion capabilities between features, effectively improving the fusion effect of multi-scale features. Finally, through the information integration of the four-layer deep feature fusion module, multi-scale image features with rich expressive capabilities are generated.

[0018] Optionally, the multi-scale CNN module uses convolutional kernels of different sizes to capture image feature information within different receptive fields. Simultaneously, it combines segmentation, fusion, and 1×1 convolution operations to further enhance the representation of image features, ultimately generating integrated local image features, specifically including:

[0019] On the one hand, the input image features are extracted at multiple scales through multiple convolutional modules with different kernel sizes. Each convolutional module contains two linear layers, one convolutional layer and a ReLU activation layer. Through these convolutional modules, image features are extracted from different receptive fields. Then, the image features under different receptive fields are integrated through channel fusion to form high-dimensional multi-scale features.

[0020] On the other hand, the global information of the input image is effectively extracted by segmentation, channel fusion and 1×1 convolution operation, and then fused with the high-dimensional multi-scale features to obtain local image features with higher expressive power.

[0021] Optionally, the cross-attention mechanism module uses the multi-scale image features of the RGB image as the query Q. RGB This is used for main information guidance, and the multi-scale image features of the multispectral image are used as key K. Multi Sum V Multi This is used to supplement detailed information and spectral feature representation, and to perform high-dimensional feature fusion between the two to generate high-dimensional fused image features. The cross-attention mechanism formula is as follows:

[0022]

[0023] in, This represents the scaling factor, corresponding to K. Multi The feature dimensions.

[0024] Optionally, the hybrid expert digital representation module flexibly adjusts the representation strategy according to different scenario requirements and current image features, dynamically generates an adaptive digital representation sequence, and extracts multi-level and multi-view visual feature representations by inputting the high-dimensional fused image features into each expert model. At the same time, based on the sensor features generated by the current sensor parameter knowledge base and the high-dimensional fused image features, a gating network is used to dynamically generate corresponding weight coefficients for each expert model, and the expert models are weighted in real time. Thus, according to different scenarios and task requirements, each expert model is given a different degree of attention. Finally, the required digital representation result is generated through feature flattening.

[0025] Optionally, the input to each expert model is the current high-dimensional fused image features. The overall architecture of the expert model includes three parallel feature processing paths. The main path uses multiple 3i×3i convolutional kernels of different sizes and image normalization to capture image features under different receptive fields, where i is the expert model number. The two branch paths use average pooling and max pooling respectively to extract global smooth features and local key features of the image, enhancing the model's ability to model global context information. In addition, both branch paths use fully connected layers and 1×1 convolutions to align features, effectively achieving the superposition of features between the three branches, and finally generating expert features processed by each expert model.

[0026] Optionally, the gated network receives the high-dimensional fused image features and sensor features as input. The sensor features are generated from sensor parameter knowledge bases for different scenarios. These knowledge bases include fixed parameters of multiple sensors involved in the scenario. The sensor parameters in the knowledge base are mapped to a higher dimension using word embedding to generate the required sensor features. These sensor features are then processed through n 1×1 convolutions to generate n weight labels, where n is the number of expert models. After passing through two fully connected layers and normalization, the high-dimensional fused image features are further mapped to a higher dimension. The resulting higher-dimensional image features and these weight labels are then fused into each weight label using a self-attention mechanism to generate n weight vectors. Finally, these weight vectors are averaged and normalized to dynamically generate the weight coefficients E1, W2, W3…W of each expert model. n These weight coefficients are used to dynamically activate different branches in the expert model module, ensuring that the model can select the optimal combination of feature processing for specific sensor parameters, scene requirements, and real-time image results.

[0027] Optionally, the multivariate time series analysis module includes a multi-source sequence fusion module and a state analysis module;

[0028] The multi-source sequence fusion module performs word embedding on the digital representation sequence and the multi-sensor time-series data respectively, maps them to a high-dimensional vector space, and generates corresponding feature mapping matrices. Then, the feature mapping matrix of the digital representation is interacted with the feature mapping matrix of any sensor time series through the feature interaction module to generate a hybrid feature. The hybrid feature will continue to interact and fuse with the feature mapping matrices of other sensors in sequence to generate the final multi-source fusion feature. The multi-source fusion feature integrates the correlation information between all sequences, providing strong support for subsequent blast furnace burden state analysis.

[0029] The feature interaction module first aligns the two feature mapping matrices using a dimension alignment module, which consists of a 3×3 convolution, a ReLU activation function, and a 1×1 convolution. This module aims to extract higher-dimensional features and adjust the number of channels for alignment. Then, it achieves pairwise interaction between channels by using channel-interval interaction, further fusing the high-dimensional features of the two sequences to generate two high-dimensional interaction features. These two high-dimensional interaction features are then further integrated using a 3×3 convolution with shared weights, which helps to fuse features across sequences. Finally, these two high-dimensional interaction features are superimposed to form a hybrid feature.

[0030] The state analysis module performs dimensionality compression on the multi-source fusion features to reduce feature dimensionality and focus on key features. Then, a self-attention mechanism is introduced into the compressed feature information to capture global dependencies and enhance the expressive power of the features. Finally, the features are integrated through a multilayer perceptron to extract key information and generate the required material surface state analysis sequence. The value of the material surface state analysis sequence is the stability assessment percentage of the real-time material surface state inside the blast furnace after the integration of multi-source information data. It is used to quantify the dynamic stability level of the material surface and provide an accurate characterization of the blast furnace operating state.

[0031] On the other hand, a metallurgical blast furnace burden condition monitoring system integrating multimodal information is provided, the system comprising:

[0032] The acquisition module is used to acquire RGB time-series image sequences and multispectral time-series image sequences of the blast furnace charge surface to be monitored.

[0033] The multimodal data fusion and digital representation module is used to input the RGB time-series image sequence and the multispectral time-series image sequence into the multimodal data fusion and digital representation module. The multimodal data fusion and digital representation module performs multimodal data fusion on each frame of the RGB time-series image sequence and the multispectral time-series image sequence, extracts multi-scale image features by combining local and global methods, and achieves deep fusion of multi-source images through cross-modal feature interaction to generate high-dimensional fused image features. Combined with the guidance of sensor parameter knowledge base and hybrid expert model, the high-dimensional fused image features are dynamically digitally represented, and key digital information is extracted for different scenario requirements to obtain a digital representation sequence of the hybrid image.

[0034] The multivariate time series analysis module is used to input the digital representation sequence and multi-sensor time series data into the multivariate time series analysis module. The multivariate time series analysis module combines the digital representation sequence and multi-sensor time series data to construct a unified multivariate time series analysis framework, fully explore the time dependence and interaction characteristics between different modal data, perform refined state judgment of the material surface, and generate a material surface state analysis sequence.

[0035] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the above-mentioned method for monitoring the condition of metallurgical blast furnace burden surface by integrating multimodal information.

[0036] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the above-described method for monitoring the condition of metallurgical blast furnace charge surface by integrating multimodal information.

[0037] The beneficial effects of the technical solution provided by this invention include at least the following:

[0038] This invention breaks through the limitations of traditional single sensor or RGB image analysis. By integrating multi-spectral, RGB, and sensor information, it achieves multi-dimensional perception of the blast furnace burden state and provides more comprehensive and detailed monitoring capabilities.

[0039] This invention designs a dynamic digital representation method based on hybrid experts. By incorporating a sensor parameter knowledge base, this method can flexibly adjust the digital representation strategy according to different scenario requirements and sensor types, achieving diversified and targeted dynamic feature extraction and expression.

[0040] This invention incorporates time-series information features and, by fully mining key historical data and long-term state information during the operation of blast furnace burden, accurately captures the evolution trend of blast furnace burden operation status, thereby significantly enhancing the robustness and applicability of blast furnace burden status monitoring.

[0041] This invention introduces a multispectral information dimension and combines it with RGB images to form a richer and more comprehensive image data source. This makes the data source not only include the corresponding color and texture features, but also incorporates the spectral features of the object, enabling precise monitoring of the blast furnace burden surface condition from a spectral dimension. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart of a metallurgical blast furnace burden condition monitoring method that integrates multimodal information, provided by an embodiment of the present invention.

[0044] Figure 2 This is a general block diagram of a metallurgical blast furnace burden condition monitoring method that integrates multimodal information, provided by an embodiment of the present invention;

[0045] Figure 3 This is a structural block diagram of the multimodal data fusion and digital representation module provided in an embodiment of the present invention;

[0046] Figure 4 This is a block diagram of the feature encoder structure provided in an embodiment of the present invention;

[0047] Figure 5 This is a block diagram of the multi-scale CNN module structure provided in the embodiments of the present invention;

[0048] Figure 6 This is a block diagram of the cross-attention mechanism module provided in an embodiment of the present invention;

[0049] Figure 7 This is a structural block diagram of the hybrid expert digital representation module provided in an embodiment of the present invention;

[0050] Figure 8 This is a block diagram of the expert model structure provided in the embodiments of the present invention;

[0051] Figure 9 This is a block diagram of the gated network structure provided in an embodiment of the present invention;

[0052] Figure 10 This is a block diagram of the multivariate time series analysis module provided in an embodiment of the present invention;

[0053] Figure 11 This is a structural block diagram of the multi-source sequence fusion module provided in an embodiment of the present invention;

[0054] Figure 12 This is a structural block diagram of the feature interaction module provided in the embodiments of the present invention;

[0055] Figure 13 This is a structural block diagram of the state analysis module provided in an embodiment of the present invention;

[0056] Figure 14 This is a block diagram of a metallurgical blast furnace burden condition monitoring system that integrates multimodal information, provided by an embodiment of the present invention.

[0057] Figure 15 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0058] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0059] This invention provides a method for monitoring the condition of metallurgical blast furnace burden surface by integrating multimodal information. This method can be implemented by an electronic device, which can be a terminal or a server. Figure 1 The flowchart of this method is shown below. Figure 2 The diagram shown is an overall block diagram of the method. The processing flow may include the following steps:

[0060] S1. Acquire the RGB time-series image sequence and multispectral time-series image sequence of the blast furnace charge surface to be monitored;

[0061] The sequence lengths of the RGB time-series image sequence and the multispectral time-series image sequence of the metallurgical blast furnace burden obtained in the embodiments of the present invention can be preset as needed, as long as their sequence lengths are consistent.

[0062] S2. Input the RGB time-series image sequence and the multispectral time-series image sequence into the multimodal data fusion and digital representation module. The multimodal data fusion and digital representation module performs multimodal data fusion on each frame of the images at each time point in the RGB time-series image sequence and the multispectral time-series image sequence. It extracts multi-scale image features by combining local and global methods, and achieves deep fusion of multi-source images through cross-modal feature interaction to generate high-dimensional fused image features. Combined with the guidance of the sensor parameter knowledge base and the hybrid expert model, the high-dimensional fused image features are dynamically digitally represented. Key digital information is extracted for different scenario requirements to obtain the digital representation sequence of the hybrid image.

[0063] Optionally, such as Figure 3 As shown, the multimodal data fusion and digital representation module includes a feature encoder, a cross-attention mechanism module, and a hybrid expert digital representation module;

[0064] The feature encoder uses a dual-branch network combining CNN and Transformer to extract multi-scale image features of RGB images and multi-scale image features of multispectral images.

[0065] The cross-attention mechanism module dynamically captures the interdependencies between the multi-scale image features of the RGB image and the multi-scale image features of the multispectral image, performs fine-grained cross-modal feature alignment and fusion, and generates high-dimensional fused image features.

[0066] The hybrid expert digital representation module interacts with the high-dimensional fused image features in a multimodal manner through the introduced sensor parameter knowledge base. With the guidance of sensor parameter knowledge, it optimizes the mapping process of high-dimensional image features to low-dimensional key digital parameters and performs dynamic digital representation for different scenarios.

[0067] Optionally, such as Figure 4 As shown, the feature encoder includes a Transformer encoder, a CNN encoder, and a deep feature fusion module;

[0068] The Transformer encoder is responsible for modeling global context information, which is used to capture long-range dependencies and global features of the image. It extracts high-dimensional features of the image through a 4-layer ViT architecture, and finally integrates the features through image patch embedding to generate global image features.

[0069] The CNN encoder is responsible for extracting local detail features of the image to enhance the model's ability to perceive targets of different sizes and complex scenes. It extracts high-dimensional features of the image by a 4-layer multi-scale CNN module and integrates the features through image patch embedding to generate local image features.

[0070] The deep feature fusion module utilizes a cross-attention mechanism to further interact and fuse the global and local image features after layer-normalized LN. The formula for the cross-attention mechanism is as follows:

[0071]

[0072] The output features of the Transformer encoder are used as the query. The output features of the CNN encoder are used as key K CNN With value V CNN , This represents the scaling factor, corresponding to K. CNN This fusion method uses global features as queries to guide the main information, while local features that enhance details are used as keys and values ​​to enrich feature expression and capture details, achieving efficient fusion of global and local information. In order to make global features dominant, query information is retained and skip connections are introduced to sum with the features obtained by cross attention, which more effectively fuses global and local features. In addition, in the subsequent integration process, layer normalization (LN), multilayer perceptron (MLP) and skip connections are introduced to further enhance the information interaction and fusion capabilities between features, effectively improving the fusion effect of multi-scale features. Finally, through the information integration of the four-layer deep feature fusion module, multi-scale image features with rich expressive capabilities are generated.

[0073] Optionally, such as Figure 5 As shown, the multi-scale CNN module uses convolutional kernels of different sizes to capture image feature information within different receptive fields. Simultaneously, it combines segmentation, fusion, and 1×1 convolution operations to further enhance the representation of image features, ultimately generating integrated local image features, specifically including:

[0074] On the one hand, the input image features are extracted at multiple scales through multiple convolutional modules with different kernel sizes (7×7, 5×5, 3×3). Each convolutional module contains two linear layers, one convolutional layer and a ReLU activation layer. Through these convolutional modules, image features are extracted from different receptive fields. Then, the image features under different receptive fields are integrated through channel fusion to form high-dimensional multi-scale features.

[0075] On the other hand, the global information of the input image is effectively extracted by segmentation, channel fusion and 1×1 convolution operation, and then fused with the high-dimensional multi-scale features to obtain local image features with higher expressive power.

[0076] Optionally, such as Figure 6 As shown, the cross-attention mechanism module uses the multi-scale image features of the RGB image as the query Q. RGB This is used for main information guidance, and the multi-scale image features of the multispectral image are used as key K. Multi Sum V Multi This is used to supplement detailed information and spectral feature representation, and to perform high-dimensional feature fusion between the two to generate high-dimensional fused image features. The cross-attention mechanism formula is as follows:

[0077]

[0078] in, This represents the scaling factor, corresponding to K. Multi The feature dimensions.

[0079] To better integrate image and sensor sequences and avoid excessive bias towards image information during feature fusion due to high image resolution, which could mask the role of sensor data, this invention introduces a hybrid expert dynamic digital representation method. This method transforms the image sequence into a digital sequence with specifications consistent with the sensor sequence, helping to balance the information contributions of both and thus obtaining more refined and balanced fused features.

[0080] Optionally, such as Figure 7 As shown, the hybrid expert digital representation module flexibly adjusts the representation strategy according to different scenario requirements and current image features, dynamically generating an adaptive digital representation sequence. By inputting the high-dimensional fused image features into each expert model, it extracts multi-level and multi-view visual feature representations. Simultaneously, based on the sensor features generated from the current sensor parameter knowledge base and the high-dimensional fused image features, it uses a gating network to dynamically generate corresponding weight coefficients for each expert model, and weights each expert model in real time. Thus, according to different scenarios and task requirements, it assigns different levels of attention to each expert model. Finally, through feature flattening, it generates the required digital representation result.

[0081] The digital representation results can dynamically determine the representation method based on the current scene and image features. For example, in different scenarios, the representation results will be presented in different forms, such as material surface coverage percentage or material surface uniformity coefficient. Specifically, the model will select the most suitable representation direction for the current scene based on the current sensor data and image features. In some scenarios, the representation result will be expressed as material surface coverage percentage (e.g., 80%, 60%, 70%), intuitively reflecting the coverage degree of different materials. In other scenarios, the system will represent the image as a material surface uniformity coefficient (e.g., 0.7, 0.6), to describe the uniformity of the material surface. Furthermore, digital representation is not limited to these forms; the model can dynamically select different representation methods and presentation forms according to the needs of different scenarios and real-time image features to provide more accurate and flexible analysis results.

[0082] Optionally, such as Figure 8 As shown, the input to each expert model is the current high-dimensional fused image feature. The overall architecture of the expert model includes three parallel feature processing paths. The main path uses multiple 3i×3i convolution kernels of different sizes and image normalization to capture image features under different receptive fields, where i is the expert model number (with a value range of 1...n). The two branch paths use average pooling and max pooling respectively to extract global smooth features and local key features of the image, enhancing the model's ability to model global context information. In addition, both branch paths use fully connected layers and 1×1 convolutions to align features, effectively achieving the superposition of features between the three branches, and finally generating expert features processed by each expert model.

[0083] Optionally, such as Figure 9 As shown, the gated network receives the high-dimensional fused image features and sensor features as input. The sensor features are generated from sensor parameter knowledge bases for different scenarios. These knowledge bases include fixed parameters of multiple sensors involved in the scenario. The sensor parameters in the knowledge base are mapped to a higher dimension using word embedding to generate the required sensor features. These sensor features are then processed through n 1×1 convolutions to generate n weight labels, where n is the number of expert models. The high-dimensional fused image features are further mapped to a higher dimension after passing through two fully connected layers and normalization. The resulting higher-dimensional image features and these weight labels are then fused into each weight label using a self-attention mechanism to generate n weight vectors. Finally, these weight vectors are averaged and normalized to dynamically generate the weight coefficients W1, W2, W3…W of each expert model. n These weight coefficients are used to dynamically activate different branches in the expert model module, ensuring that the model can select the optimal combination of feature processing for specific sensor parameters, scene requirements, and real-time image results.

[0084] S3. Input the digital characterization sequence and multi-sensor time-series data into the multivariate time-series analysis module. The multivariate time-series analysis module combines the digital characterization sequence and multi-sensor time-series data to construct a unified multivariate time-series analysis framework, fully explore the time dependence and interaction characteristics between different modal data, perform refined state judgment of the material surface, and generate a material surface state analysis sequence.

[0085] Optionally, such as Figure 10 As shown, the multivariate time series analysis module includes a multi-source sequence fusion module and a state analysis module;

[0086] like Figure 11 As shown, the multi-source sequence fusion module performs word embedding on the digital representation sequence and the multi-sensor time-series data respectively, maps them to a high-dimensional vector space, and generates corresponding feature mapping matrices. Then, the feature mapping matrix of the digital representation is interacted with the feature mapping matrix of any sensor time series through the feature interaction module to generate a hybrid feature. The hybrid feature will continue to interact and fuse with the feature mapping matrices of other sensors in sequence to generate the final multi-source fusion feature. The multi-source fusion feature integrates the correlation information between all sequences, providing strong support for subsequent blast furnace burden state analysis.

[0087] like Figure 12 As shown, the feature interaction module first aligns the two feature mapping matrices using a dimension alignment module. This module consists of a 3×3 convolution, a ReLU activation function, and a 1×1 convolution, aiming to extract higher-dimensional features and adjust the channel number alignment. Then, it achieves pairwise interaction between channels by using channel-interval interaction, further fusing the high-dimensional features of the two sequences to generate two high-dimensional interaction features. These two high-dimensional interaction features are further integrated using a 3×3 convolution with shared weights, which helps to fuse cross-sequence features. Finally, these two high-dimensional interaction features are superimposed to form a hybrid feature.

[0088] like Figure 13 As shown, the state analysis module performs dimensionality compression on the multi-source fusion features to reduce feature dimensionality and focus on key features. Then, a self-attention mechanism is introduced into the compressed feature information to capture global dependencies and enhance the expressive power of the features. Finally, the features are integrated through a multilayer perceptron to extract key information and generate the required material surface state analysis sequence. The value of the material surface state analysis sequence is the stability assessment percentage of the real-time material surface state inside the blast furnace after the integration of multi-source information data. It is used to quantify the dynamic stability level of the material surface and provide an accurate characterization of the blast furnace operating state.

[0089] like Figure 14 As shown in the figure, this embodiment of the invention also provides a metallurgical blast furnace burden condition monitoring system that integrates multimodal information, the system comprising:

[0090] The acquisition module 1410 is used to acquire the RGB time-series image sequence and the multispectral time-series image sequence of the blast furnace material surface to be monitored.

[0091] The multimodal data fusion and digital representation module 1420 is used to input the RGB time-series image sequence and the multispectral time-series image sequence into the multimodal data fusion and digital representation module. The multimodal data fusion and digital representation module performs multimodal data fusion on each frame of the RGB time-series image sequence and the multispectral time-series image sequence. It extracts multi-scale image features by combining local and global methods, and achieves deep fusion of multi-source images through cross-modal feature interaction to generate high-dimensional fused image features. Combined with the guidance of sensor parameter knowledge base and hybrid expert model, it performs dynamic digital representation of the high-dimensional fused image features, extracts key digital information for different scenario requirements, and obtains a digital representation sequence of the hybrid image.

[0092] The multivariate time series analysis module 1430 is used to input the digital representation sequence and multi-sensor time series data into the multivariate time series analysis module. The multivariate time series analysis module combines the digital representation sequence and multi-sensor time series data to construct a unified multivariate time series analysis framework, fully explore the time dependence and interaction characteristics between different modal data, perform refined state judgment of the material surface, and generate a material surface state analysis sequence.

[0093] The metallurgical blast furnace burden condition monitoring system that integrates multimodal information provided in this embodiment of the invention has a functional structure that corresponds to the metallurgical blast furnace burden condition monitoring method that integrates multimodal information provided in this embodiment of the invention, and will not be described again here.

[0094] Figure 15 This is a schematic diagram of the structure of an electronic device 1500 provided in an embodiment of the present invention. The electronic device 1500 may vary considerably due to different configurations or performance. It may include one or more central processing units (CPUs) 1501 and one or more memories 1502. The memory 1502 stores at least one instruction, which is loaded and executed by the processor 1501 to implement the steps of the above-mentioned method for monitoring the condition of metallurgical blast furnace burden surface by integrating multimodal information.

[0095] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to complete the aforementioned method for monitoring the condition of metallurgical blast furnace burden by fusing multimodal information. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0096] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0097] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for monitoring the condition of metallurgical blast furnace burden surface by integrating multimodal information, characterized in that, The method includes: S1. Acquire the RGB time-series image sequence and multispectral time-series image sequence of the blast furnace charge surface to be monitored; S2. Input the RGB time-series image sequence and the multispectral time-series image sequence into the multimodal data fusion and digital representation module. The multimodal data fusion and digital representation module performs multimodal data fusion on each frame of the images at each time point in the RGB time-series image sequence and the multispectral time-series image sequence. It extracts multi-scale image features by combining local and global methods, and achieves deep fusion of multi-source images through cross-modal feature interaction to generate high-dimensional fused image features. Combined with the guidance of the sensor parameter knowledge base and the hybrid expert model, the high-dimensional fused image features are dynamically digitally represented. Key digital information is extracted for different scenario requirements to obtain the digital representation sequence of the hybrid image. S3. Input the digital characterization sequence and multi-sensor time series data into the multivariate time series analysis module. The multivariate time series analysis module combines the digital characterization sequence and multi-sensor time series data to construct a unified multivariate time series analysis framework, fully explore the time dependence and interaction characteristics between different modal data, perform refined state judgment of the material surface, and generate a material surface state analysis sequence. The multimodal data fusion and digital representation module includes a feature encoder, a cross-attention mechanism module, and a hybrid expert digital representation module. The feature encoder uses a dual-branch network combining CNN and Transformer to extract multi-scale image features of RGB images and multi-scale image features of multispectral images. The cross-attention mechanism module dynamically captures the interdependencies between the multi-scale image features of the RGB image and the multi-scale image features of the multispectral image, performs fine-grained cross-modal feature alignment and fusion, and generates high-dimensional fused image features. The hybrid expert digital representation module interacts with the high-dimensional fused image features in a multimodal manner through the introduced sensor parameter knowledge base. With the guidance of sensor parameter knowledge, it optimizes the mapping process of high-dimensional image features to low-dimensional key digital parameters and performs dynamic digital representation for different scenarios.

2. The method according to claim 1, characterized in that, The feature encoder includes a Transformer encoder, a CNN encoder, and a deep feature fusion module; The Transformer encoder is responsible for modeling global context information, which is used to capture long-range dependencies and global features of the image. It extracts high-dimensional features of the image through a 4-layer ViT architecture, and finally integrates the features through image patch embedding to generate global image features. The CNN encoder is responsible for extracting local detail features of the image to enhance the model's ability to perceive targets of different sizes and complex scenes. It extracts high-dimensional features of the image by a 4-layer multi-scale CNN module and integrates the features through image patch embedding to generate local image features. The deep feature fusion module utilizes a cross-attention mechanism to further interact and fuse the global and local image features after layer-normalized LN. The formula for the cross-attention mechanism is as follows: The output features of the Transformer encoder are used as the query Q. Trans The output features of the CNN encoder serve as key K CNN With value V CNN , This represents the scaling factor, corresponding to K. CNN This fusion method uses global features as queries to guide the main information, while local features that enhance details are used as keys and values ​​to enrich feature expression and capture details, achieving efficient fusion of global and local information. Furthermore, to ensure global features dominate, query information is retained, and skip connections are introduced to sum with features obtained through cross-attention, further fusion of global and local features. In subsequent integration, layer normalization (LN), multilayer perceptron (MLP), and skip connections are introduced to further enhance information interaction and fusion capabilities between features, effectively improving the fusion effect of multi-scale features. Finally, through information integration by the four-layer deep feature fusion module, multi-scale image features with rich expressive capabilities are generated.

3. The method according to claim 2, characterized in that, The multi-scale CNN module employs convolutional kernels of different sizes to capture image feature information within different receptive fields. Simultaneously, it combines segmentation, fusion, and 1×1 convolution operations to further enhance the representation of image features, ultimately generating integrated local image features, specifically including: On the one hand, the input image features are extracted at multiple scales through multiple convolutional modules with different kernel sizes. Each convolutional module contains two linear layers, one convolutional layer and a ReLU activation layer. Through these convolutional modules, image features are extracted from different receptive fields. Then, the image features under different receptive fields are integrated through channel fusion to form high-dimensional multi-scale features. On the other hand, the global information of the input image is effectively extracted by segmentation, channel fusion and 1×1 convolution operation, and then fused with the high-dimensional multi-scale features to obtain local image features with higher expressive power.

4. The method according to claim 1, characterized in that, The cross-attention mechanism module uses the multi-scale image features of the RGB image as the query Q. RGB This is used for main information guidance, and the multi-scale image features of the multispectral image are used as key K. Multi Sum V Multi This is used to supplement detailed information and spectral feature representation, and to perform high-dimensional feature fusion between the two to generate high-dimensional fused image features. The cross-attention mechanism formula is as follows: in, This represents the scaling factor, corresponding to K. Multi The feature dimensions.

5. The method according to claim 1, characterized in that, The hybrid expert digital representation module flexibly adjusts its representation strategy based on different scenario requirements and current image features, dynamically generating an adaptive digital representation sequence. By inputting the high-dimensional fused image features into each expert model, it extracts multi-level and multi-view visual feature representations. Simultaneously, based on the sensor features generated from the current sensor parameter knowledge base and the high-dimensional fused image features, it uses a gating network to dynamically generate corresponding weight coefficients for each expert model, weighting each expert model in real time. This allows for assigning different levels of attention to each expert model according to different scenarios and task requirements. Finally, through feature flattening, it generates the required digital representation result.

6. The method according to claim 5, characterized in that, The input to each expert model is the current high-dimensional fused image features. The overall architecture of the expert model includes three parallel feature processing paths. The main path uses multiple 3i×3i convolutional kernels of different sizes and image normalization to capture image features under different receptive fields, where i is the expert model number. The two branch paths use average pooling and max pooling respectively to extract global smooth features and local key features of the image, enhancing the model's ability to model global context information. In addition, both branch paths use fully connected layers and 1×1 convolutions to align features, effectively achieving the superposition of features between the three branches, and finally generating expert features processed by each expert model.

7. The method according to claim 5, characterized in that, The gated network receives the high-dimensional fused image features and sensor features as input. The sensor features are generated from sensor parameter knowledge bases for different scenarios. These knowledge bases include fixed parameters of multiple sensors involved in the scenario. The sensor parameters in the knowledge base are mapped to a higher dimension using word embedding to generate the required sensor features. These sensor features are then processed through n 1×1 convolutions to generate n weight labels, where n is the number of expert models. The high-dimensional fused image features are further mapped to a higher dimension after passing through two fully connected layers and normalization. The resulting higher-dimensional image features and these weight labels are then fused into each weight label using a self-attention mechanism to generate n weight vectors. Finally, these weight vectors are averaged and normalized to dynamically generate the weight coefficients W1, W2, W3…W of each expert model. n These weight coefficients are used to dynamically activate different branches in the expert model module, ensuring that the model can select the optimal combination of feature processing for specific sensor parameters, scene requirements, and real-time image results.

8. The method according to claim 1, characterized in that, The multivariate time series analysis module includes a multi-source sequence fusion module and a state analysis module; The multi-source sequence fusion module performs word embedding on the digital representation sequence and the multi-sensor time-series data respectively, maps them to a high-dimensional vector space, and generates corresponding feature mapping matrices. Then, the feature mapping matrix of the digital representation is interacted with the feature mapping matrix of any sensor time series through the feature interaction module to generate a hybrid feature. The hybrid feature will continue to interact and fuse with the feature mapping matrices of other sensors in sequence to generate the final multi-source fusion feature. The multi-source fusion feature integrates the correlation information between all sequences, providing strong support for subsequent blast furnace burden state analysis. The feature interaction module first aligns the two feature mapping matrices using a dimension alignment module, which consists of a 3×3 convolution, a ReLU activation function, and a 1×1 convolution. This module aims to extract higher-dimensional features and adjust the number of channels for alignment. Then, it achieves pairwise interaction between channels by using channel-interval interaction, further fusing the high-dimensional features of the two sequences to generate two high-dimensional interaction features. These two high-dimensional interaction features are then further integrated using a 3×3 convolution with shared weights, which helps to fuse features across sequences. Finally, these two high-dimensional interaction features are superimposed to form a hybrid feature. The state analysis module performs dimensionality compression on the multi-source fusion features to reduce feature dimensionality and focus on key features. Then, a self-attention mechanism is introduced into the compressed feature information to capture global dependencies and enhance the expressive power of the features. Finally, the features are integrated through a multilayer perceptron to extract key information and generate the required material surface state analysis sequence. The value of the material surface state analysis sequence is the stability assessment percentage of the real-time material surface state inside the blast furnace after the integration of multi-source information data. It is used to quantify the dynamic stability level of the material surface and provide an accurate characterization of the blast furnace operating state.

9. A metallurgical blast furnace burden condition monitoring system integrating multimodal information, characterized in that, The system includes: The acquisition module is used to acquire RGB time-series image sequences and multispectral time-series image sequences of the blast furnace charge surface to be monitored. The multimodal data fusion and digital representation module is used to input the RGB time-series image sequence and the multispectral time-series image sequence into the multimodal data fusion and digital representation module. The multimodal data fusion and digital representation module performs multimodal data fusion on each frame of the RGB time-series image sequence and the multispectral time-series image sequence, extracts multi-scale image features by combining local and global methods, and achieves deep fusion of multi-source images through cross-modal feature interaction to generate high-dimensional fused image features. Combined with the guidance of sensor parameter knowledge base and hybrid expert model, the high-dimensional fused image features are dynamically digitally represented, and key digital information is extracted for different scenario requirements to obtain a digital representation sequence of the hybrid image. The multivariate time series analysis module is used to input the digital representation sequence and multi-sensor time series data into the multivariate time series analysis module. The multivariate time series analysis module combines the digital representation sequence and multi-sensor time series data to construct a unified multivariate time series analysis framework, fully explore the time dependence and interaction characteristics between different modal data, perform refined state judgment of the material surface, and generate a material surface state analysis sequence. The multimodal data fusion and digital representation module includes a feature encoder, a cross-attention mechanism module, and a hybrid expert digital representation module. The feature encoder uses a dual-branch network combining CNN and Transformer to extract multi-scale image features of RGB images and multi-scale image features of multispectral images. The cross-attention mechanism module dynamically captures the interdependencies between the multi-scale image features of the RGB image and the multi-scale image features of the multispectral image, performs fine-grained cross-modal feature alignment and fusion, and generates high-dimensional fused image features. The hybrid expert digital representation module interacts with the high-dimensional fused image features in a multimodal manner through the introduced sensor parameter knowledge base. With the guidance of sensor parameter knowledge, it optimizes the mapping process of high-dimensional image features to low-dimensional key digital parameters and performs dynamic digital representation for different scenarios.

Images