Metallurgy blast furnace charge level state monitoring method and system fused with multi-modal information

Abstract

The invention provides a metallurgical blast furnace burden surface state monitoring method and system fused with multi-modal information, and the method comprises the steps: inputting an RGB time sequence image sequence and a multispectral time sequence image sequence into a multi-modal data fusion and digital representation module, and carrying out the multi-modal data fusion of images at all moments frame by frame, the method comprises the following steps: extracting multi-scale image features by adopting a local and global combined mode, realizing deep fusion of multi-source images through cross-modal feature interaction, generating high-dimensional fusion image features, and performing dynamic digital representation on the high-dimensional fusion image features by combining the guiding effects of a sensor parameter knowledge base and a hybrid expert model; and inputting the digital representation sequence and the multi-sensor time sequence data into a multivariable time sequence analysis module, combining the digital representation sequence and the multi-sensor time sequence data to construct a unified multivariable time sequence analysis framework, fully mining time dependence and interaction characteristics among different modal data, performing refined state judgment on the charge level, and generating a charge level state analysis sequence. According to the invention, the charge level state can be monitored.

Description

Technical Field

[0001] The present invention relates to the technical field of metallurgical blast furnace charge surface status monitoring, and in particular to a metallurgical blast furnace charge surface status monitoring method and system integrating multi-modal information. Background Art

[0002] In order to optimize the blast furnace ironmaking production process and realize accurate prediction and optimization control of the blast furnace smelting process, many experts and scholars have conducted in-depth research and exploration, and achieved certain results. After many production practices, it has been shown that controlling the blast furnace gas flow distribution through upper adjustment is crucial to blast furnace smelting, and the secondary and tertiary distribution of gas flow is mainly determined by the shape of the throat material surface and the distribution of ore-coke ratio. Therefore, reasonable material distribution adjustment is the core of optimizing gas flow distribution, and material surface state monitoring becomes a key link. Although there are methods for material surface state monitoring, most of these methods have some problems. On the one hand, the existing prediction models of blast furnace material surface state are mainly based on vector machines, adaptive neural networks, sequential extreme learning machines and other methods. These methods have good prediction effects under specific working conditions, but when faced with complex, dynamic, and nonlinear working conditions in the blast furnace ironmaking process, their adaptability and robustness are relatively weak. In addition, most of these methods rely on static data or single-objective optimization, and it is difficult to respond to uncertainties in the production process in real time and dynamically. On the other hand, existing research has achieved certain results in predicting single targets such as blast furnace permeability index, gas utilization rate and furnace thermal state, but lacks systematic exploration and modeling of the coupling relationship between the various targets, which limits the all-round monitoring of the blast furnace charge surface status. Summary of the invention

[0003] In order to solve the technical problems existing in the above-mentioned prior art, the present invention provides a method for monitoring the charge surface of a metallurgical blast furnace by integrating multimodal information. The technical solution is as follows:

[0004] On the one hand, a method for monitoring the charge surface of a metallurgical blast furnace by integrating multimodal information is provided, the method comprising:

[0005] S1, obtaining an RGB time-series image sequence and a multi-spectral time-series image sequence of the material surface of a metallurgical blast furnace to be monitored;

[0006] S2. Input the RGB time-series image sequence and the multispectral time-series image sequence into a multimodal data fusion and digital representation module. The multimodal data fusion and digital representation module performs multimodal data fusion on each moment image in the RGB time-series image sequence and the multispectral time-series image sequence frame by frame, extracts multi-scale image features by combining local and global methods, and realizes deep fusion of multi-source images through cross-modal feature interaction to generate high-dimensional fused image features. Combined with the guidance of the sensor parameter knowledge base and the hybrid expert model, the high-dimensional fused image features are dynamically digitally represented, and key digital information is extracted according to different scene requirements to obtain a digital representation sequence of the hybrid image.

[0007] S3. Input the digital characterization sequence and the multi-sensor time series data into a multivariate time series analysis module. The multivariate time series analysis module combines the digital characterization sequence with the multi-sensor time series data to construct a unified multivariate time series analysis framework, fully explore the time dependency and interaction characteristics between different modal data, perform refined state judgment of the material surface, and generate a material surface state analysis sequence.

[0008] Optionally, the multimodal data fusion and digital representation module includes a feature encoder, a cross-attention mechanism module and a hybrid expert digital representation module;

[0009] The feature encoder uses a dual-branch network combining CNN and Transformer to extract multi-scale image features of RGB images and multi-scale image features of multi-spectral images;

[0010] The cross-attention mechanism module dynamically captures the interdependence between the multi-scale image features of the RGB image and the multi-scale image features of the multispectral image, performs fine-grained cross-modal feature alignment and fusion, and generates high-dimensional fused image features;

[0011] The hybrid expert digital characterization module, through the introduction of the sensor parameter knowledge base, performs multimodal interaction with the high-dimensional fusion image features, and with the guidance of the sensor parameter knowledge, optimizes the mapping process of high-dimensional image features to low-dimensional key digital parameters, and performs dynamic digital characterization for different scenarios.

[0012] Optionally, the feature encoder includes a Transformer encoder, a CNN encoder and a deep feature fusion module;

[0013] The Transformer encoder is responsible for modeling global context information, which is used to capture the long-range dependencies and global features of the image. The 4-layer ViT architecture extracts the high-dimensional features of the image, and finally integrates the features through image block embedding to generate global image features.

[0014] The CNN encoder is responsible for extracting local detail features of the image to enhance the model's perception of objects of different sizes and complex scenes. The 4-layer multi-scale CNN module extracts high-dimensional features of the image and integrates the features through image block embedding to generate local image features.

[0015] The deep feature fusion module uses the cross attention mechanism to further interact and fuse the global image features and local image features after layer normalization LN. The cross attention mechanism formula is as follows:

[0016]

[0017] Among them, the output features of the Transformer encoder are used as queries The output features of the CNN encoder are used as the key K CNN With value V CNN , Represents the scaling factor, corresponding to K CNN feature dimension. This fusion method uses global features as queries for main information guidance, and uses local features that enhance details as keys and values ​​to enrich feature expression and capture details, thereby achieving efficient fusion of global and local information. At the same time, in order to make global features dominant, query information is retained, and jump connections are introduced to add them to the features obtained by cross-attention, so as to more effectively fuse global and local features. In addition, in the subsequent integration process, layer normalization LN, multi-layer perceptron MLP and jump connections are introduced to further enhance the information interaction and fusion capabilities between features, and effectively improve the fusion effect of multi-scale features. Finally, through the information integration of the four-layer deep feature fusion module, multi-scale image features with rich expression capabilities are generated.

[0018] Optionally, the multi-scale CNN module uses convolution kernels of different sizes to capture image feature information within different receptive fields, and combines segmentation, fusion and 1×1 convolution operations to further enhance the expression of image features, and finally generates integrated local image features, specifically including:

[0019] On the one hand, the input image features are extracted from multiple convolution modules with different convolution kernel sizes for multi-scale information. Each convolution module contains two linear layers, one convolution layer and a Relu activation layer. Through these convolution modules, image features are extracted from different receptive fields. After that, the image features under different receptive fields are integrated through channel fusion to form high-dimensional multi-scale features.

[0020] On the other hand, the global information of the input image is effectively extracted through segmentation, channel fusion and 1×1 convolution operations, and the global information is channel-fused with the high-dimensional multi-scale features to obtain local image features with higher expressiveness.

[0021] Optionally, the cross attention mechanism module uses the multi-scale image features of the RGB image as the query Q RGB , used for main information guidance, and the multi-scale image features of the multispectral image are used as the key K Multi Sum value V Multi , used to supplement the expression of detail information and spectral features, perform high-dimensional feature fusion between the two, and generate high-dimensional fused image features. The cross-attention mechanism formula is as follows:

[0022]

[0023] in, Represents the scaling factor, corresponding to K Multi feature dimension.

[0024] Optionally, the hybrid expert digital representation module flexibly adjusts the representation strategy according to different scene requirements and current image features, dynamically generates an adaptive digital representation sequence, and extracts multi-level, multi-perspective visual feature representations by inputting the high-dimensional fused image features into each expert model. At the same time, based on the sensor features generated by the current sensor parameter knowledge base and the high-dimensional fused image features, a gating network is used to dynamically generate corresponding weight coefficients for each expert model, and each expert model is weighted in real time, so as to give each expert model a different degree of attention according to different scenes and task requirements, and finally generate the required digital representation results through feature flattening;

[0025] Optionally, the input of each expert model is the current high-dimensional fusion image feature. The overall architecture of the expert model includes three parallel feature processing paths. The trunk path uses multiple 3i×3i convolution kernels of different sizes and image normalization to capture image features under different receptive fields. i is the expert model number. The two branch paths use average pooling and maximum pooling respectively to extract global smooth features and local key features of the image, thereby enhancing the model's modeling ability for global contextual information. In addition, both branch paths use fully connected layers and 1×1 convolutions to align features, effectively realizing the superposition of the three branch features, and finally generating expert features processed by each expert model.

[0026] Optionally, the gated network receives the high-dimensional fused image features and sensor features as input, wherein the sensor features are generated by a sensor parameter knowledge base of different scenes, the sensor parameter knowledge base includes fixed parameters of multiple sensors involved in the scenes, and the sensor parameters in the sensor parameter knowledge base are mapped to high dimensions through word embedding to generate the required sensor features. The sensor features are processed through n 1×1 convolutions to generate n weight identifiers, where n is the number of expert models. After the high-dimensional fused image features are processed through two fully connected layers and normalization, the image features are further mapped to a higher dimension. The obtained higher-dimensional image features and these weight identifiers are used to fuse the high-dimensional image feature information into each weight identifier through a self-attention mechanism to generate n weight vectors. Finally, these weight vectors are processed by averaging and normalization to dynamically generate weight coefficients E of each expert model. 1 ,W 2 ,W 3 …W n ,These weight coefficients are used to dynamically activate different branches in the expert model module,,ensuring that the model can select the optimal feature processing combination,for specific sensor parameters, scene requirements, and real-time image results.

[0027] Optionally, the multivariate time series analysis module includes a multi-source sequence fusion module and a state analysis module;

[0028] The multi-source sequence fusion module performs word embedding on the digital representation sequence and the multi-sensor time series data, respectively, maps them to a high-dimensional vector space, generates a corresponding feature mapping matrix, and then interacts with the feature mapping matrix of the digital representation and the feature mapping matrix of any sensor time series through feature interaction modules to generate mixed features. The mixed features will continue to interact and fuse with other sensor feature mapping matrices in turn to generate the final multi-source fusion features. The multi-source fusion features fuse the correlation information between all sequences, providing strong support for the subsequent blast furnace charge surface state analysis;

[0029] The feature interaction module first aligns the dimensions of the two feature mapping matrices respectively through a dimension alignment module. The dimension alignment module is composed of a 3×3 convolution, a Relu activation function, and a 1×1 convolution, and is intended to extract higher-dimensional features and adjust the number of channels for alignment. After that, the two-to-two interactions between channels are achieved in a channel interval interaction manner, and the high-dimensional features of the two sequences are further fused to generate two high-dimensional interaction features. The two high-dimensional interaction features are further integrated with the help of a 3×3 convolution with shared weights, thereby facilitating the fusion of cross-sequence features. After that, the two high-dimensional interaction features are superimposed to form a mixed feature.

[0030] The state analysis module compresses the dimensions of the multi-source fusion features to reduce the feature dimensions and focus on key features. Then, a self-attention mechanism is introduced into the compressed feature information to capture global dependencies and enhance the expressiveness of the features. Finally, the features are integrated through a multi-layer perceptron to extract key information and generate the required material surface state analysis sequence. The value of the material surface state analysis sequence is the stability assessment percentage of the real-time material surface state inside the blast furnace after the integration of multi-source information data, which is used to quantify the dynamic stability level of the material surface and provide an accurate representation of the blast furnace operation state.

[0031] On the other hand, a metallurgical blast furnace charge surface status monitoring system integrating multimodal information is provided, the system comprising:

[0032] An acquisition module is used to acquire an RGB time-series image sequence and a multi-spectral time-series image sequence of the material surface of a metallurgical blast furnace to be monitored;

[0033] A multimodal data fusion and digital representation module is used to input the RGB time-series image sequence and the multispectral time-series image sequence into the multimodal data fusion and digital representation module, and the multimodal data fusion and digital representation module performs multimodal data fusion on each moment image in the RGB time-series image sequence and the multispectral time-series image sequence frame by frame, extracts multi-scale image features by combining local and global methods, and realizes deep fusion of multi-source images through cross-modal feature interaction to generate high-dimensional fused image features. Combined with the guidance of the sensor parameter knowledge base and the hybrid expert model, the high-dimensional fused image features are dynamically digitally represented, and key digital information is extracted according to different scene requirements to obtain a digital representation sequence of the hybrid image;

[0034] A multivariate timing analysis module is used to input the digital representation sequence and the multi-sensor timing data into the multivariate timing analysis module. The multivariate timing analysis module combines the digital representation sequence with the multi-sensor timing data to construct a unified multivariate timing analysis framework, fully explore the time dependency and interaction characteristics between different modal data, perform refined state judgment of the material surface, and generate a material surface state analysis sequence.

[0035] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the above-mentioned metallurgical blast furnace material surface status monitoring method integrating multimodal information.

[0036] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement the above-mentioned metallurgical blast furnace material surface status monitoring method integrating multimodal information.

[0037] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0038] The present invention breaks through the limitations of traditional single sensor or RGB image analysis, and realizes multi-dimensional perception of the blast furnace material surface status by integrating multi-source information such as multi-spectrum, RGB, and sensors, providing more comprehensive and detailed monitoring capabilities.

[0039] The present invention designs a hybrid expert dynamic digital characterization method. By combining the sensor parameter knowledge base, the method can flexibly adjust the digital characterization strategy according to different scene requirements and sensor types, and realize diversified and targeted dynamic feature extraction and expression.

[0040] The present invention incorporates time series information features, fully mines key historical data and long-term status information in the operation process of blast furnace material surface, accurately captures the evolution trend of the operation status of blast furnace material surface, thereby significantly enhancing the robustness and applicability of blast furnace material surface status monitoring.

[0041] The present invention introduces the multi-spectral information dimension and combines it with the RGB image to form a richer and more comprehensive image data source. This makes the data source not only include the corresponding color features and texture features, but also introduces the spectral features of the object, thereby realizing accurate monitoring of the blast furnace charge surface status from the spectral dimension. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0043] Figure 1 It is a flow chart of a metallurgical blast furnace charge surface status monitoring method integrating multi-modal information provided by an embodiment of the present invention;

[0044] Figure 2 It is an overall block diagram of a metallurgical blast furnace charge surface status monitoring method integrating multi-modal information provided by an embodiment of the present invention;

[0045] Figure 3 It is a structural block diagram of a multimodal data fusion and digital representation module provided by an embodiment of the present invention;

[0046] Figure 4 is a structural block diagram of a feature encoder provided by an embodiment of the present invention;

[0047] Figure 5 is a block diagram of the multi-scale CNN module structure provided by an embodiment of the present invention;

[0048] Figure 6 It is a structural block diagram of a cross-attention mechanism module provided by an embodiment of the present invention;

[0049] Figure 7 It is a structural block diagram of a hybrid expert digital representation module provided by an embodiment of the present invention;

[0050] Figure 8 is a structural block diagram of an expert model provided by an embodiment of the present invention;

[0051] Fig. 9 is a block diagram of a gated network structure provided by an embodiment of the present invention;

[0052] Fig.10 is a structural block diagram of a multivariate timing analysis module provided by an embodiment of the present invention;

[0053] Fig.11 is a structural block diagram of a multi-source sequence fusion module provided by an embodiment of the present invention;

[0054] Fig.12 is a structural block diagram of a feature interaction module provided by an embodiment of the present invention;

[0055] Fig.13 is a structural block diagram of a state analysis module provided by an embodiment of the present invention;

[0056] Fig.14 It is a block diagram of a metallurgical blast furnace material surface status monitoring system integrating multi-modal information provided by an embodiment of the present invention;

[0057] Fig.15 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0059] An embodiment of the present invention provides a method for monitoring the charge surface status of a metallurgical blast furnace by integrating multimodal information. The method can be implemented by an electronic device, which can be a terminal or a server. Figure 1 The flowchart of this method is shown in FIG. Figure 2 The overall block diagram of the method is shown, and the processing flow may include the following steps:

[0060] S1, obtaining an RGB time-series image sequence and a multi-spectral time-series image sequence of the material surface of a metallurgical blast furnace to be monitored;

[0061] The sequence lengths of the RGB time-series image sequence and the multi-spectral time-series image sequence of the material surface of the metallurgical blast furnace to be monitored obtained by the embodiment of the present invention can be preset as needed, as long as their sequence lengths are consistent.

[0062] S2. Input the RGB time-series image sequence and the multispectral time-series image sequence into a multimodal data fusion and digital representation module. The multimodal data fusion and digital representation module performs multimodal data fusion on each moment image in the RGB time-series image sequence and the multispectral time-series image sequence frame by frame, extracts multi-scale image features by combining local and global methods, and realizes deep fusion of multi-source images through cross-modal feature interaction to generate high-dimensional fused image features. Combined with the guidance of the sensor parameter knowledge base and the hybrid expert model, the high-dimensional fused image features are dynamically digitally represented, and key digital information is extracted according to different scene requirements to obtain a digital representation sequence of the hybrid image.

[0063] Alternatively, if Figure 3 As shown, the multimodal data fusion and digital representation module includes a feature encoder, a cross-attention mechanism module and a hybrid expert digital representation module;

[0064] The feature encoder uses a dual-branch network combining CNN and Transformer to extract multi-scale image features of RGB images and multi-scale image features of multi-spectral images;

[0065] The cross-attention mechanism module dynamically captures the interdependence between the multi-scale image features of the RGB image and the multi-scale image features of the multispectral image, performs fine-grained cross-modal feature alignment and fusion, and generates high-dimensional fused image features;

[0066] The hybrid expert digital characterization module, through the introduction of the sensor parameter knowledge base, performs multimodal interaction with the high-dimensional fusion image features, and with the guidance of the sensor parameter knowledge, optimizes the mapping process of high-dimensional image features to low-dimensional key digital parameters, and performs dynamic digital characterization for different scenarios.

[0067] Alternatively, if Figure 4 As shown, the feature encoder includes a Transformer encoder, a CNN encoder and a deep feature fusion module;

[0068] The Transformer encoder is responsible for modeling global context information, which is used to capture the long-range dependencies and global features of the image. The 4-layer ViT architecture extracts the high-dimensional features of the image, and finally integrates the features through image block embedding to generate global image features.

[0069] The CNN encoder is responsible for extracting local detail features of the image to enhance the model's perception of objects of different sizes and complex scenes. The 4-layer multi-scale CNN module extracts high-dimensional features of the image and integrates the features through image block embedding to generate local image features.

[0070] The deep feature fusion module uses the cross attention mechanism to further interact and fuse the global image features and local image features after layer normalization LN. The cross attention mechanism formula is as follows:

[0071]

[0072] Among them, the output features of the Transformer encoder are used as queries The output features of the CNN encoder are used as the key K CNN With value V CNN , Represents the scaling factor, corresponding to K CNN feature dimension. This fusion method uses global features as queries for main information guidance, and uses local features that enhance details as keys and values ​​to enrich feature expression and capture details, thereby achieving efficient fusion of global and local information. At the same time, in order to make global features dominant, query information is retained, and jump connections are introduced to add them to the features obtained by cross-attention, so as to more effectively fuse global and local features. In addition, in the subsequent integration process, layer normalization LN, multi-layer perceptron MLP and jump connections are introduced to further enhance the information interaction and fusion capabilities between features, and effectively improve the fusion effect of multi-scale features. Finally, through the information integration of the four-layer deep feature fusion module, multi-scale image features with rich expression capabilities are generated.

[0073] Alternatively, if Figure 5 As shown in the figure, the multi-scale CNN module uses convolution kernels of different sizes to capture image feature information within different receptive fields, and combines segmentation, fusion and 1×1 convolution operations to further enhance the expression of image features, and finally generates integrated local image features, including:

[0074] On the one hand, the input image features are extracted from multiple convolution modules with different convolution kernel sizes (7×7, 5×5, 3×3) for multi-scale information. Each convolution module contains two linear layers, one convolution layer and a Relu activation layer. Through these convolution modules, image features are extracted from different receptive fields. After that, the image features under different receptive fields are integrated through channel fusion to form high-dimensional multi-scale features.

[0075] On the other hand, the global information of the input image is effectively extracted through segmentation, channel fusion and 1×1 convolution operations, and the global information is channel-fused with the high-dimensional multi-scale features to obtain local image features with higher expressiveness.

[0076] Alternatively, if Figure 6 As shown, the cross attention mechanism module uses the multi-scale image features of the RGB image as the query Q RGB , used for main information guidance, and the multi-scale image features of the multispectral image are used as the key K Multi Sum value V Multi , used to supplement the expression of detail information and spectral features, perform high-dimensional feature fusion between the two, and generate high-dimensional fused image features. The cross-attention mechanism formula is as follows:

[0077]

[0078] in, Represents the scaling factor, corresponding to K Multi feature dimension.

[0079] In order to better fuse the image sequence and the sensor sequence, it is necessary to avoid excessive bias towards image information during feature fusion due to the high image resolution, thereby masking the role of sensor data. At the same time, in order to further refine the image information, the optimal representation method is flexibly selected to maximize the extraction of effective information in the image. Therefore, the embodiment of the present invention introduces a hybrid expert dynamic digital representation method to convert the image sequence into a digital sequence consistent with the specifications of the sensor sequence, which helps to balance the information contribution of the two, thereby obtaining a more concise and balanced fusion feature.

[0080] Alternatively, if Figure 7 As shown, the hybrid expert digital representation module flexibly adjusts the representation strategy according to different scene requirements and current image features, dynamically generates an adaptive digital representation sequence, and extracts multi-level, multi-perspective visual feature representations by inputting the high-dimensional fused image features into each expert model. At the same time, based on the sensor features generated by the current sensor parameter knowledge base and the high-dimensional fused image features, a gating network is used to dynamically generate corresponding weight coefficients for each expert model, and each expert model is weighted in real time, thereby giving each expert model a different degree of attention according to different scene and task requirements, and finally generating the required digital representation results through feature flattening;

[0081] The digital characterization result can dynamically determine the characterization method based on the current scene and image features. For example, in different scenarios, the characterization results will be presented in different forms, such as material surface coverage percentage or material surface uniformity coefficient. Specifically, the model will select the characterization direction that best suits the current scene based on the current sensor data and image features. In some scenarios, the characterization results will be expressed as material surface coverage percentage (such as 80%, 60%, 70%, etc.), which intuitively reflects the coverage degree of different materials. In some scenarios, the system characterizes the image as a material surface uniformity coefficient (such as 0.7, 0.6, etc.) to describe the degree of uniformity of the material surface. In addition, digital characterization is not limited to these forms. The model can dynamically select different characterization methods and forms of expression according to the needs of different scenarios and real-time image features to provide more accurate and flexible analysis results.

[0082] Alternatively, if Figure 8 As shown in the figure, the input of each expert model is the current high-dimensional fusion image feature. The overall architecture of the expert model includes three parallel feature processing paths. The trunk path uses multiple 3i×3i convolution kernels of different sizes and image normalization to capture image features under different receptive fields. i is the expert model number (the value range is 1...n). The two branch paths use average pooling and maximum pooling to extract the global smooth features and local key features of the image, respectively, to enhance the model's modeling ability for global context information. In addition, both branch paths use fully connected layers and 1×1 convolutions to align features, effectively realize the superposition of the three branch features, and finally generate the expert features processed by each expert model.

[0083] Alternatively, if Fig. 9 As shown, the gated network receives the high-dimensional fused image features and sensor features as inputs, wherein the sensor features are generated by a sensor parameter knowledge base of different scenarios, and the sensor parameter knowledge base includes fixed parameters of multiple sensors involved in the scenarios. The sensor parameters in the sensor parameter knowledge base are mapped to high dimensions through word embedding to generate the required sensor features. The sensor features are processed through n 1×1 convolutions to generate n weight identifiers, where n is the number of expert models. After the high-dimensional fused image features are processed through two fully connected layers and normalization, the image features are further mapped to a higher dimension. The obtained higher-dimensional image features and these weight identifiers are fused into each weight identifier through a self-attention mechanism to generate n weight vectors. Finally, these weight vectors are processed through averaging and normalization to dynamically generate the weight coefficients W of each expert model. 1 ,W 2 ,W 3 …W n,These weight coefficients are used to dynamically activate different branches in the expert model module,,ensuring that the model can select the optimal feature processing combination,for specific sensor parameters, scene requirements, and real-time image results.

[0084] S3. Input the digital characterization sequence and the multi-sensor time series data into a multivariate time series analysis module. The multivariate time series analysis module combines the digital characterization sequence with the multi-sensor time series data to construct a unified multivariate time series analysis framework, fully explore the time dependency and interaction characteristics between different modal data, perform refined state judgment of the material surface, and generate a material surface state analysis sequence.

[0085] Alternatively, if Fig.10 As shown, the multivariate time series analysis module includes a multi-source sequence fusion module and a state analysis module;

[0086] like Fig.11 As shown, the multi-source sequence fusion module performs word embedding on the digital representation sequence and the multi-sensor time series data respectively, maps them to the high-dimensional vector space, generates a corresponding feature mapping matrix, and then interacts with the feature mapping matrix of the digital representation and the feature mapping matrix of any sensor time series through the feature interaction module to generate a mixed feature, which will continue to interact and fuse with other sensor feature mapping matrices in turn to generate the final multi-source fusion feature, which fuses the correlation information between all sequences and provides strong support for the subsequent blast furnace material surface state analysis;

[0087] like Fig.12 As shown in the figure, the feature interaction module first aligns the dimensions of the two feature mapping matrices through the dimension alignment module. The dimension alignment module is composed of 3×3 convolution, Relu activation function and 1×1 convolution, which is intended to extract higher-dimensional features and adjust the number of channels for alignment. After that, the two-to-two interactions between channels are realized in a channel interval interaction manner, and the high-dimensional features of the two sequences are further fused to generate two high-dimensional interaction features. The two high-dimensional interaction features are further integrated with the help of 3×3 convolution with shared weights, which is conducive to the fusion of cross-sequence features. After that, the two high-dimensional interaction features are superimposed to form a mixed feature.

[0088] like Fig.13As shown, the state analysis module performs dimension compression on the multi-source fusion features to reduce feature dimensions and focus on key features. Afterwards, a self-attention mechanism is introduced into the compressed feature information to capture global dependencies and enhance the expressiveness of features. Finally, the features are integrated through a multi-layer perceptron to extract key information and generate the required material surface state analysis sequence. The value of the material surface state analysis sequence is the stability assessment percentage of the real-time material surface state inside the blast furnace after the integration of multi-source information data, which is used to quantify the dynamic stability level of the material surface and provide an accurate representation of the operating status of the blast furnace.

[0089] like Fig.14 As shown, an embodiment of the present invention further provides a metallurgical blast furnace material surface status monitoring system integrating multimodal information, the system comprising:

[0090] An acquisition module 1410 is used to acquire an RGB time-series image sequence and a multi-spectral time-series image sequence of a metallurgical blast furnace charge surface to be monitored;

[0091] The multimodal data fusion and digital representation module 1420 is used to input the RGB time-series image sequence and the multispectral time-series image sequence into the multimodal data fusion and digital representation module, and the multimodal data fusion and digital representation module performs multimodal data fusion on each moment image in the RGB time-series image sequence and the multispectral time-series image sequence frame by frame, extracts multi-scale image features by combining local and global methods, and realizes deep fusion of multi-source images through cross-modal feature interaction to generate high-dimensional fused image features. Combined with the guidance of the sensor parameter knowledge base and the hybrid expert model, the high-dimensional fused image features are dynamically digitally represented, and key digital information is extracted according to different scene requirements to obtain a digital representation sequence of the hybrid image;

[0092] The multivariate timing analysis module 1430 is used to input the digital representation sequence and the multi-sensor timing data into the multivariate timing analysis module. The multivariate timing analysis module combines the digital representation sequence with the multi-sensor timing data to construct a unified multivariate timing analysis framework, fully explore the time dependency and interaction characteristics between different modal data, perform refined state judgment of the material surface, and generate a material surface state analysis sequence.

[0093] An embodiment of the present invention provides a metallurgical blast furnace material surface status monitoring system integrating multimodal information, and its functional structure corresponds to a metallurgical blast furnace material surface status monitoring method integrating multimodal information provided by an embodiment of the present invention, which will not be repeated here.

[0094] Fig.15It is a structural schematic diagram of an electronic device 1500 provided in an embodiment of the present invention. The electronic device 1500 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 1501 and one or more memories 1502, wherein the memory 1502 stores at least one instruction, and the at least one instruction is loaded and executed by the processor 1501 to implement the steps of the above-mentioned metallurgical blast furnace material surface status monitoring method integrating multimodal information.

[0095] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, which can be executed by a processor in a terminal to complete the above-mentioned metallurgical blast furnace charge surface state monitoring method integrating multimodal information. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0096] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0097] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for monitoring the charge surface status of a metallurgical blast furnace by integrating multimodal information, characterized in that: The method comprises: S1, obtaining an RGB time-series image sequence and a multi-spectral time-series image sequence of the material surface of a metallurgical blast furnace to be monitored; S2. Input the RGB time-series image sequence and the multispectral time-series image sequence into a multimodal data fusion and digital representation module. The multimodal data fusion and digital representation module performs multimodal data fusion on each moment image in the RGB time-series image sequence and the multispectral time-series image sequence frame by frame, extracts multi-scale image features by combining local and global methods, and realizes deep fusion of multi-source images through cross-modal feature interaction to generate high-dimensional fused image features. Combined with the guidance of the sensor parameter knowledge base and the hybrid expert model, the high-dimensional fused image features are dynamically digitally represented, and key digital information is extracted according to different scene requirements to obtain a digital representation sequence of the hybrid image. S3. Input the digital characterization sequence and the multi-sensor time series data into a multivariate time series analysis module. The multivariate time series analysis module combines the digital characterization sequence with the multi-sensor time series data to construct a unified multivariate time series analysis framework, fully explore the time dependency and interaction characteristics between different modal data, perform refined state judgment of the material surface, and generate a material surface state analysis sequence.

2. The method according to claim 1, characterized in that The multimodal data fusion and digital representation module includes a feature encoder, a cross-attention mechanism module and a hybrid expert digital representation module; The feature encoder uses a dual-branch network combining CNN and Transformer to extract multi-scale image features of RGB images and multi-scale image features of multi-spectral images; The cross-attention mechanism module dynamically captures the interdependence between the multi-scale image features of the RGB image and the multi-scale image features of the multispectral image, performs fine-grained cross-modal feature alignment and fusion, and generates high-dimensional fused image features; The hybrid expert digital characterization module, through the introduction of the sensor parameter knowledge base, performs multimodal interaction with the high-dimensional fusion image features, and with the guidance of the sensor parameter knowledge, optimizes the mapping process of high-dimensional image features to low-dimensional key digital parameters, and performs dynamic digital characterization for different scenarios.

3. The method according to claim 2, characterized in that The feature encoder includes a Transformer encoder, a CNN encoder and a deep feature fusion module; The Transformer encoder is responsible for modeling global context information, which is used to capture the long-range dependencies and global features of the image. The 4-layer ViT architecture extracts the high-dimensional features of the image, and finally integrates the features through image block embedding to generate global image features. The CNN encoder is responsible for extracting local detail features of the image to enhance the model's perception of objects of different sizes and complex scenes. The 4-layer multi-scale CNN module extracts high-dimensional features of the image and integrates the features through image block embedding to generate local image features. The deep feature fusion module uses the cross attention mechanism to further interact and fuse the global image features and local image features after layer normalization LN. The cross attention mechanism formula is as follows: Among them, the output features of the Transformer encoder are used as query Q Trans , the output feature of the CNN encoder is used as the key K CNN With value V CNN , Represents the scaling factor, corresponding to K CNN feature dimension. This fusion method uses global features as queries for main information guidance, and uses local features that enhance details as keys and values ​​to enrich feature expression and detail capture, thereby achieving efficient fusion of global and local information. At the same time, in order to make global features dominant, query information is retained, and jump connections are introduced to add them to the features obtained by cross-attention, so as to more effectively fuse global and local features. In addition, in the subsequent integration process, layer normalization LN, multi-layer perceptron MLP and jump connections are introduced to further enhance the information interaction and fusion capabilities between features, and effectively improve the fusion effect of multi-scale features. Finally, through the information integration of the four-layer deep feature fusion module, multi-scale image features with rich expression capabilities are generated.

4. The method according to claim 3, characterized in that: The multi-scale CNN module uses convolution kernels of different sizes to capture image feature information within different receptive fields, and combines segmentation, fusion and 1×1 convolution operations to further enhance the expression of image features, and finally generates integrated local image features, including: On the one hand, the input image features are extracted from multiple convolution modules with different convolution kernel sizes for multi-scale information. Each convolution module contains two linear layers, one convolution layer and a Relu activation layer. Through these convolution modules, image features are extracted from different receptive fields. After that, the image features under different receptive fields are integrated through channel fusion to form high-dimensional multi-scale features. On the other hand, the global information of the input image is effectively extracted through segmentation, channel fusion and 1×1 convolution operations, and the global information is channel-fused with the high-dimensional multi-scale features to obtain local image features with higher expressiveness.

5. The method according to claim 2, characterized in that: The cross attention mechanism module uses the multi-scale image features of the RGB image as the query Q RGB , used for main information guidance, and the multi-scale image features of the multispectral image are used as the key K Multi Sum value V Multi , which is used to supplement the expression of detail information and spectral features, perform high-dimensional feature fusion between the two, and generate high-dimensional fused image features. The formula of the cross attention mechanism is as follows: in, Represents the scaling factor, corresponding to K Multi feature dimension.

6. The method according to claim 2, characterized in that The hybrid expert digital representation module flexibly adjusts the representation strategy according to different scene requirements and current image features, and dynamically generates an adaptive digital representation sequence. By inputting the high-dimensional fused image features into each expert model, a multi-level, multi-perspective visual feature representation is extracted. At the same time, based on the sensor features generated by the current sensor parameter knowledge base and the high-dimensional fused image features, a gating network is used to dynamically generate corresponding weight coefficients for each expert model, and each expert model is weighted in real time, so as to give each expert model a different degree of attention according to different scenes and task requirements, and finally generate the required digital representation results through feature flattening.

7. The method according to claim 6, characterized in that The input of each expert model is the current high-dimensional fusion image features. The overall architecture of the expert model includes three parallel feature processing paths. The trunk path uses multiple 3i×3i convolution kernels of different sizes and image normalization to capture image features under different receptive fields. i is the expert model number. The two branch paths use average pooling and maximum pooling respectively to extract the global smooth features and local key features of the image, thereby enhancing the model's modeling ability for global contextual information. In addition, both branch paths use fully connected layers and 1×1 convolutions to align features, effectively realizing the superposition of the three branch features, and finally generating expert features processed by each expert model.

8. The method according to claim 6, characterized in that The gated network receives the high-dimensional fused image features and sensor features as inputs, wherein the sensor features are generated by a sensor parameter knowledge base of different scenarios, the sensor parameter knowledge base includes fixed parameters of multiple sensors involved in the scenarios, and the sensor parameters in the sensor parameter knowledge base are mapped to high dimensions through word embedding to generate the required sensor features. The sensor features are processed through n 1×1 convolutions to generate n weight identifiers, where n is the number of expert models. After the high-dimensional fused image features are processed through two fully connected layers and normalization, the image features are further mapped to a higher dimension. The obtained higher-dimensional image features and these weight identifiers are used to fuse the high-dimensional image feature information into each weight identifier through a self-attention mechanism to generate n weight vectors. Finally, these weight vectors are processed through averaging and normalization to dynamically generate weight coefficients W1, W2, W3…W of each expert model. n ,These weight coefficients are used to dynamically activate different branches in the expert model module,,ensuring that the model can select the optimal feature processing combination,for specific sensor parameters, scene requirements, and real-time image results.

9. The method according to claim 1, characterized in that: The multivariate time series analysis module includes a multi-source sequence fusion module and a state analysis module; The multi-source sequence fusion module performs word embedding on the digital representation sequence and the multi-sensor time series data, respectively, maps them to a high-dimensional vector space, generates a corresponding feature mapping matrix, and then interacts with the feature mapping matrix of the digital representation and the feature mapping matrix of any sensor time series through feature interaction modules to generate mixed features. The mixed features will continue to interact and fuse with other sensor feature mapping matrices in turn to generate the final multi-source fusion features. The multi-source fusion features fuse the correlation information between all sequences, providing strong support for the subsequent blast furnace charge surface state analysis; The feature interaction module first aligns the dimensions of the two feature mapping matrices respectively through a dimension alignment module. The dimension alignment module is composed of a 3×3 convolution, a Relu activation function, and a 1×1 convolution, and is intended to extract higher-dimensional features and adjust the number of channels for alignment. After that, the two-to-two interactions between channels are achieved in a channel interval interaction manner, and the high-dimensional features of the two sequences are further fused to generate two high-dimensional interaction features. The two high-dimensional interaction features are further integrated with the help of a 3×3 convolution with shared weights, thereby facilitating the fusion of cross-sequence features. After that, the two high-dimensional interaction features are superimposed to form a mixed feature. The state analysis module compresses the dimensions of the multi-source fusion features to reduce the feature dimensions and focus on key features. Then, a self-attention mechanism is introduced into the compressed feature information to capture global dependencies and enhance the expressiveness of the features. Finally, the features are integrated through a multi-layer perceptron to extract key information and generate the required material surface state analysis sequence. The value of the material surface state analysis sequence is the stability assessment percentage of the real-time material surface state inside the blast furnace after the integration of multi-source information data, which is used to quantify the dynamic stability level of the material surface and provide an accurate representation of the blast furnace operation state.

10. A metallurgical blast furnace material surface status monitoring system integrating multimodal information, characterized in that: The system comprises: An acquisition module is used to acquire an RGB time-series image sequence and a multi-spectral time-series image sequence of the material surface of a metallurgical blast furnace to be monitored; A multimodal data fusion and digital representation module is used to input the RGB time-series image sequence and the multispectral time-series image sequence into the multimodal data fusion and digital representation module, and the multimodal data fusion and digital representation module performs multimodal data fusion on each moment image in the RGB time-series image sequence and the multispectral time-series image sequence frame by frame, extracts multi-scale image features by combining local and global methods, and realizes deep fusion of multi-source images through cross-modal feature interaction to generate high-dimensional fused image features. Combined with the guidance of the sensor parameter knowledge base and the hybrid expert model, the high-dimensional fused image features are dynamically digitally represented, and key digital information is extracted according to different scene requirements to obtain a digital representation sequence of the hybrid image; A multivariate timing analysis module is used to input the digital representation sequence and the multi-sensor timing data into the multivariate timing analysis module. The multivariate timing analysis module combines the digital representation sequence with the multi-sensor timing data to construct a unified multivariate timing analysis framework, fully explore the time dependency and interaction characteristics between different modal data, perform refined state judgment of the material surface, and generate a material surface state analysis sequence.

Images