Sintered ore FeO content prediction method and system based on multi-modal fusion

Abstract

The invention provides a method and system for predicting the FeO content of sintered ore based on multi-modal fusion, and belongs to the field of industrial intelligent manufacturing and ironmaking quality monitoring. The method comprises the following steps: acquiring image data and time sequence data in a sintering process in real time in a production cycle, and aligning the image data and the time sequence data in the same time window; constructing a sintering process temperature field mechanism model, and performing numerical solution on the mechanism model through a finite difference method; early fusion of the mechanism data and the time sequence data is realized through splicing, mechanism enhanced time sequence data is obtained, and time sequence features are extracted; extracting image features from the image data; and performing deep fusion on the time sequence features and the image features through a cross-attention mechanism to obtain two paths of cross-modal response features, splicing the two paths of cross-modal response features with original modal features corresponding to each mode to obtain combined features as input, and predicting and outputting the FeO content of the sintered ore based on a Transform structure. According to the method, the accuracy and robustness of FeO content prediction are improved.

Description

Technical Field

[0001] This invention belongs to the fields of industrial intelligent manufacturing and ironmaking quality monitoring, specifically involving a method and system for predicting the FeO content of sintered ore based on multimodal fusion. Background Technology

[0002] As a raw material for blast furnace ironmaking, the quality of sinter directly affects the blast furnace's output, energy consumption, and stability. FeO content is a crucial indicator of sinter quality and blast furnace production capacity, thus requiring monitoring of FeO content during the production process. While traditional chemical analysis can accurately determine FeO, it is costly, time-consuming, and results are delayed, making it difficult to provide timely feedback to production control and potentially negatively impacting ironmaking plans and finished product quality. Therefore, developing a rapid, online FeO content prediction model is of great significance for optimizing process parameters and improving blast furnace operating efficiency.

[0003] In actual production, modern sintering lines can obtain multi-source data: firstly, process time-series data, used to characterize the dynamic changes of operating conditions over time; secondly, infrared image data collected from the surface or tail of the sintering line, reflecting the material surface morphology and apparent temperature distribution; in addition, temperature field mechanism models can be established and solved under in-plant operating conditions to obtain relevant thermodynamic state quantities, serving as mechanistic data that is difficult to measure directly and is instead calculated indirectly. However, the above three types of information are highly heterogeneous in terms of sampling methods, spatial resolution, and semantics. Simply relying on any single mode will lead to problems of incomplete information and insufficient robustness.

[0004] Existing soft measurement methods mainly fall into three categories, but all have limitations: single-modal models that rely solely on time series or images are sensitive to noise and operating condition drift, making it difficult to stably map to deep quality indicators such as FeO; temperature field models based on energy conservation and reaction mechanisms are interpretable, but are sensitive to parameters and boundary conditions, and are difficult to calibrate and adapt in the field; existing multimodal fusion models generally use simple splicing, forcibly treating different mass dimensions as similar features, ignoring spatiotemporal semantic alignment. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a method and system for predicting FeO content in sintered ore based on multimodal fusion. The method uses the finite difference method to numerically solve the temperature field mechanism model of the sintering process, then achieves deep fusion of time-series data and image data through a cross-attention mechanism, extracts fused features, and predicts the FeO content of sintered ore based on a Transformer structure. This improves the accuracy and real-time performance of the prediction, enhances prediction precision and robustness, and simultaneously addresses the problems of prediction instability and insufficient generalization.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows: In a first aspect, embodiments of the present invention provide a method for predicting the FeO content of sintered ore based on multimodal fusion, the method comprising the following steps: Step S1: During the production cycle, acquire historical and real-time image data and time-series data of the sintering process of the sintered ore, and align the image data and time-series data within the same time window; Step S2: Construct a temperature field mechanism model for the sintering process; Step S3: The temperature field mechanism model is numerically solved using the finite difference method and historical time series data to obtain solid phase temperature field data as mechanism data. Step S4: Early fusion of mechanistic data and historical time series data is achieved by splicing to obtain mechanistic-enhanced time series data; Step S5: Extract temporal features from mechanism-enhanced temporal data; simultaneously, extract image features from historical image data and align them with the temporal features in terms of dimension and time; and during image feature extraction, the MobileViT module performs patch expansion, using single-frame feature map patches. Step S6: The temporal features and image features are deeply fused through a cross-attention mechanism to obtain two cross-modal response features; Step S7: Concatenate the two cross-modal response features with the time-series features and image features to obtain joint features; use the FeO content at historical moments as historical labels, construct data pairs with the joint features at corresponding moments, and form training and testing sets; Step S8: Construct a prediction model based on the Transformer structure, and train the prediction model using the training set and the test set to obtain a mature prediction model; Step S9: After preprocessing the real-time image data and time-series data, the joint features are obtained. The joint features are then input into the trained prediction model to output the predicted value of FeO content in the sinter at the current moment.

[0007] In a preferred embodiment of the present invention, the image data in step S1 includes: infrared images collected at the sintering belt or tail end; the time series data includes production parameter variables and production process variables; wherein the production parameter variables include at least the batching ratio and ignition temperature, and the production process variables include at least the machine tool speed and the exhaust gas temperature of the bellows.

[0008] In a preferred embodiment of the present invention, step S2, constructing a temperature field mechanism model for the sintering process, includes: S201: Based on the law of conservation of energy, the energy conservation equations for the gas phase and the solid phase are obtained; S202: Under the assumed conditions, the energy conservation equations for the gas and solid phases are simplified to obtain the temperature field control equations. S203: Set up a layered temperature field structure, and deform the temperature field control equations according to the physicochemical reactions that occur in different material layers to construct a temperature field mechanism model based on the layered temperature field structure and the simplified energy conservation equation.

[0009] In a preferred embodiment of the present invention, step S3 involves numerically solving the mechanism model using the finite difference method, including: S301: Set a uniform step size on the time axis and a uniform interval in the material layer thickness direction to form a spatiotemporal grid. Set the initial conditions and boundary conditions of the temperature field mechanism model based on historical time series data and initialize the model. S302: Perform implicit difference calculations on the temperature field mechanism model to obtain the difference equations of the temperature field; S303: Solve the gas and solid phase temperature field parameters at the current time step based on the difference equations and update the temperature field; S304: Based on the updated solid phase temperature, perform region determination for each spatial unit and switch the corresponding control equations and parameters; S305: Determine whether sintering has ended; if it has ended, output the solved solid phase temperature field data as mechanism data; otherwise, return to step S303.

[0010] In a preferred embodiment of the present invention, step S4 includes: S401: Reduce the dimensionality of the mechanism data to a preset dimension by interval sampling; S402: The dimensionality-reduced mechanistic data and historical time series data are synchronously sliced ​​under the same time benchmark as the prediction window, and then spliced ​​according to the channel dimension to form mechanistic-enhanced time series data.

[0011] In a preferred embodiment of the present invention, in step S5: When extracting time series data features, within a unified prediction time window, mechanism-enhanced time series data is input into a time series branch composed of a multi-layer Transformer encoder to extract time series features; The process of extracting image data features involves sequentially stitching together image frames acquired within the same time window into an image sequence, and then extracting image features using a MobileViT visual encoder that incorporates local convolution and global self-attention.

[0012] In a preferred embodiment of the present invention, extracting image data features specifically includes: S501: Consolidate N single-frame image data within a time window in a time series image into an N-frame image sequence data; S502: After extracting local features through convolution and downsampling, N frames of convolutional maps are obtained from the N frames of image sequence data. Then, the N frames of convolutional maps are sequentially fed into M MobileViT modules to extract N frames of feature maps of different sizes. S503: The PL-MobileViT module is used to reduce the dimensionality of the image features obtained from the last MobileViT module to achieve alignment with the temporal features.

[0013] In a preferred embodiment of the present invention, step S6 includes: S601: The obtained temporal features and image features are used as inputs for cross-attention, and the two types of features are linearly mapped to obtain the query and key-value representation: (5) (6) In equations (5) and (6), Q, K, and V represent the query, key, and value vectors, respectively. and Let each mode be a linear weight matrix. , , The dimension of a single attention head; S602: Using time-series features as queries to match the keys and values ​​of image features, and simultaneously using image features as queries to match the keys and values ​​of time-series features, two cross-modal response features are obtained: (7) (8) In equations (7) and (8), softmax represents the attention distribution normalization operation at each time step. Scaling factor and This represents the characteristics of a two-way response.

[0014] In a preferred embodiment of the present invention, the prediction model in step S8 includes an encoder, a decoder, and a regression prediction head; after inputting the joint features into the encoder, the encoder output representation is obtained, and the FeO content at historical time is used as the historical label; at the same time, the encoder output representation and the historical label are input into the decoder; the historical label interacts with the multimodal information of the encoder output representation in the decoder through a cross-attention mechanism to obtain the decoder output features, which are then passed through the regression prediction head to output the predicted value of the sintered FeO content at the current time.

[0015] Secondly, embodiments of the present invention also provide a sinter FeO content prediction system based on multimodal fusion. The system includes a data acquisition module, a time alignment module, a mechanism model construction module, a temperature field solution module, an early fusion module, a temporal feature extraction module, a MobileViT visual encoder, a deep fusion module, a feature combination module, a prediction model construction module, and an FeO content prediction module; wherein... The data acquisition module is used to acquire historical and real-time image data and time-series data of the sintering process of sintered ore during the production cycle. The time alignment module is used to align image data and time series data within the same time window; The mechanism model construction module is used to construct a temperature field mechanism model for the sintering process. The temperature field solving module is used to numerically solve the temperature field mechanism model using the finite difference method and historical time series data, and obtain solid phase temperature field data as mechanism data and send it to the early fusion module. The early fusion module is used to achieve early fusion of mechanistic data and historical time series data by splicing, so as to obtain mechanistic-enhanced time series data. The temporal feature extraction module is used to extract temporal features from mechanism-enhanced temporal data; The MobileViT visual encoder is used to extract image features from historical image data and align them with temporal features in terms of dimension and time. When extracting image features, the MobileViT module performs patch expansion, and the patch used is a single-frame feature map patch. The deep fusion module is used to deeply fuse temporal features and image features through a cross-attention mechanism to obtain two cross-modal response features; The feature concatenation module is used to concatenate two cross-modal response features with time-series features and image features to obtain joint features; the FeO content at historical moments is used as historical labels, and data pairs are constructed with the joint features at corresponding moments to form training sets and test sets; The prediction model building module is used to build a prediction model based on the Transformer structure, and to train the prediction model using a training set and a test set to obtain a mature prediction model. The FeO content prediction module is used to input the joint features obtained after preprocessing real-time image data and time-series data into the trained prediction model and output the predicted value of FeO content in sinter at the current moment.

[0016] The solutions of the embodiments of the present invention have the following beneficial effects: The method and system for predicting FeO content in sintered ore based on multimodal fusion provided in this invention effectively solves the problem of predicting FeO content in sintered ore during the sintering process. By numerically solving the temperature field mechanism model to obtain mechanism data, the thermodynamic state variables output by the mechanism model are combined with time-series data to introduce physical information and enhance the reliability of basic features. By extracting features from image data through MobileViT, the connections between image sequences are explored. Cross-attention deep fusion is used to achieve nonlinear interaction between image features and time-series features at the feature layer, exploring the deep correlation of multimodal data, breaking through the limitations of single-modal information, realizing cross-modal alignment and deep interaction, and improving the robustness of prediction results.

[0017] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of a method for predicting FeO content in sinter based on multimodal fusion provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the MobileViT visual encoder in an embodiment of the present invention; Figure 3 yes Figure 2 A schematic diagram of the MobileViT module in the middle; Figure 4 yes Figure 2 A schematic diagram of the PL-MobileViT module. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. It should be noted that, without conflict, the embodiments and features in the embodiments of the present invention can also be combined with each other.

[0021] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In the description of this invention, the terms "first," "second," "third," "fourth," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0022] This invention provides a method and system for predicting FeO content in sintered ore based on multimodal fusion. The method involves constructing a training dataset based on acquired sintering process image and time-series data; building a temperature field mechanism model for the sintering process; numerically solving the mechanism model using the finite difference method; achieving early fusion of mechanism and time-series data through splicing; extracting time-series and image data features; achieving deep fusion of time-series and image data through a cross-attention mechanism; extracting joint features; and predicting the FeO content of sintered ore based on a Transformer structure.

[0023] like Figure 1 As shown, the method for predicting FeO content in sinter based on multimodal fusion includes the following steps: Step S1: During the production cycle, acquire historical and real-time image data and time-series data of the sintering process of the sintered ore, and align the image data and time-series data within the same time window.

[0024] Specifically, the image data includes infrared images acquired at the sintering belt or tail section; the time-series data includes production parameter variables such as batching ratio and ignition temperature, and production process variables such as machine tool speed and exhaust gas temperature from the bellows. The acquired image data and time-series data are aligned within the same time window, and historical data is used as training data. Aligning and synchronizing image data and time-series data within the same time window ensures a one-to-one correspondence between samples and actual production periods, enhancing the representativeness and spatiotemporal consistency of the training data.

[0025] Step S2: Construct a temperature field mechanism model for the sintering process.

[0026] One possible implementation involves constructing a temperature field mechanism model for the sintering process, including: S201: Based on the law of conservation of energy, the energy conservation equations for the gas phase and the solid phase are obtained.

[0027] It should be noted that, in order to fit the field data and ensure the feasibility of the calculation, a simplification commonly used in engineering is adopted: radiative heat transfer, gas phase mass diffusion and internal heat conduction of solid particles are ignored, and temperature fluctuations along the running direction and laterally are regarded as secondary effects.

[0028] Specifically, a bed temperature field mechanism model is established based on actual production line operating data. The model approximates the sintering machine material bed in the thickness direction (Z-axis direction) as a one-dimensional unsteady heat transfer and exothermic / endothermic reaction system. The sintering machine is modeled as a homogeneous cuboid. Taking any unit volume of the material layer, the sintering process is considered as static unsteady heat transfer within a unit time, resulting in the energy conservation equations for the gas and solid phases: (1) (2) In equations (1) and (2), f g The mass flow rate of the gas. f s For the density of the material layer, v s The flow rate of the solid. ε The porosity of the material layer, C pg Let be the specific heat capacity of the gas. C ps Specific heat capacity of a solid. h u The thermal convection coefficient, S B This represents the total number of particles per unit volume of material layer surface area. T g For gas temperature, T s For solid temperature, Q g It is the heat of gas-phase reaction. Q s It is the heat of reaction in the solid phase. ρ g The density is the gas density.

[0029] S202: Under the assumed conditions, the energy conservation equations for the gas and solid phases are simplified to obtain the temperature field control equations.

[0030] Specifically, the sintering process is considered as an ideal state, neglecting lateral thermal fluctuations and longitudinal heat transfer in a static state. Based on engineering simplification, the following assumptions are made and the equations are simplified: radiative heat transfer, gas-phase mass diffusion, and internal heat conduction of solid particles are neglected; temperature fluctuations along the operating direction and laterally are considered secondary effects. That is, based on these assumptions, secondary heat transfer and some convective transport terms are ignored, while interphase convective heat transfer and reaction heat source terms are retained. The energy conservation equations for the gas and solid phases are simplified, resulting in a concise set of temperature field control equations: (3) (4) In this step, the energy conservation equations for the gas and solid phases are simplified based on the assumptions. The simplified equations reflect the main physical mechanisms of temperature evolution during the sintering process. They are concise yet retain key information, reducing model complexity, improving computational efficiency, and enhancing the model's deployability in industrial settings.

[0031] S203: Set up a layered temperature field structure, and deform the temperature field control equations according to the physicochemical reactions that occur in different material layers to construct a temperature field mechanism model based on the layered temperature field structure and the simplified energy conservation equation.

[0032] It should be noted that, to more realistically reflect the temperature distribution and thermal reaction behavior inside the sintering bed, the temperature field is divided into several physically meaningful sub-regions based on the vertical structural characteristics of the sintering layer in actual production. According to the typical thermal reaction mechanism of the sintering process, the sintering layer is divided into four regions: the sintered ore layer with only heat exchange, the combustion layer where oxidation and decomposition reactions occur, the preheating and drying layer for dehydration, and the super-wet layer where condensation occurs. Different regions differ in heat exchange forms and reaction mechanisms; substituting different reaction coefficients into the equation yields the complete energy conservation equation. The temperature field mechanism model here is a series of layered partial differential equations, with different heat source terms used in different material layers (…). Q s ), heat convection coefficient ( h u The simplified energy conservation equations, i.e. the temperature field control equations, are instantiated to obtain a set of partial differential equations with different coefficients. These are then combined with the layering criteria, boundary conditions, and initial conditions to form a complete and solvable temperature field mechanism model.

[0033] Specifically, in the combustion layer, the oxidation reaction of coke and the decomposition reaction of carbonates proceed vigorously, exhibiting a significant exothermic peak. A corresponding exothermic source term is added to the solid-phase governing equation, where the reaction rate constant is set based on the coke content and oxygen concentration in the plant. In the drying layer, the temperature is relatively low, and there are endothermic processes such as the evaporation of a large amount of free water and the release of crystal water. Latent heat absorption terms are introduced into the gas-phase and solid-phase energy equations, respectively. In the sinter layer and the over-wet layer, the temperature difference between gas and solid is small, and the chemical reaction is weak; sensible heat transfer is the primary consideration. The boundaries of each region are set according to the reaction temperature, from high to low: combustion layer, sinter layer, preheating and drying layer, and over-wet layer.

[0034] This step introduces a layered temperature field structure, which not only fully considers the non-uniformity of the reaction space within the sintering layer but also enhances the model's ability to express the contributions of different heat sources. This modeling approach improves the model's physical interpretability and predictive accuracy.

[0035] Step S3: The temperature field mechanism model is numerically solved using the finite difference method and historical time series data to obtain solid phase temperature field data as mechanism data.

[0036] In one feasible implementation, the temperature field mechanism model is numerically solved using the finite difference method, including: S301: Set a uniform step size on the time axis and a uniform interval in the material layer thickness direction to form a spatiotemporal grid. Set the initial and boundary conditions of the temperature field mechanism model based on historical time series data and initialize the model.

[0037] Specifically, initial conditions include bed height and ignition temperature, boundary conditions include air temperature and initial temperature of sinter, and characteristic parameters of the sinter include porosity, density, and moisture content. Preferably, the time axis step is set to 2 seconds, and the bed thickness interval is set to 2 mm.

[0038] S302: Perform implicit difference calculations on the temperature field mechanism model to obtain the difference equations of the temperature field.

[0039] Specifically, based on the energy conservation equations for the gas and solid phases, an implicit difference scheme is selected, and the time derivative term is discretized using forward difference and the spatial derivative term using central difference. Through variable transposition and algebraic transformation, a temperature update formula with a unified form is obtained.

[0040] S303: Solve the gas and solid phase temperature field parameters at the current time step based on the difference equations and update the temperature field.

[0041] Specifically, based on the constructed difference scheme and the calculated difference coefficients, the coupled differential equations of the gas and solid phases are solved step by step using an implicit iterative method. Within each time step, the heat source term is first calculated based on the current temperature value and reaction rate. Then, the temperature of each spatial node is updated using the difference formula, thus advancing the evolution of the temperature field in the time dimension.

[0042] S304: Based on the updated solid phase temperature, perform region determination for each spatial unit and switch the corresponding control equations and parameters.

[0043] Specifically, during the temperature field solution process, the solid phase temperature at different locations in the material layer gradually increases as time evolves and the heat conduction process progresses. When the solid phase temperature of any calculation unit reaches the preset layer-switching threshold, it is determined that the unit has entered a new reaction region, and at this time, the corresponding reaction entropy and heat exchange parameters need to be switched synchronously.

[0044] S305: Determine whether sintering has ended; if it has ended, output the solved solid phase temperature field data as mechanism data; otherwise, return to step S303.

[0045] Specifically, after each calculation step is completed, it is determined whether the current time step has reached the termination condition of the sintering process. The determination is based on whether the set sintering time has been exceeded. If not, the calculation of the next time step continues, and the process returns to step S303.

[0046] This step, by constructing a complete time- and spatial discrete grid system and combining it with a physics-based parameter setting and dynamic region identification mechanism, can efficiently solve complex temperature evolution processes and achieve accurate characterization of the temperature field over time. This solution process not only reflects the heat transfer behavior and multi-stage reaction characteristics during sintering but also possesses good convergence and stability, making it suitable for industrial sites with different operating conditions and parameter settings.

[0047] Step S4: Early fusion of mechanistic data and historical time series data is achieved by splicing to obtain mechanistic-enhanced time series data.

[0048] In one feasible implementation, S4 specifically includes sub-steps S401 to S402: S401: Reduce the dimensionality of the mechanism data to a preset dimension by interval sampling.

[0049] Specifically, the high-frequency, high-dimensional temperature field simulation results generated by S3 are sampled at intervals along the material layer height dimension. The sampling method involves taking one value every ten points, reducing the dimensionality of the mechanistic data in the feature dimension to obtain the dimensionality-reduced mechanistic data. Compared to the direct full-dimensional stitching in existing technologies, this invention controls the mechanistic modal dimension through interval sampling, avoiding the dimensionality crowding and gradient dominance of temporal features by mechanistic features, thus improving fusion stability and generalization ability. Simultaneously, it reduces computational overhead without sacrificing key thermodynamic information, facilitating online applications.

[0050] S402: The dimensionality-reduced mechanistic data and historical time series data are synchronously sliced ​​under the same time benchmark as the prediction window, and then spliced ​​according to the channel dimension to form mechanistic-enhanced time series data.

[0051] Specifically, based on a unified prediction time window, the mechanistic data and the process time-series data collected on-site are synchronously sliced ​​to ensure strict alignment of the two types of data in the time dimension. Subsequently, the two are stitched together in the feature dimension to form enhanced time-series data that integrates mechanistic information.

[0052] In this step, dimensionality reduction of the mechanistic data to a preset dimension through interval sampling can reduce the computational complexity of subsequent modeling and improve its compatibility with time-series data, avoiding the problem of mechanistic data dominating the main modes. Simultaneously, the concatenation of mechanistic and time-series data forms enhanced time-series data that incorporates mechanistic information, effectively integrating interpretable thermodynamic priors into the data-driven path. This enhances the model's responsiveness and predictive robustness to changes in key operating conditions without introducing additional perceptual latency.

[0053] Step S5: Extract temporal features from mechanism-enhanced temporal data; simultaneously, extract image features from historical image data and align them with temporal features in terms of dimension and time; and during image feature extraction, the MobileViT module performs patch expansion, using single-frame feature map patches.

[0054] In this step, when extracting time series data features, within a unified prediction time window, the mechanism-enhanced time series data is input into a time series branch composed of a multi-layer Transformer encoder to extract time series features.

[0055] The image data feature extraction process involves sequentially stitching together image frames acquired within the same time window into an image sequence, and then extracting image features using a MobileViT visual encoder that incorporates local convolution and global self-attention. The specific steps for extracting image data features include:

[0056] S501: Consolidate N single-frame image data within a time window in a time-series image into an N-frame image sequence data.

[0057] Specifically, within each prediction time window, several frames of sintering belt or tail images acquired in chronological order are selected and stitched together in frame-level chronological order to form an image sequence. The temporal distribution of the image sequence is completely consistent with the temporal data within the corresponding time window, achieving strict alignment of multimodal inputs in the temporal dimension.

[0058] S502: After extracting local features through convolution and downsampling, N frames of convolutional maps are obtained from the N frame image sequence data. Then, the N frames of convolutional maps are sequentially fed into M MobileViT modules to extract N frames of feature maps of different sizes.

[0059] Specifically, the stitched N-frame image sequence data is first input into a convolutional encoder consisting of multiple lightweight convolutions and batch normalization to extract local texture features and edge information. After downsampling and size compression, N-frame convolutional maps corresponding to the N-frame sequence are obtained. Subsequently, the N-frame convolutional maps are fed into a stacked MobileViT module, with downsampling added after each MobileViT module to progressively obtain M N-frame feature maps at different spatial scales. Preferably, the M MobileViT modules can be 2 or 3. The N-frame feature maps first undergo local representation to extract local features. During patch unfolding, a single-frame feature map is used as a patch, treating each frame feature map as a whole as a frame feature patch. These are then arranged sequentially in time to form a cross-frame sequence. During global representation, the Transformer captures the dependencies between frame sequences to obtain a fused frame-level representation. Subsequently, during the patch folding stage, the frame-level representation is written back to the corresponding frame feature map. Finally, size restoration adjusts the spatial and channel scales to the size of the input N-frame feature maps for subsequent alignment and prediction.

[0060] S503: The PL-MobileViT module is used to reduce the dimensionality of the image features obtained from the last MobileViT module to achieve alignment with the temporal features.

[0061] Specifically, unlike the MobileViT module, after obtaining the frame-level representation through global representation, it is compressed into a one-dimensional feature vector by global average pooling in the channel dimension, thereby achieving alignment with temporal features to enable subsequent multimodal fusion operations.

[0062] In this step, the image feature extraction branch combines local texture perception with global temporal modeling capabilities, effectively characterizing complex visual features such as material surface structure and temperature distribution, thus enhancing the image modality's support for FeO content prediction. Through the efficient visual modeling structure of the MobileViT module, deep semantic expression extraction of image sequences is achieved while maintaining low computational complexity, laying a solid foundation for subsequent fusion and prediction.

[0063] Step S6: The temporal features and image features are deeply fused through a cross-attention mechanism to obtain two cross-modal response features.

[0064] In one feasible implementation, S6 specifically includes sub-steps S601 to S602: S601: The obtained temporal features and image features are used as inputs for cross-attention, and the two types of features are linearly mapped to obtain the query and key-value representation: (5) (6) In equations (5) and (6), Q, K, and V represent the query, key, and value vectors, respectively. and Let each mode be a linear weight matrix. , , The dimension of a single attention head.

[0065] S602: Using time-series features as queries to match the keys and values ​​of image features, and simultaneously using image features as queries to match the keys and values ​​of time-series features, two cross-modal response features are obtained: (7) (8) In equations (7) and (8), softmax represents the attention distribution normalization operation at each time step. Scaling factor and This represents the characteristics of a two-way response.

[0066] This step introduces a bidirectional cross-modal attention mechanism, using temporal features and image features as queries to construct response representations to information from another modality, significantly enhancing the deep semantic interaction capabilities between the two modalities. The cross-attention mechanism dynamically adjusts attention weights based on task relevance, enabling temporal features to actively focus on the most discriminative regions in the image; similarly, image features can selectively respond to process variables highly correlated with their semantics. The response features constructed through bidirectional interactive fusion integrate multimodal redundancy and complementary information, providing more expressive feature representations for subsequent prediction tasks. Furthermore, compared to the two-way cross-modal response features, the temporal features output from the original temporal branch and the image features output from the visual branch are used as their respective original modal features. Compared to existing technologies that only perform unidirectional queries or simple concatenation, this invention's bidirectional attention can adaptively highlight complementary information and suppress redundant information, remaining robust even with short-term image degradation or increased temporal noise.

[0067] Step S7 involves concatenating the two cross-modal response features with temporal features and image features to obtain joint features. Compared to existing technologies that only retain response features, this invention retains both original and interaction information, avoiding information bias caused by over-reliance on attention-derived features and improving robustness and interpretability to changes in operating conditions. The FeO content at historical moments is used as historical labels, and data pairs are constructed with the joint features at corresponding moments to form training and testing sets.

[0068] In this step, the temporal features and image features, which serve as the original modal features, are combined with the two cross-modal response features preprocessed by the cross-attention mechanism to obtain the joint features.

[0069] Step S8: Construct a prediction model based on the Transformer structure, and train the prediction model using the training set and test set to obtain a mature prediction model.

[0070] In this step, the prediction model includes an encoder, a decoder, and a regression prediction head; when training the prediction model, the model is trained using the training set and test set composed of the historical data pairs; after training, the input of the prediction model is the joint features obtained from image data and time series data, and the output is the FeO content in the sintered ore.

[0071] Specifically, in the prediction model, the joint features are input into the encoder to obtain the encoder output representation, and the FeO content at historical time is used as the historical label. At the same time, the encoder output representation and the historical label are input into the decoder. The historical label interacts with the multimodal information of the encoder output representation in the decoder through a cross-attention mechanism to obtain the decoder output features, which are then passed through the regression prediction head to output the predicted value of the sintered FeO content at the current time.

[0072] Step S9: After preprocessing the real-time image data and time-series data, the joint features are obtained. The joint features are then input into the trained prediction model to output the predicted value of FeO content in the sinter at the current moment.

[0073] Based on the same idea, this invention also provides a sinter FeO content prediction system based on multimodal fusion. The system includes a data acquisition module, a time alignment module, a mechanism model construction module, a temperature field solution module, an early fusion module, a temporal feature extraction module, a MobileViT visual encoder, a deep fusion module, a feature joint module, a prediction model construction module, and an FeO content prediction module. The data acquisition module is used to acquire historical and real-time image data and time-series data of the sintering process of sintered ore during the production cycle. The time alignment module is used to align image data and time series data within the same time window; The mechanism model construction module is used to construct a temperature field mechanism model for the sintering process. The temperature field solving module is used to numerically solve the temperature field mechanism model using the finite difference method and historical time series data, and obtain solid phase temperature field data as mechanism data and send it to the early fusion module. The early fusion module is used to achieve early fusion of mechanistic data and historical time series data by splicing, so as to obtain mechanistic-enhanced time series data. The temporal feature extraction module is used to extract temporal features from mechanism-enhanced temporal data; The MobileViT visual encoder is used to extract image features from historical image data and align them with temporal features in terms of dimension and time. When extracting image features, the MobileViT module performs patch expansion, and the patch used is a single-frame feature map patch. The deep fusion module is used to deeply fuse temporal features and image features through a cross-attention mechanism to obtain two cross-modal response features; The feature concatenation module is used to concatenate two cross-modal response features with time-series features and image features to obtain joint features; the FeO content at historical moments is used as historical labels, and data pairs are constructed with the joint features at corresponding moments to form training sets and test sets; The prediction model building module is used to build a prediction model based on the Transformer structure, and to train the prediction model using a training set and a test set to obtain a mature prediction model. The FeO content prediction module is used to input the joint features obtained after preprocessing real-time image data and time-series data into the trained prediction model and output the predicted value of FeO content in sinter at the current moment.

[0074] like Figure 2 As shown, in one feasible implementation, the MobileViT visual encoder includes an image sequence data input interface, a convolutional encoder, several MobileViT modules, and a PL-MobileViT module.

[0075] Among them, such as Figure 3 As shown, the MobileViT module includes several sub-modules such as local representation, patch unpacking, global representation, patch folding, and size restoration. In the patch unpacking sub-module, a strategy of using single-frame feature maps as patches is adopted, treating each frame's feature map as a whole as a frame feature patch. Compared to existing technologies that use partial feature maps as patches, this avoids boundary fragmentation and cross-frame misalignment caused by small patch segmentation, improving the stability of temporal alignment. Simultaneously, it allows any frame to directly exchange global information with other frames, which is more conducive to capturing the overall pattern of material surface evolution. Furthermore, it reduces the number of patches, lowers attention computation and memory overhead, simplifies parameter selection and engineering optimization, and facilitates online deployment and cross-modal alignment. Preferably, the M MobileViT modules can be 2 or 3.

[0076] like Figure 4As shown, the PL-MobileViT module includes sub-modules such as local representation, patch unrolling, global representation, and average pooling. The patch unrolling sub-module also employs a strategy of using single-frame feature maps as patches. Compared to existing technologies that rely on additional linear mappings, this invention achieves dimensional alignment while maintaining key discriminative information through global pooling, reducing parameters and computations, improving training and inference efficiency, and facilitating deployment on resource-constrained devices.

[0077] The prediction model includes an encoder, a decoder, and a regression prediction head. In one executable embodiment, the encoder is constructed using a multi-layer Transformer encoder; the regression prediction head may employ a fully connected / MLP linear regression layer to map hidden features to scalar FeO prediction values.

[0078] In this embodiment, each module is implemented using a processor, with additional memory added as needed for storage. The processor can be, but is not limited to, a microprocessor (MPU), a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), other programmable logic devices, discrete gates, transistor logic devices, discrete hardware components, etc. The memory can include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0079] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0080] It should also be noted that the sinter FeO content prediction system based on multimodal fusion described in this embodiment corresponds to the sinter FeO content prediction method based on multimodal fusion. The description and limitations of the method also apply to the system, and will not be repeated here.

[0081] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to complete the aforementioned multimodal fusion method for predicting the FeO content of sintered ore. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0082] The method and system for predicting FeO content in sintered ore based on multimodal fusion (hereinafter referred to as the MF-Transformer method) described in this invention were applied to a production line in a sintered ore plant. Continuous data collection included 18 time-series process variables, 1 image sequence variable (i.e., consecutive video frames), and 1 output variable. Data was collected in December 2021, with the sampling interval based on the time difference between adjacent keyframes in the video frame, totaling 2370 samples, of which 1896 were training samples and 474 were test samples. Specifically, two simulation experiments were set up to verify the effectiveness of this multimodal fusion method for predicting FeO content in sintered ore.

[0083] Firstly, the prediction performance of the prediction method described in this embodiment is verified compared with other soft sensor methods, aiming to evaluate the advantages of this embodiment in terms of prediction accuracy. Representative soft sensor methods are selected for comparison, including traditional single-modal models and simple fusion strategy models, and comparative experiments are conducted on the same dataset and evaluation metrics (RMSE, R²).

[0084] In one feasible implementation, this method was also compared with other types of soft measurement methods.

[0085] The Transformer and ViT methods perform predictions based on time-series data and image data, respectively.

[0086] The EF-Transformer method employs an early fusion strategy that combines mechanistic data with time-series data for prediction.

[0087] The DF-Transformer method employs a deep fusion strategy that combines temporal and image data across attention exchanges for prediction.

[0088] The MF-Transformer method in this embodiment employs a multi-level fusion approach, first fusing mechanistic data with temporal data to obtain enhanced temporal data, and then performing deep fusion of this temporal data with image data across attention to achieve prediction.

[0089] The CNN-LSTM method and LF-Transformer are used as comparative experiments with other fusion strategies. The CNN-LSTM method uses CNN to extract image features, and then fuses the extracted features with temporal data and inputs them into LSTM for prediction, realizing early fusion of temporal data and image data. The LF-Transformer model uses trained ResNet18 and Transformer to predict image and temporal data independently, and then fuses the prediction results to achieve late fusion.

[0090] The prediction results are listed in Table 1.

[0091] Table 1. Prediction results of different soft sensing methods As shown in Table 1, among these soft measurement prediction methods, the MF-Transformer method in this embodiment has the best prediction performance with an RMSE of 0.0348 and an R² of 0.849. It effectively leverages the complementary advantages of various modal information and fully verifies the effectiveness and advancement of this method in multimodal industrial data modeling. In particular, it demonstrates strong prediction accuracy and generalization ability under complex working conditions, and has good engineering application value.

[0092] Secondly, to verify the generalization of the MF-Transformer method in this embodiment to other architectures, the multimodal fusion framework is embedded in network architectures different from Transformer to verify the effectiveness and universality of the multimodal fusion framework in the MF-Transformer method of this embodiment.

[0093] In one feasible implementation, the multimodal fusion framework of the MF-Transformer method in this embodiment is embedded into the following backbone networks with different structures, including but not limited to Gateformer and Sentinel. While maintaining consistent settings for input data, training epochs, and learning rates across different backbone models, the generalization prediction ability of the models is evaluated. The results are shown in Table 2.

[0094] Table 2 Among the three architectures, the single-modal prediction strategy based on time-series data, the deep fusion strategy based on the fusion of image data and time-series data, and the multi-level fusion strategy based on the fusion of mechanistic data, time-series data and image data proposed in this invention were compared. The method used in this invention achieved the best results, which fully demonstrates that the method of this invention has strong architecture independence and can be efficiently compatible with existing mainstream deep learning frameworks, providing feasible support for their wide application in industrial scenarios.

[0095] As can be seen from the above technical solutions, the method and system for predicting FeO content in sintered ore based on multimodal fusion described in the embodiments of the present invention are constructed based on the Transformer system at the model level. On the basis of the original temporal encoding and visual encoding, an early fusion channel with temperature field mechanism prior and a bidirectional cross-attention fusion module are added. This not only utilizes mechanism prior to improve the physical consistency of input representation, but also suppresses redundancy and highlights complementary information through the attention mechanism, thereby significantly improving prediction accuracy and robustness while ensuring interpretability. At the same time, it is adaptable to different sequence backbones, has good architectural versatility and engineering deployment feasibility, and effectively improves the problems of unstable prediction and insufficient generalization of existing single-modal or shallow fusion methods under complex working conditions.

[0096] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed, and is not intended to limit the scope of the claimed invention, but merely to illustrate preferred embodiments of the invention. Those skilled in the art should understand that the scope of the invention is not limited to the specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

Claims

1. A method for predicting FeO content in sintered ore based on multimodal fusion, characterized in that, The method includes the following steps: Step S1: During the production cycle, acquire historical and real-time image data and time-series data of the sintering process of the sintered ore, and align the image data and time-series data within the same time window; Step S2: Construct a temperature field mechanism model for the sintering process; Step S3: The temperature field mechanism model is numerically solved using the finite difference method and historical time series data to obtain solid phase temperature field data as mechanism data. Step S4: Early fusion of mechanistic data and historical time series data is achieved by splicing to obtain mechanistic-enhanced time series data; Step S5: Extract temporal features from mechanism-enhanced temporal data; simultaneously, extract image features from historical image data and align them with the temporal features in terms of dimension and time; and during image feature extraction, the MobileViT module performs patch expansion, using single-frame feature map patches. Step S6: The temporal features and image features are deeply fused through a cross-attention mechanism to obtain two cross-modal response features; Step S7: Concatenate the two cross-modal response features with the temporal features and image features to obtain joint features; The FeO content at historical moments is used as historical labels, and data pairs are constructed with the joint features at the corresponding moments to form training and testing sets. Step S8: Construct a prediction model based on the Transformer structure, and train the prediction model using the training set and the test set to obtain a mature prediction model; Step S9: After preprocessing the real-time image data and time-series data, the joint features are obtained. The joint features are then input into the trained prediction model to output the predicted value of FeO content in the sinter at the current moment.

2. The method according to claim 1, characterized in that, The image data in step S1 includes: infrared images collected at the sintering belt or tail end; the time-series data includes production parameter variables and production process variables; wherein the production parameter variables include at least the batching ratio and ignition temperature, and the production process variables include at least the machine tool speed and the exhaust gas temperature of the air box.

3. The method according to claim 1, characterized in that, Step S2 involves constructing a temperature field mechanism model for the sintering process, including: S201: Based on the law of conservation of energy, the energy conservation equations for the gas phase and the solid phase are obtained; S202: Under the assumed conditions, the energy conservation equations for the gas and solid phases are simplified to obtain the temperature field control equations. S203: Set up a layered temperature field structure, and deform the temperature field control equations according to the physicochemical reactions that occur in different material layers to construct a temperature field mechanism model based on the layered temperature field structure and the simplified energy conservation equation.

4. The method according to claim 1, characterized in that, Step S3 involves numerically solving the mechanism model using the finite difference method, including: S301: Set a uniform step size on the time axis and a uniform interval in the material layer thickness direction to form a spatiotemporal grid. Set the initial conditions and boundary conditions of the temperature field mechanism model based on historical time series data and initialize the model. S302: Perform implicit difference calculations on the temperature field mechanism model to obtain the difference equations of the temperature field; S303: Solve the gas and solid phase temperature field parameters at the current time step based on the difference equations and update the temperature field; S304: Based on the updated solid phase temperature, perform region determination for each spatial unit and switch the corresponding control equations and parameters; S305: Determine whether sintering has ended; if it has ended, output the solved solid phase temperature field data as mechanism data; otherwise, return to step S303.

5. The method according to claim 1, characterized in that, Step S4 includes: S401: Reduce the dimensionality of the mechanism data to a preset dimension by interval sampling; S402: The dimensionality-reduced mechanistic data and historical time series data are synchronously sliced ​​under the same time benchmark as the prediction window, and then spliced ​​according to the channel dimension to form mechanistic-enhanced time series data.

6. The method according to claim 1, characterized in that, In step S5: When extracting time series data features, within a unified prediction time window, mechanism-enhanced time series data is input into a time series branch composed of a multi-layer Transformer encoder to extract time series features; The process of extracting image data features involves sequentially stitching together image frames acquired within the same time window into an image sequence, and then extracting image features using a MobileViT visual encoder that incorporates local convolution and global self-attention.

7. The method according to claim 6, characterized in that, Extracting image data features specifically includes: S501: Consolidate N single-frame image data within a time window in a time series image into an N-frame image sequence data; S502: After extracting local features through convolution and downsampling, N frames of convolutional maps are obtained from the N frames of image sequence data. Then, the N frames of convolutional maps are sequentially fed into M MobileViT modules to extract N frames of feature maps of different sizes. S503: The PL-MobileViT module is used to reduce the dimensionality of the image features obtained from the last MobileViT module to achieve alignment with the temporal features.

8. The method according to claim 1, characterized in that, Step S6 includes: S601: The obtained temporal features and image features are used as inputs for cross-attention, and the two types of features are linearly mapped to obtain the query and key-value representation: (5) (6) In equations (5) and (6), Q, K, and V represent the query, key, and value vectors, respectively. and Let each mode be a linear weight matrix. , , The dimension of a single attention head; S602: Using time-series features as queries to match the keys and values ​​of image features, and simultaneously using image features as queries to match the keys and values ​​of time-series features, two cross-modal response features are obtained: (7) (8) In equations (7) and (8), softmax represents the attention distribution normalization operation at each time step. Scaling factor and This represents the characteristics of a two-way response.

9. The method according to claim 1, characterized in that, The prediction model in step S8 includes an encoder, a decoder, and a regression prediction head. After inputting the joint features into the encoder, the encoder output representation is obtained, and the FeO content at historical moments is used as historical labels. At the same time, the encoder output representation and historical labels are input into the decoder. The historical labels interact with the multimodal information of the encoder output representation in the decoder through a cross-attention mechanism to obtain the decoder output features, which are then passed through the regression prediction head to output the predicted value of the sintered FeO content at the current moment.

10. A sinter FeO content prediction system based on multimodal fusion, characterized in that, The system includes a data acquisition module, a time alignment module, a mechanism model construction module, a temperature field solution module, an early fusion module, a temporal feature extraction module, a MobileViT visual encoder, a deep fusion module, a feature joint module, a prediction model construction module, and an FeO content prediction module; among which, The data acquisition module is used to acquire historical and real-time image data and time-series data of the sintering process of sintered ore during the production cycle. The time alignment module is used to align image data and time series data within the same time window; The mechanism model construction module is used to construct a temperature field mechanism model for the sintering process. The temperature field solving module is used to numerically solve the temperature field mechanism model using the finite difference method and historical time series data, and obtain solid phase temperature field data as mechanism data and send it to the early fusion module. The early fusion module is used to achieve early fusion of mechanistic data and historical time series data by splicing, so as to obtain mechanistic-enhanced time series data. The temporal feature extraction module is used to extract temporal features from mechanism-enhanced temporal data; The MobileViT visual encoder is used to extract image features from historical image data and align them with temporal features in terms of dimension and time. When extracting image features, the MobileViT module performs patch expansion, and the patch used is a single-frame feature map patch. The deep fusion module is used to deeply fuse temporal features and image features through a cross-attention mechanism to obtain two cross-modal response features; The feature concatenation module is used to concatenate two cross-modal response features with time-series features and image features to obtain joint features; the FeO content at historical moments is used as historical labels, and data pairs are constructed with the joint features at corresponding moments to form training sets and test sets; The prediction model building module is used to build a prediction model based on the Transformer structure, and to train the prediction model using a training set and a test set to obtain a mature prediction model. The FeO content prediction module is used to input the joint features obtained after preprocessing real-time image data and time-series data into the trained prediction model and output the predicted value of FeO content in sinter at the current moment.

Images