Continuous casting quality data processing method, system and equipment based on multi-mode cognitive reasoning, medium and program product

Abstract

The invention relates to the technical field of iron and steel smelting, in particular to a continuous casting quality data processing method based on multi-modal cognitive reasoning, which comprises the following steps of: synchronously acquiring multi-source sensor data, performing time alignment, format standardization, noise filtering and normalization processing, extracting image / thermal image spatial features through CNN (Convolutional Neural Network), and calculating the continuous casting quality data according to the image / thermal image spatial features. The LSTM is combined with frequency domain analysis to extract vibration / sound time sequence features, cross-modal alignment fusion is realized through feature vectorization, quality prediction, anomaly detection and real-time deployment are realized based on a deep neural network and an attention mechanism by fusing process knowledge, and then slab defect causes are accurately identified by integrating a multi-modal reasoning result. Through time synchronization, format unification, noise filtering and data filling, the problem of heterogeneity of multi-source sensor data is solved, and time dislocation and noise interference are eliminated; abnormal data are cleaned by an outlier removal technology, the reliability of input data and the model robustness are improved, and efficient cooperative processing of multi-modal data is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of steel smelting, and specifically to a continuous casting quality data processing method, system, device, medium and program product based on multi-modal cognitive reasoning. Background Art

[0002] Continuous casting is an advanced casting method. Its principle is to continuously pour molten metal into a special metal mold called a mold. The solidified casting is continuously pulled out from the other end of the mold, and it can obtain castings of any length or a specific length. Continuous casting is a key link in steel production, and the efficient processing of its quality data is crucial for improving product quality and reducing defects.

[0003] After retrieval, the Chinese patent publication number is: CN118193684B, which discloses a multi-modal reasoning method and device based on a large language model and a knowledge graph. By dividing the training reasoning problem into text data and image data; according to the text data and image data, retrieving through a preset multi-modal knowledge graph to obtain a multi-modal knowledge subgraph; constructing a model based on the network structure of the encoder and the network structure of the adapter to obtain the MR-MKG framework; adding the MR-MKG framework to the large language model; using the text data, image data and multi-modal knowledge subgraph to train and optimize the multi-modal reasoning model to be trained to obtain a multi-modal reasoning model; according to the target reasoning problem, reasoning through the multi-modal reasoning model to obtain a reasoning result. However, the technical field of this invention is mainly multi-modal reasoning based on a large language model and a knowledge graph, and can obtain multi-modal knowledge graph retrieval. This technical field is not applicable to reasoning in continuous casting.

[0004] The Chinese patent publication number is: CN117764114B, which discloses a high-performance multi-modal large model reasoning system and method. The system includes a multi-modal accelerated reasoning unit, a search unit, a cache unit and a database; the multi-modal accelerated reasoning unit determines a multi-modal model and generates a reasoning result according to multi-modal information. The search unit searches for the reasoning result in the database according to the multi-modal information. The cache unit saves the reasoning result to the database. The present invention supports various multi-modal component acceleration capabilities, provides a caching ability, provides the ability to flexibly combine different multi-modal model component reasoning pipelines, provides a distributed deployment ability, effectively saves computing power, reduces the reasoning time, improves the system throughput, and enhances the user experience. However, the technical field of this invention is mainly the reasoning system and method of a high-performance multi-modal large model, which relates to the technical field of machine learning, but cannot effectively solve the reasoning problem in the continuous casting scenario. Summary of the Invention

[0005] In view of the deficiencies of the prior art, the present invention provides a continuous casting quality data processing method, system, device, medium and program product based on multi-modal cognitive reasoning, which solves the problem that current traditional continuous casting algorithms often fail to consider all influencing factors in complex production environments, resulting in low accuracy of continuous casting quality reasoning.

[0006] To achieve the above objectives, the present invention is realized through the following technical solutions: A continuous casting quality data processing method based on multi-modal cognitive reasoning, comprising the following steps:

[0007] S1. Multi-modal data integration and processing

[0008] First, the multi-modal data integration module synchronously collects data from different sensors, and performs time synchronization, format unification, and noise filtering on the collected data. Through data preprocessing, these multi-modal data are normalized to achieve the effect of data synchronization.

[0009] S2. Feature extraction and fusion

[0010] Through an architecture that combines a convolutional neural network and a long short-term memory network, corresponding feature extraction is performed through different extraction feature techniques for different data sources, and the feature fusion layer integrates features of different modalities to form a feature vector for a comprehensive description of the continuous casting process.

[0011] S3. Multi-modal cognitive reasoning

[0012] Based on the fused feature vector, deep neural networks are used for reasoning, including classification, regression, or anomaly detection tasks. At the same time, the existing results are deployed to the platform to achieve online multi-modal cognitive reasoning, and by training a neural network with a supervised learning algorithm, the prediction of continuous casting quality is realized.

[0013] S4. Quality data processing

[0014] The reasoning result enters the quality discrimination link of the slab, and corresponding quality discrimination is performed to achieve a comprehensive multi-modal determination effect, effectively analyze the quality anomalies of the current continuous casting machine, and form anomaly determination results in various modes and modalities.

[0015] Preferably, in the S1 step, the multi-modal data is also filled with blanks by adopting a data filling method, and the data is filtered by adopting an outlier removal method to obtain clean and tidy data.

[0016] Preferably, the S2 step is specifically as follows:

[0017] The convolutional neural network is used to extract spatial features from image and thermal image data, and the long short-term memory network is used to extract temporal features from time series data. Among them, in the extraction process of the long short-term memory network, vibration and sound signals adopt the method of frequency conversion to extract corresponding features, which can achieve an effective feature extraction effect. The extracted features are stored and integrated in a vectorized manner, that is, the outermost spatial vector features of the model data source extracted from the convolutional neural network for image and thermal image data are stored in a one-dimensional vector manner. Similarly, for the processing of time series data, the outermost vector of the long short-term memory network is also stored in a one-dimensional vector of the feature vector of the data through the method of feature change, so that feature values of the same dimension and information can be obtained, and then enter the feature fusion layer for integration to form a feature vector for comprehensively describing the continuous casting process.

[0018] Preferably, the inference architecture in the step S3 is specifically as follows:

[0019] The multi-modal cognitive inference engine is based on existing multi-modal data, combines the knowledge of existing industry experts and the process characteristics of existing defect causes and defect mechanisms, combines with the deep learning framework, and combines the convolutional neural network, long short-term memory network and attention mechanism in the multi-modal to process the spatial and time series features of the convolutional neural network and long short-term memory network respectively;

[0020] The attention mechanism is used to weight the importance of different modal features to improve the accuracy of inference.

[0021] Preferably, the inference process in the step S3 is specifically as follows:

[0022] Image and video data enter the visual abstraction layer through video encoding, and at the same time, time series data is also connected to the visual abstraction layer. In addition, text data is connected through text encoding, and voice data is connected to the upper layer through voice encoding. These data are all aggregated into the multi-modal adaptive module, and then input into the final result through the feedforward-feedback network.

[0023] Preferably, a continuous casting quality data processing system based on multi-modal cognitive inference includes:

[0024] The multi-modal data integration module monitors and collects various parameters in the continuous casting production process based on various types of sensors;

[0025] The feature extraction and fusion module is used to perform corresponding feature extraction and fusion on the data source;

[0026] The multi-modal cognitive inference module is based on the fused feature vector and uses a deep neural network for inference to realize the prediction of continuous casting quality.

[0027] Preferably, the multimodal data integration module includes, but is not limited to:

[0028] A high-resolution camera for capturing the surface image of the continuous casting billet, the tundish pouring video image during the continuous casting production process, the submerged entry nozzle breakage video image during the continuous casting production process, and the segment spray water video image during the continuous casting process;

[0029] An infrared thermal imager for collecting the temperature distribution data of the continuous casting billet;

[0030] A vibration sensor for recording the vibration signal of the continuous casting machine;

[0031] A sound sensor for obtaining the sound characteristics during the continuous casting process.

[0032] Preferably, a continuous casting quality data processing device based on multimodal cognitive reasoning includes:

[0033] A multimodal sensor array for real-time collecting multimodal data during the continuous casting process;

[0034] An edge synchronization controller for realizing multi-sensor time alignment and performing data normalization and noise filtering processing;

[0035] A computing server for integrated processing of multi-source data;

[0036] A process parameter adjustment actuator for performing process adjustment of the production line according to the quality prediction result.

[0037] The present invention provides a continuous casting quality data processing method, system, device, medium and program product based on multimodal cognitive reasoning. It has the following beneficial effects:

[0038] 1. The present invention solves the heterogeneity problem of multi-source sensor data, eliminates time misalignment and noise interference through time synchronization, format unification, noise filtering and data filling; the outlier removal technology cleans abnormal data, improves the reliability of the input data and the robustness of the model, and ensures the efficient collaborative processing of multimodal data.

[0039] 2. The present invention extracts the surface defect and temperature field distribution characteristics of the casting billet from the image / thermal image data by using a convolutional neural network, captures the spatial local correlation, and then extracts the periodicity and transient anomalies from the vibration and sound signals through a long short-term memory network combined with Fourier transform / wavelet transform to enhance the time-dependent modeling; finally, the spatial feature vector output by the CNN and the temporal feature vector output by the LSTM are unified into a one-dimensional same-dimensional vector, eliminating the modal difference, realizing the alignment of the feature space, and achieving the accurate extraction and fusion of spatio-temporal features.

[0040] 3. In the feature fusion stage of the present invention, the attention mechanism is used to automatically assign weights to different modality features, enhancing the decision-making contribution of key modalities; the process expert rules and defect mechanism models are fused to constrain the output of the deep learning black box, improving the credibility and interpretability of the inference results; through the joint training of defect type recognition, defect size prediction, and anomaly detection, end-to-end quality assessment is achieved, meeting the requirements of high-precision multi-modal joint inference. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a schematic diagram of the overall process of the present invention;

[0042] Figure 2 is a schematic diagram of the feature vector inference process of the present invention;

[0043] Figure 3 is a schematic diagram of the process of the multi-modal adaptive module of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0045] Embodiment 1:

[0046] As Figures 1-3 shown, the embodiment of the present invention provides a continuous casting quality data processing method based on multi-modal cognitive inference, including the following steps:

[0047] S1. Multi-modal data integration and processing

[0048] First, the multi-modal data integration module synchronously collects data from different sensors, and performs time synchronization, format unification, and noise filtering on the collected data. Through data preprocessing, these multi-modal data are normalized to achieve the effect of data synchronization; in addition, the multi-modal data are filled with blanks by data filling, and the data are filtered by outlier removal to obtain clean and tidy data.

[0049] S2. Feature extraction and fusion

[0050] The convolutional neural network is used to extract spatial features from image and thermal image data, and the long short-term memory network is used to extract temporal features from time series data (such as vibration and sound signals); among them, in the extraction process of the long short-term memory network, the vibration and sound signals adopt the frequency conversion method to extract the corresponding features, which can achieve an effective feature extraction effect; the extracted features are stored and integrated in a vectorized manner, that is, the outermost spatial vector features of the model data source extracted from the convolutional neural network image and thermal image data are stored in a one-dimensional vector manner. Similarly, for the processing of time series data, the outermost vector of the long short-term memory network is also stored as a one-dimensional vector of the feature vector of the data through the feature change method, so that feature values of the same dimension and information can be obtained, and then enter the feature fusion layer for integration to form a feature vector for comprehensively describing the continuous casting process.

[0051] S3. Multimodal Cognitive Reasoning

[0052] Based on the fused feature vector, a deep neural network is used for reasoning, including classification, regression or anomaly detection tasks. At the same time, the existing results are deployed to the platform to realize online multimodal cognitive reasoning. By training a neural network with a supervised learning algorithm, the prediction of continuous casting quality is realized, such as surface defects, internal cracks, composition segregation, etc.

[0053] S4. Quality Data Processing

[0054] The reasoning result enters the quality discrimination link of the slab to perform corresponding quality discrimination, achieving a comprehensive multimodal determination effect, effectively analyzing the quality anomalies of the current continuous casting machine, and forming anomaly determination results in various modes and modalities.

[0055] The reasoning architecture in step S3 is specifically as follows:

[0056] The multimodal cognitive reasoning engine is based on the existing multimodal data, combines the knowledge of existing industry experts and the process characteristics of existing defect causes and defect mechanisms, combines the deep learning framework, and combines the convolutional neural network, long short-term memory network and attention mechanism in the multimodal to process the spatial and time series features of the convolutional neural network and long short-term memory network respectively;

[0057] The attention mechanism is used to weight the importance of different modal features and improve the accuracy of reasoning.

[0058] The reasoning process in step S3 is specifically as follows:

[0059] Such as Figure 2As shown in the figure, image and video data enter the visual abstraction layer through video encoding. At the same time, time series data is also connected to the visual abstraction layer. In addition, text data is connected through text encoding, and voice data is connected to the upper layer through voice encoding. These data are all aggregated into the multi-modal adaptive module and then input into the final result through the feedforward-feedback network.

[0060] The architecture of the multi-modal adaptive module is as Figure 3 shown. Main information such as images, voices, and texts is connected to the preprocessing modules of Norm0 and Norm1 through input information to standardize the information. The data after information standardization is respectively connected to the networks with weights WV0 and WV1. At the same time, the data preprocessed and standardized by Norm1 is connected to the networks with weights WK0 and WK1. The weight networks connected on the left are directly connected to the tensor MatMul matrix multiplier, while the networks with weights WK0 and WK1 and the network with weight WQ are simultaneously connected to the MatMul&Scale matrix-vector multiplier. The result is then input into Softmax for classification and then into the MatMul multiplier. Finally, the result of the multiplier enters the next link through linear transformation. At the same time, in the link of inputting the multi-modal model, it is connected to the linear network through feedforward closed-loop and then connected to the post-processing modules of Norm0 and Norm1 again, and finally the cross-modal feature result is obtained.

[0061] Embodiment 2:

[0062] The embodiment of the present invention provides a continuous casting quality data processing system based on multi-modal cognitive reasoning, including:

[0063] A multi-modal data integration module that monitors and collects various parameters in the continuous casting production process based on various types of sensors;

[0064] A feature extraction and fusion module for performing corresponding feature extraction and fusion on the data source;

[0065] A multi-modal cognitive reasoning module that uses a deep neural network for reasoning based on the fused feature vectors to achieve the prediction of continuous casting quality.

[0066] The multi-modal data integration module includes but is not limited to:

[0067] A high-resolution camera for capturing the surface image of the continuous casting billet, the tundish pouring video image during the continuous casting production process, the long nozzle breakage video image during the continuous casting production process, and the segment spray water video image during the continuous casting process;

[0068] An infrared thermal imager for collecting the temperature distribution data of the continuous casting billet;

[0069] A vibration sensor for recording the vibration signal of the continuous casting machine;

[0070] A sound sensor for acquiring the sound characteristics of the continuous casting process.

[0071] Embodiment III:

[0072] The embodiment of the present invention provides a continuous casting quality data processing device based on multi-modal cognitive reasoning, including:

[0073] A multi-modal sensor array for real-time acquisition of multi-modal data in the continuous casting process;

[0074] An edge synchronization controller for realizing time alignment of multiple sensors and performing data normalization and noise filtering processing;

[0075] A computing server for integrated processing of multi-source data;

[0076] A process parameter adjustment actuator for performing process adjustment of the production line according to the quality prediction result.

[0077] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A continuous casting quality data processing method based on multimodal cognitive reasoning, characterized in that: The following steps are involved: S1. Multimodal data integration processing First, the multimodal data integration module is used to synchronously collect data from different sensors, and the collected data is synchronized in time, formatted, and noise filtered. Through data preprocessing, these multimodal data are normalized to achieve data synchronization. S2. Feature extraction and fusion Through the architecture combining convolutional neural network and long short-term memory network, different data sources are used to extract features according to different feature extraction technologies. The feature fusion layer integrates the features of different modes to form a feature vector that comprehensively describes the continuous casting process. S3. Multimodal cognitive reasoning Based on the fused feature vectors, deep neural networks are used for reasoning, including classification, regression or anomaly detection tasks. At the same time, the existing results are deployed to the platform to realize online multimodal cognitive reasoning. By training neural networks with supervised learning algorithms, the continuous casting quality can be predicted. S4. Quality data processing The inference results enter the slab quality judgment link, and the corresponding quality judgment is carried out to achieve a comprehensive multi-modal judgment effect, effectively analyze the quality abnormalities of the current continuous casting machine, and form abnormal judgment results under multiple modes and modal conditions.

2. The method for processing continuous casting quality data based on multimodal cognitive reasoning according to claim 1, characterized in that: In the step S1, the multimodal data is filled with blanks by using data filling, and the data is filtered by using outlier removal to obtain clean and tidy data.

3. The method for processing continuous casting quality data based on multimodal cognitive reasoning according to claim 1, characterized in that: The S2 step is specifically as follows: Convolutional neural networks are used to extract spatial features from images and thermal imaging data, and long short-term memory networks are used to extract temporal features from time series data. The long short-term memory network extracts vibration and sound signals in a frequency conversion manner to extract corresponding features, which can achieve an effective feature extraction effect. The extracted features are stored and integrated in a vectorized manner, that is, the outermost spatial vector features of the model data source extracted from the convolutional neural network image and thermal imaging data are stored in a one-dimensional vector manner. Similarly, the processing method for time series data also stores the outermost vector of the long short-term memory network as a one-dimensional vector of the data feature vector through feature changes. In this way, the feature values ​​of the same dimension and information can be obtained, and then enter the feature fusion layer for integration to form a feature vector that comprehensively describes the continuous casting process.

4. The method for processing continuous casting quality data based on multimodal cognitive reasoning according to claim 1, characterized in that: The reasoning architecture in the S3 step is as follows: The multimodal cognitive reasoning engine is based on existing multimodal data, combined with the knowledge of existing industry experts and the process characteristics of existing defect causes and defect mechanisms, combined with a deep learning framework, and combined with convolutional neural networks, long short-term memory networks and attention mechanisms to process spatial and time series features in the multimodal convolutional neural networks and long short-term memory networks respectively; The attention mechanism is used to weight the importance of features of different modalities and improve the accuracy of reasoning.

5. The method for processing continuous casting quality data based on multimodal cognitive reasoning according to claim 1, characterized in that: The reasoning process in step S3 is as follows: Image and video data enter the visual abstraction layer through video encoding, and time series data is also connected to the visual abstraction layer. In addition, text data is connected through text encoding, and voice data is connected to the previous layer through voice encoding. All of these data are aggregated into the multimodal adaptive module and then input into the final result through the feedforward feedback network.

6. A continuous casting quality data processing system based on multimodal cognitive reasoning, characterized in that: include: Multimodal data integration module, which monitors and collects various parameters in the continuous casting production process based on multiple types of sensors; Feature extraction and fusion module, used to extract and fuse corresponding features of data sources; The multimodal cognitive reasoning module uses a deep neural network for reasoning based on the fused feature vector to predict the continuous casting quality.

7. The continuous casting quality data processing system based on multimodal cognitive reasoning according to claim 6 is characterized by: The multimodal data integration module includes but is not limited to: High-resolution camera, used to capture the surface image of continuous casting billet, video image of ladle pouring during continuous casting production, video image of shroud damage during continuous casting production, and video image of water spraying from fan-shaped segments during continuous casting; Infrared thermal imager, used to collect temperature distribution data of continuous casting billets; Vibration sensor, used to record the vibration signal of the continuous casting machine; Sound sensor, used to acquire the sound characteristics of the continuous casting process.

8. A continuous casting quality data processing device based on multimodal cognitive reasoning, characterized in that: include: Multimodal sensor array for real-time acquisition of multimodal data during continuous casting; Edge synchronization controller for multi-sensor time alignment and data normalization and noise filtering; Computing server, used for integrated processing of multi-source data; The process parameter adjustment actuator is used to execute the quality prediction results to make process adjustments on the production line.

9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the comment generation information processing method as described in any one of claims 1-5 are implemented.

11. A computer program product, characterized in that: When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the steps of the continuous casting quality data processing method as described in any one of claims 1-5.

Images