Abnormal Condition Recognition Method and System for Rotary Kiln Based on Multimodal Data Fusion

Abstract

Method and system for identifying abnormal conditions of rotary kilns based on multimodal data fusion. The method includes collecting a sample set of rotary kiln operating conditions; a feature extraction module extracts features of a flame video image sequence; extracts features of multi-thermal variable data; extracts fusion features of the flame video image sequence and multi-thermal variable data; constructs a composite loss function to train each feature extraction module; cascades the dynamic features of the flame video F<subgt;fast< / subgt>, the dynamic features of multi-thermal variables F<subgt;slow< / subgt>, and the multimodal fusion features F<subgt;fusion< / subgt> to form a unified sintering condition feature F; constructs a conformal transformation function to correct the kernel function of the mcODM classifier to obtain a KM-mcODM classifier; inputs the unified sintering condition feature F and the corresponding class labels into the KM-mcODM classifier for classification and identification. It also includes a system for identifying abnormal conditions of rotary kilns based on multimodal data fusion. The present invention realizes the complementary fusion of multimodal information and, combined with the improvement of the long-tail sample classifier model, can effectively improve the recognition accuracy of abnormal conditions of tail class samples.

Description

Technical Field

[0001] The present invention relates to the identification of abnormal conditions of rotary kilns, and specifically to a method and system for identifying abnormal conditions of rotary kilns based on multi-modal data fusion. Background Art

[0002] The coal-fired high-temperature industrial process is the main consumption field of fossil energy in China, and is widely used in traditional process industries such as power generation, cement, and metal smelting. The stable control of the key equipment in the production process - the rotary kiln, is the key to reducing carbon emissions, improving product quality, and reducing energy consumption. Due to the inherent problems in the production process of coal-fired rotary kilns, such as complex production mechanisms and difficulty in online detection of key process parameters, there are still great challenges in the high-precision detection of abnormal conditions.

[0003] The detection of abnormal conditions of rotary kilns usually relies on thermal data and flame video data collected by multiple sensors. The multi-variable thermal data collected by DCS (Distributed Control System) contains rich process parameter information such as working condition fluctuations and equipment states, which can reflect the long-term change laws of working conditions. The flame video images collected by the video monitoring system have more intuitive advantages in reflecting the combustion conditions. The soft measurement of working conditions driven by these two types of unstructured data has become the mainstream method for identifying the production working conditions of rotary kilns. However, the existing research on the high-precision detection of rotary kiln production working conditions still faces the following problems: (1) Due to the multi-major lags and strong spatio-temporal coupling characteristics between multi-variable thermal data, it is difficult to mine working condition information; (2) Due to environmental factors such as high temperature and dust in the coal-fired rotary kiln, the flame video images are usually blurred, and the flame itself has the characteristic of unclear texture, resulting in insufficient discrimination of the working condition characteristics extracted by conventional vision-based working condition identification methods; (3) The above characteristics of the two types of process data usually lead to insufficient information volume and weak robustness of the single-modal working condition characteristics extracted based on a single data source; (4) Due to the difference in data volume between abnormal conditions and normal conditions, the working condition samples show a long-tail distribution, and conventional classification and recognition models usually have difficulty in achieving high-precision recognition of small-sample abnormal conditions.

[0004] CN 111680715 A discloses a method for identifying the sintering state of a rotary kiln considering class imbalance. This method is for identifying abnormal conditions of a rotary kiln based on single-modal working condition characteristics (thermal signals). Due to the insufficient information volume and weak robustness of the single-modal working condition characteristics extracted from a single data source, the recognition rate of the rotary kiln sintering state is low; and the kernel correction method it uses is based on calculating the conformal transformation function based on the interval mean, which requires pre-training a standard classifier to obtain the sample interval, and the calculation efficiency is low, making it difficult to meet the real-time requirements of industrial applications. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the deficiencies of the above-mentioned background technology, and provide a method and system for identifying abnormal conditions of rotary kilns based on multi-modal data fusion, realizing the complementary fusion of multi-modal information, and being able to effectively improve the recognition accuracy of abnormal conditions of tail class samples.

[0006] The technical solution adopted by the present invention to solve its technical problem is a method for identifying abnormal conditions of rotary kilns based on multi-modal data fusion, including the following steps:

[0007] Step S1: Collect the working condition sample set of the rotary kiln;

[0008] Step S2: Use the flame video image sequence feature extraction module to extract the features of the flame video image sequence, and obtain the flame video dynamic feature F fast ;

[0009] Step S3: Use the multi-thermodynamic variable data feature extraction module to extract the features of the multi-thermodynamic variable data, and obtain the multi-thermodynamic variable dynamic feature F slow ;

[0010] Step S4: Use the fusion feature extraction module to extract the fusion features of the flame video image sequence and the multi-thermodynamic variable data, and obtain the multi-modal fusion feature F fusion ;

[0011] [[ID=

[24] ]Step S5: Construct a composite loss function to train the flame video image sequence feature extraction module, the multi-thermodynamic variable data feature extraction module, and the fusion feature extraction module;

[0012] Step S6: Cascade the flame video dynamic feature F fast , the multi-thermodynamic variable dynamic feature F slow and the multi-modal fusion feature F fusion , form a unified sintering working condition feature F, and label the unified sintering working condition feature F for class label division;

[0013] Step S7: Construct a conformal transformation function to correct the kernel function of the mcODM classifier, and obtain a KM-mcODM classifier suitable for the classification and recognition of long-tail samples;

[0014] Step S8: Input the unified sintering working condition feature F and the corresponding class label into the KM-mcODM classifier for classification and recognition.

[0015] Furthermore, in step S2, the specific method for extracting the features of the flame video image sequence is:

[0016] Step S2-1: Extract the flame video image sequence X mThree image feature sequences of brightness Fv1, flame area size Fv2, and flame abundance Fv3 form a multi-feature sequence of the flame video Fv = [Fv1; Fv2; Fv3];

[0017] Step S2-2: Input the multi-feature sequence Fv of the flame video into the first Bi-LSTM network to obtain the dynamic feature F of the flame video fast .

[0018] Furthermore, in step S3, the specific method for extracting the multi-thermodynamic variable data features is as follows:

[0019] Step S3-1: Perform Lipschitz calculation on each thermodynamic variable to obtain the relevant time period X of each thermodynamic variable DRP ;

[0020] Step S3-2: Use the linear interpolation method to unify the lengths of the relevant time periods of each thermodynamic variable to form multi-thermodynamic variable data with consistent dimensions;

[0021] Step S3-3: Input the multi-thermodynamic variable data with consistent dimensions into the second Bi-LSTM network to obtain the dynamic feature F of the multi-thermodynamic variables slow .

[0022] Furthermore, in step S4, the specific method for extracting the fusion features of the flame video image sequence and the multi-thermodynamic variable data is as follows:

[0023] Downsample the multi-feature sequence Fv of the flame video, and concatenate the downsampled Fv' and the relevant time period X of each thermodynamic variable DRP in parallel and input them into a convolutional neural network to obtain the multi-modal fusion feature F fusion .

[0024] Furthermore, in step S5, the constructed composite loss function includes a single-modal classification loss function L c and a cross-modal triplet loss function L r in two parts,

[0025] The single-modal classification loss function L c is defined as:

[0026]

[0027] In the formula, represents the possibility that the i-th sample of the m-th modality belongs to the i-th class label, is the i-th sample of the m-th modality, i is the class label type, and the value of i is [normal burning, underburning, overburning], m ∈ [fast, slow, fusion];

[0028] The cross-modal triplet loss function L rDefined as:

[0029]

[0030] Where, is the i-type sample of m modes, is a non-m-modal i-type sample, is a non-i-type sample of non-m-modality, m∈[fast, slow, fusion];

[0031] Based on the unimodal classification loss function L c And the cross-modal triplet loss function L r Construct a composite loss function Loss. The definition of the composite loss function Loss is:

[0032] Loss=λL c +(1-λ)L r (8)

[0033] Where λ is a weight parameter, and its value range is [0, 1].

[0034] Furthermore, in step S7, a conformal transformation function is constructed to modify the kernel function of the mcODM classifier to obtain a KM-mcODM classifier suitable for classification and recognition of long-tail samples. The specific method is as follows:

[0035] Step S7-1: Select the RBF kernel as the kernel function of the mcODM classifier and calculate the mean distance of heterogeneous samples using the kernel technique;

[0036] Step S7-2: constructing a conformal transformation function based on the heterogeneous distance mean;

[0037] Step S7-3: Use the conformal transformation function to modify the kernel matrix K of the RBF kernel function to obtain the modified kernel matrix The corrected kernel matrix As the kernel matrix of the mcODM classifier, the KM-mcODM classifier is obtained.

[0038] Furthermore, in step S7-1, the method for calculating the mean distance of heterogeneous samples using the kernel technique is as follows:

[0039] When the sample data volume is not large, use formula (10):

[0040]

[0041] Among them, x∈c means that sample x is a c-type sample, x i is the heterogeneous sample of x, n represents the total number of training samples, n crepresents the number of samples of the c-th class; J(x) represents the average distance between sample x and other different-class samples; represents of sample x, represents sample x i 's function, The function represents the kernel space projection function determined by a pre-selected kernel function, indicating the representation form projected into the high-dimensional feature space;

[0042] The distance between two different-class samples in the kernel space is converted to be calculated by the kernel function:

[0043]

[0044] In the formula, K() is a pre-selected kernel function, that is, the RBF kernel function. K(x, x) represents that the input of the kernel function K() is x, x, and K(x, x i ) represents that the input of the kernel function K() is x, x i , K(x i , x l ) represents that the input of the kernel function K() is x i , x i , x l represents other different-class samples of x except x i ;

[0045] When the sample data volume is large, formula (12) is adopted:

[0046]

[0047] Among them, A c represents the set of different-class sample clustering centers of the c-th class, and a i represents the i-th clustering center in A c .

[0048] Furthermore, in step S7-2, the specific method for constructing the conformal transformation function based on the different-class distance mean is as follows:

[0049] When the sample data volume is not large, the conformal transformation function is constructed by formula (13):

[0050]

[0051] Among them, N is a parameter reflecting the imbalance ratio IR of the training samples, and N is calculated by the following formula:

[0052]

[0053] Among them, max(n c) represents the number of samples in the largest class, and ρ is a parameter that controls the spatial expansion coefficient of different class regions in the kernel space. According to experience, it is optimized in [0, 0.1,..., 2];

[0054] When the sample data volume is large, the conformal transformation function is constructed using Equation (15):

[0055] The average distance from the sample to the outlier samples is replaced by the average distance from the sample to the outlier sample cluster centers. Therefore, the conformal transformation function in Equation (13) is replaced by:

[0056]

[0057] In the formula, A c represents the set of outlier sample cluster centers of class c, and a i represents the i-th cluster center in A c ;

[0058] If the vector u(x) = (D1(x), D2(x),..., D d (x)) T is defined, where D i (x) represents the partial derivative of the conformal transformation function with respect to the i-th dimension feature of the sample When using the RBF kernel function, the partial derivative of the conformal transformation function defined in Equation (13) with respect to the j-th dimension of the sample is calculated by the following formula:

[0059]

[0060] In the formula, x(j) represents the j-th dimensional element of the sample x, and x i (j) represents the j-th dimensional element of the sample x i ;

[0061] Furthermore, in step S7-3, the corrected kernel matrix is expressed as:

[0062]

[0063] In the formula, represents the element in the i-th row and j-th column of the corrected kernel matrix , and k ij represents the element in the i-th row and j-th column of the kernel matrix K.

[0064] A system for an abnormal condition recognition method of a rotary kiln based on multi-modal data fusion, including a flame video image sequence feature extraction module, a multi-thermal variable data feature extraction module, a fusion feature extraction module, a composite loss function construction module, a feature concatenation module, and a kernel correction module. The flame video image sequence feature extraction module and the multi-thermal variable data feature extraction module are respectively connected to the fusion feature extraction module. The feature concatenation module is respectively connected to the flame video image sequence feature extraction module, the multi-thermal variable data feature extraction module, and the fusion feature extraction module. The composite loss function construction module is respectively connected to the flame video image sequence feature extraction module, the multi-thermal variable data feature extraction module, and the fusion feature extraction module. The kernel correction module is connected to the feature concatenation module;

[0065] The flame video image sequence feature extraction module is used to extract the features of the flame video image sequence and obtain the flame video dynamic feature F fast ;

[0066] The multi-thermal variable data feature extraction module is used to extract the features of the multi-thermal variable data and obtain the multi-thermal variable dynamic feature F slow ;

[0067] The fusion feature extraction module is used to extract the fusion features of the flame video image sequence and the multi-thermal variable data and obtain the multi-modal fusion feature F fusion ;

[0068] The composite loss function construction module is used to construct a composite loss function;

[0069] The feature concatenation module is used to concatenate the flame video dynamic feature F fast 、the multi-thermal variable dynamic feature F slow and the multi-modal fusion feature F fusion to obtain the unified sintering condition feature F;

[0070] The kernel correction module is used to construct a conformal transformation function to correct the kernel function of the mcODM classifier and obtain the KM-mcODM classifier;

[0071] The KM-mcODM classifier is used to classify and identify the unified sintering condition feature F.

[0072] Compared with the prior art, the advantages of the present invention are as follows:

[0073] The abnormal condition recognition method of a rotary kiln based on multi-modal data fusion extracts the characteristic features of each modality (flame video image features and multi-thermodynamic variable data features), fuses the multi-modal condition features, and realizes the complementary fusion of multi-modal information; combined with the proposed kernel modified classifier (KM-mcODM classifier), during the kernel modification process, a conformal transformation function is constructed using the mean of the inter-class sample intervals, and the mean of the inter-class distances can be directly calculated through the kernel trick without pre-training a standard classifier, effectively improving the calculation efficiency and better meeting the real-time requirements of industrial applications; the kernel modified classifier can effectively improve the recognition accuracy of abnormal conditions of tail-class samples, and the recognition rate of abnormal conditions such as over-burning and under-burning of the rotary kiln can reach more than 90%. Description of the Drawings

[0074] Figure 1 is a schematic diagram of multi-modal data feature extraction for the abnormal condition recognition method of a rotary kiln based on multi-modal data fusion according to an embodiment of the present invention.

[0075] Figure 2 is a schematic diagram of the structure of the abnormal condition recognition system of a rotary kiln based on multi-modal data fusion according to an embodiment of the present invention.

[0076] In the figure, M1-1 - flame video image sequence feature extraction module, M1-2 - multi-thermodynamic variable data feature extraction module, M1-3 - fusion feature extraction module, M2 - composite loss function construction module, M3 - feature concatenation module, M4 - kernel modification module. Detailed Embodiment

[0077] The present invention will be further described in detail below with reference to the drawings and specific embodiments.

[0078] Refer to Figure 1 , the abnormal condition recognition method of a rotary kiln based on multi-modal data fusion in this embodiment includes the following steps:

[0079] Step S1: Collect the rotary kiln condition sample set:

[0080] In this embodiment, from April to June 2021, a rotary kiln condition sample set was collected from a production site of an alumina rotary kiln in Inner Mongolia. There are 1509, 1414, and 8755 under-burning, over-burning, and normal-burning condition samples in the condition sample set, which belong to typical long-tail distribution samples. 70% of the samples in each type of condition are used as the training set, 20% of the samples are used as the test set, and the rest are used as the validation set. The rotary kiln condition sample set includes the collected multi-thermodynamic variable data and flame video images. The sampling frequency of the multi-thermodynamic variable data is 4s, and the flame video images are in RGB format with 25 frames per second and a size of 704*576.

[0081] Step S2: Use the flame video image sequence feature extraction module M1-1 to extract the flame video image sequence features, and obtain the flame video dynamic feature F fast , and the specific method is as follows:

[0082] Step S2-1: Extract three image feature sequences of the brightness Fv1, the flame area size Fv2, and the flame abundance Fv3 of the flame video image sequence X m , and form the flame video multi-feature sequence FV = [Fv1; Fv2; Fv3];

[0083] Step S2-2: Input the flame video multi-feature sequence Fv into the first Bi-LSTM network to obtain the flame video dynamic feature F fast ,

[0084] For the convenience of intuitive understanding, the calculation processes of steps S2-1 and S2-2 can be simplified as:

[0085] F fast = Bi-LSTM(Parallel(Feat(X m ))) (1)

[0086] Step S3: Use the multi-thermodynamic variable data feature extraction module M1-2 to extract the multi-thermodynamic variable data features, and obtain the multi-thermodynamic variable dynamic feature F slow ; The multi-thermodynamic variable data includes the coal feeding speed CFV, the raw material feeding speed RMF, the primary air PA, the negative pressure at the kiln tail NP, the kiln head gas temperature KHT, the kiln tail gas temperature KTT, the main motor current MDC, and the temperature ST of the sintering zone flame. The specific method for extracting the multi-thermodynamic variable data features is as follows:

[0087] Step S3-1: Considering the characteristics of large lag, nonlinearity, and strong coupling among the thermodynamic variables, perform Lipschitz calculation on each thermodynamic variable to obtain the relevant time period X DRP of each thermodynamic variable; The specific method is as follows:

[0088]

[0089]

[0090] In the formula, refers to the Lipschitz quotient obtained by the thermodynamic variable X D with s elements added, L(k) is the k-th largest quotient among all , s is the number of elements of the thermodynamic variable X D , X D (i) represents the i-th element in the input thermodynamic variable X D , X D(j) represents the j-th element of the thermotechnical variable X as input D in X D (i - s) represents the (i - s)-th element of the thermotechnical variable X as input D in X D (j - s) represents the (j - s)-th element of the thermotechnical variable X as input D in X D belongs to one of the eight thermotechnical variables {CFV, RMF, PA, NP, KHT, KTT, MDC, ST}, X RT (i), X RT (j) represent the i-th element and the j-th element in the relevant time period of the thermotechnical variable related to the output, L s represents the Lipschitz number obtained after adding s elements to the thermotechnical variable X D ;

[0091] As the elements in the thermotechnical variable X D increase, that is, increase s, calculate the Lipschitz number L for different s values s , and as the s value increases, draw the curve relationship diagram between s and the Lipschitz number L s Between them, by finding the descending turning point and ascending turning point on the curve relationship diagram, find the last relevant input (LRI), the time period between the first relevant input (FRI) and the last relevant input (LRI) is the relevant time period X of the thermotechnical variable XD DRP ;

[0092] Step S3 - 2: Since the time delay characteristics of each thermotechnical variable are different, resulting in inconsistent lengths of the corresponding relevant time periods X DRP , use the linear interpolation method to unify the lengths of the relevant time periods of each thermotechnical variable to form multi-thermotechnical variable data with consistent dimensions;

[0093] Step S3 - 3: Input the multi-thermotechnical variable data with consistent dimensions into the second Bi-LSTM network to obtain the multi-thermotechnical variable dynamic feature F slow ;

[0094] For the convenience of intuitive understanding, the calculation processes of steps S3 - 1, S3 - 2, and S3 - 3 can be simplified as follows:

[0095] F slow = Bi-LSTM(Parallel(Interp(Lips(X D )))) (4)

[0096] Step S4: Use the fusion feature extraction module M1 - 3 to extract the fusion feature of the flame video image sequence and the multi-thermotechnical variable data to obtain the multi-modal fusion feature F fusion ;

[0097] The specific method is as follows: downsample the multi-feature sequence Fv of the flame video, and concatenate the obtained Fv' after downsampling with the relevant time periods X of each thermal variable DRP in parallel, and input it into a convolutional neural network (Conv). The convolutional neural network is a two-dimensional convolutional neural network. The convolutional neural network uses convolutional layers and pooling layers to calculate and mine the complementary information between the multi-feature sequence of the flame video and the data of multiple thermal variables, and at the same time further deeply mines the coupling characteristics within the same modality to obtain the multi-modal fusion feature F fusion ;

[0098] For the convenience of intuitive understanding, the specific calculation process of step S4 can be simplified and expressed as follows:

[0099] F fusion = Conv(Parallel(Subsamp(Fv); X DRP ))) (5)

[0100] Step S5: By synchronously considering the classification and recognition ability of the multi-modal fusion feature and the low information redundancy between the multi-modal fusion features, use the composite loss function construction module M2 to construct a composite loss function, and train the flame video image sequence feature extraction module M1-1, the multi-thermal variable data feature extraction module M1-2, and the fusion feature extraction module M1-3. Set the hyperparameters for training the feature extraction modules M1-1, M1-2, and M1-3, with the batch size batchsize = 64, the learning rate lr = 0.001, and the number of iterations epoch = 1000.

[0101] The constructed composite loss function includes a single-modal classification loss function L c and a cross-modal triplet loss function L r in two parts. Among them, the single-modal classification loss function L optimizes the separability of the features of different classes in the single modality. The single-modal classification loss function L c is defined as:

[0102]

[0103] In the formula, represents the possibility that the i-th sample in the m-th modality belongs to the i-th class label, is the i-th sample in the m-th modality, i is the class label type, and the value of i is [normal burning, underburning, overburning], m ∈ [fast, slow, fusion].

[0104] And the cross-modal triplet loss function L r guides the model to improve the consistency of cross-modal same-class working condition features and the difference of different-class working condition features. The cross-modal triplet loss function L rDefined as:

[0105]

[0106] In the formula, is the i-th sample of the m modality, is the i-th sample of non-m modality, is the non-i-th sample of non-m modality, m ∈ [fast, slow, fusion].

[0107] Based on the single-modal classification loss function L c and the cross-modal triplet loss function L r Construct the composite loss function Loss. The definition formula of the composite loss function Loss is:

[0108] Loss = λL c +(1 - λ)L r (8)

[0109] In the formula, λ is the weight parameter, and its value range is [0, 1].

[0110] In the composite loss function Loss, use the Adam optimization algorithm to calculate the gradient of the network error for each weight parameter during backpropagation, and obtain the new weights through the parameter update process. Iteratively calculate the model weights until a predetermined small loss is reached and the best prediction value is obtained.

[0111] The reason for choosing Adam as the optimization algorithm is that it can design independent adaptive learning rates for different parameters and has high computational efficiency.

[0112] Step S6: Use the feature concatenation module M3 to concatenate the dynamic features F of the flame video fast , the dynamic features F of multiple thermal variables slow and the multi-modal fusion features F fusion , form the unified sintering condition feature F, and label the unified sintering condition feature F for class label division. The class labels include underfiring, overfiring, and normal.

[0113] For the convenience of intuitive understanding, the calculation process of step S6 can be simply expressed as follows:

[0114] F = Concat(F fast , F slow , F fusion ) (9)

[0115] Step S7: Use the kernel correction module M4 to construct a conformal transformation function to correct the kernel function of the mcODM classifier, and obtain the KM-mcODM classifier. The KM-mcODM classifier is suitable for the classification and recognition of long-tail samples. The specific method is:

[0116] Step S7-1: Select the kernel function of the mcODM classifier as the RBF kernel, and calculate the mean distance of heterogeneous samples through the kernel trick;

[0117] The calculation of the mean distance of heterogeneous samples does not require pre-training of a standard classifier. Only by selecting the kernel function can it be calculated completely based on the data:

[0118]

[0119] where x ∈ c indicates that the sample x is a sample of class c, x i is the heterogeneous sample of x, n represents the total number of training samples, and n c represents the number of samples of the c-th class; J(x) represents the mean distance between the sample x and other heterogeneous samples; represents the function of the sample x, represents the i of the sample x function, The function represents the kernel space projection function determined by the pre-selected kernel function, which represents the form of projection into the high-dimensional feature space. The specific form of this function is unknown, but the relationship between this function and the kernel function can be used for calculation, that is, the "kernel trick" described in the present invention.

[0120] The distance between two heterogeneous samples in the kernel space can also be converted into a kernel function calculation by the following kernel trick:

[0121]

[0122] In the formula, K() is the pre-selected kernel function, that is, the RBF kernel function. K(x, x) represents that the input of the kernel function K() is x, x, and K(x, x i ) represents that the input of the kernel function K() is x, x i , K(x i , x l ) represents that the input of the kernel function K() is x i , x l , x l represents the other heterogeneous samples of x except x i .

[0123] The above-mentioned heterogeneous distance mean has a high computational complexity. Considering that the clustering center of the samples reflects the overall distribution of the sample set, clustering in the kernel space can also characterize the rough spatial distribution of the samples. By calculating the mean distance from the current sample to the clustering center of the heterogeneous samples, the relative position distribution between the current sample and the heterogeneous samples can also be better reflected. In view of this, under the condition of large-scale data, consider the following calculation of the heterogeneous distance mean:

[0124]

[0125] Among them, A c represents the set of outlier sample clustering centers of class c, and a i represents the i-th clustering center in A c . Both A c and a i can be obtained by the Kernel-based Fuzzy C-Means (KFCM) method. The distance between a sample and an outlier sample clustering center in the kernel space can also be calculated using the kernel trick shown in Equation (11).

[0126] The present invention provides two methods for calculating the mean outlier distance according to the size of the data volume. Among them, Equation (11) is applicable to the case where the sample data volume is not large, while Equation (12) is applicable to the case where the sample data volume is large. Since the total number of working condition samples extracted from the actual data of the rotary kiln in this embodiment is 11,678, which does not belong to large-scale data, Equation (10) can be used for calculation. According to experience, the sample data volume is not large in this embodiment means that the number of samples is less than 50,000, and the sample data volume is large means that the number of samples is not less than 50,000. In practical applications, the definition standard of the sample data volume can also be other.

[0127] Step S7-2: Construct a conformal transformation function based on the mean outlier distance;

[0128] Construct a conformal transformation function with the goal of expanding the spatial resolution of the class edge region and reducing the spatial resolution of the minority class region as follows:

[0129]

[0130] Among them, N is a parameter reflecting the imbalance ratio IR of the training samples, which can be calculated by the following formula:

[0131]

[0132] max(n c ) represents the number of samples in the class with the largest scale, and ρ is a parameter controlling the spatial expansion coefficient of different class regions in the kernel space. According to experience, it can be optimized in [0, 0.1,..., 2].

[0133] When the sample data volume is large, in order to improve the calculation efficiency of the conformal transformation function, the mean distance from the sample to the outlier samples can be replaced by the mean distance from the sample to the outlier sample clustering centers. Therefore, in the case of large-scale data, the above conformal transformation function can be replaced by:

[0134]

[0135] Wherein, A c represents the set of heterogeneous sample clustering centers of class c, and a i represents the c ith clustering center in A.

[0136] If the vector u(x) = (D1(x), D2(x),..., D d (x)) T is defined, where D i (x) represents the partial derivative of the conformal transformation function with respect to the ith-dimensional feature of the sample When using the RBF kernel function, the partial derivative of the conformal transformation function defined in Equation (13) with respect to the jth dimension of the sample can be calculated by the following formula:

[0137]

[0138] Wherein, x(j) represents the jth-dimensional element of the sample x, and x i (j) represents the jth-dimensional element of the sample x i .

[0139] Step S7-3: Use the conformal transformation function to correct the kernel matrix K of the RBF kernel function to obtain the corrected kernel matrix Use the corrected kernel matrix as the kernel matrix of the mcODM classifier, thereby obtaining the KM-mcODM classifier;

[0140] The corrected kernel matrix is as follows:

[0141]

[0142] Wherein, represents the element at the ith row and jth column of the corrected kernel matrix , and k ij represents the element at the ith row and jth column of the kernel matrix K.

[0143] Step S8: Input the unified sintering condition feature F and the corresponding class label into the KM-mcODM classifier for classification and recognition.

[0144] Refer to Figure 2, in this embodiment, the rotary kiln abnormal condition recognition system based on multi-modal data fusion includes a flame video image sequence feature extraction module M1-1, a multi-thermal variable data feature extraction module M1-2, a fusion feature extraction module M1-3, a composite loss function construction module M2, a feature concatenation module M3, and a kernel correction module M4. The flame video image sequence feature extraction module M1-1 and the multi-thermal variable data feature extraction module M1-2 are respectively connected to the fusion feature extraction module M1-3. The feature concatenation module M3 is respectively connected to the flame video image sequence feature extraction module M1-1, the multi-thermal variable data feature extraction module M1-2, and the fusion feature extraction module M1-3. The composite loss function construction module M2 is respectively connected to the flame video image sequence feature extraction module M1-1, the multi-thermal variable data feature extraction module M1-2, and the fusion feature extraction module M1-3. The kernel correction module M4 is connected to the feature concatenation module M3.

[0145] The flame video image sequence feature extraction module M1-1 is used to extract the features of the flame video image sequence and obtain the flame video dynamic feature F. fast .

[0146] The multi-thermal variable data feature extraction module M1-2 is used to extract the features of the multi-thermal variable data and obtain the multi-thermal variable dynamic feature F. slow .

[0147] The fusion feature extraction module M1-3 is used to extract the fusion features of the flame video image sequence and the multi-thermal variable data and obtain the multi-modal fusion feature F. fusion .

[0148] The composite loss function construction module M2 is used to construct a composite loss function.

[0149] The feature concatenation module M3 is used to concatenate the flame video dynamic feature F fast , the multi-thermal variable dynamic feature F slow and the multi-modal fusion feature F fusion to obtain the unified sintering condition feature F.

[0150] The kernel correction module M4 is used to construct a conformal transformation function to correct the kernel function of the mcODM classifier and obtain the KM-mcODM classifier.

[0151] The KM-mcODM classifier is used to classify and identify the unified sintering condition feature F.

[0152] The abnormal condition recognition method of the rotary kiln based on multi-modal data fusion extracts the characteristic of each modal condition (flame video image characteristic and multi-thermodynamic variable data characteristic) respectively, fuses the multi-modal condition characteristics, and realizes the complementary fusion of multi-modal information. Combined with the proposed kernel modified classifier (KM-mcODM classifier), it can effectively improve the recognition accuracy of abnormal conditions of tail samples, and the recognition rate of abnormal conditions such as overburning and underburning of the rotary kiln can reach more than 90%. There is a significant performance improvement compared with the methods using single-modal condition characteristics and conventional classifiers. Table 1 shows the experimental results of the abnormal condition recognition of the coal-fired rotary kiln by the method of the present invention.

[0153] Table 1 Recognition results of different methods for working conditions

[0154]

[0155] In Table 1, F fast +ODM represents the scheme of using the dynamic characteristics of the flame video F fast plus the ODM classifier, F slow +ODM classifier represents the scheme of using the dynamic characteristics of multi-thermodynamic variables F slow plus the ODM classifier, F+ODM represents the scheme of using the unified sintering condition characteristics F plus the ODM classifier, F+KM-mcODM represents the scheme of using the unified sintering condition characteristics F plus the KM-mcODM classifier, that is, the method of the present invention. It can be seen from Table 1 that the recognition rate of abnormal conditions such as overburning and underburning of the rotary kiln of the present invention can reach more than 90%, which is significantly higher than other methods.

[0156] Those skilled in the art can make various modifications and variations to the present invention. If these modifications and variations are within the scope of the claims of the present invention and their equivalent technologies, then these modifications and variations are also within the protection scope of the present invention.

[0157] The content not described in detail in the specification is the well-known prior art to those skilled in the art.

Claims

1. A method for identifying abnormal operating conditions of a rotary kiln based on multi-modal data fusion, characterized in that, It includes the following steps: Step S1: Collect the working condition sample set of the rotary kiln; Step S2: Extract the features of the flame video image sequence using the flame video image sequence feature extraction module to obtain the dynamic features F of the flame video fast ; Step S3: Extract multi-thermodynamic variable data features using the multi-thermodynamic variable data feature extraction module to obtain the multi-thermodynamic variable dynamic feature F slow ; Step S4: Use the fusion feature extraction module to extract the fusion features of the flame video image sequence and multi-thermodynamic variable data, and obtain the multi-modal fusion feature F fusion ; Step S5: Construct a composite loss function to train the flame video image sequence feature extraction module, the multi-thermodynamic variable data feature extraction module, and the fusion feature extraction module; Step S6: Cascade flame video dynamic feature F fast , multi-thermodynamic variable dynamic feature F slow and multi-modal fusion feature F fusion , to form a unified sintering condition feature F, label the unified sintering condition feature F, and perform class label division; Step S7: Construct a conformal transformation function to correct the kernel function of the mcODM classifier, and obtain the KM-mcODM classifier adapted to the classification and recognition of long-tail samples; Step S8: Input the unified sintering condition feature F and the corresponding class label into the KM-mcODM classifier for classification and recognition.

2. The abnormal condition recognition method for a rotary kiln based on multi-modal data fusion according to claim 1, wherein: In step S2, the specific method for extracting the flame video image sequence features is as follows: Step S2-1: Extract three image feature sequences of the brightness Fv1, the flame area size Fv2, and the flame abundance Fv3 of the flame video image sequence X m to form a flame video multi-feature sequence Fv = [Fv1; Fv2; Fv3]; Step S2-2: Input the multi-feature sequence Fv of the flame video into the first Bi-LSTM network to obtain the dynamic feature F of the flame video fast .

3. The abnormal condition recognition method for rotary kilns based on multi-modal data fusion according to claim 2, wherein: In step S3, the specific method for extracting the multi-thermodynamic variable data features is as follows: Step S3-1: Perform Lipschitz calculations on each thermotechnical variable to obtain the relevant time period X of each thermotechnical variable DRP ; Step S3-2: Use the linear interpolation method to unify the relevant time period lengths of each thermodynamic variable to form multi-thermodynamic variable data with consistent dimensions; Step S3-3: Input the multi-thermodynamic variable data with consistent dimensions into the second Bi-LSTM network to obtain the multi-thermodynamic variable dynamic feature F slow .

4. The abnormal condition recognition method for rotary kilns based on multi-modal data fusion according to claim 3, characterized in that: In step S4, the specific method for extracting the fusion features of the flame video image sequence and the multi-thermodynamic variable data is as follows: Downsample the multi-feature sequence Fv of the flame video, and concatenate the obtained Fv' after downsampling with the relevant time periods X of each thermal variable DRP in parallel, and input it into a convolutional neural network to obtain the multi-modal fusion feature F fusion .

5. The method for identifying abnormal operating conditions of a rotary kiln based on multi-modal data fusion according to any one of claims 1-4, characterized in that: In step S5, the constructed composite loss function includes a unimodal classification loss function L c and a cross-modal triplet loss function L r in two parts. The single-modal classification loss function L c is defined as: wherein, represents the possibility that the i-th sample of the m-th modality belongs to the i-th class label, is the i-th sample of the m-th modality, i is the class label type, and the value of i is [positive burning, under-burning, over-burning], m ∈ [fast, slow, fusion]; The cross-modal triple loss function L r is defined as: In the formula, is the i-th sample of the m modality, is the i-th sample of the non-m modality, is the non-i sample of the non-m modality, where m ∈ [fast, slow, fusion]; Based on the single-modal classification loss function L c and the cross-modal triplet loss function L r Construct a composite loss function Loss, and the definition formula of the composite loss function Loss is: Loss=λL c +(1-λ)L r (8) In the formula, λ is a weight parameter, and its value range is [0, 1].

6. The abnormal condition recognition method for a rotary kiln based on multi-modal data fusion according to any one of claims 1-4, characterized in that: In step S7, construct a conformal transformation function to correct the kernel function of the mcODM classifier, and obtain the KM-mcODM classifier adapted to the classification and recognition of long-tail samples. The specific method is as follows: Step S7-1: Select the RBF kernel as the kernel function of the mcODM classifier, and calculate the mean distance of different-class samples through the kernel trick; Step S7-2: Construct a conformal transformation function based on the mean distance of different-class samples; Step S7-3: Modify the kernel matrix K of the RBF kernel function using a conformal transformation function to obtain a modified kernel matrix Use the modified kernel matrix as the kernel matrix of the mcODM classifier, thereby obtaining the KM-mcODM classifier.

7. The abnormal condition recognition method for rotary kiln based on multi-modal data fusion according to claim 6, wherein: In step S7-1, the method for calculating the mean distance of different-class samples through the kernel trick is as follows: When the sample data volume is small, use formula (10): where \(x\in c\) indicates that the sample \(x\) is a sample of class \(c\), and \(x\) i is the outlier sample of \(x\), \(n\) represents the total number of training samples, and \(n\) c represents the number of samples of the \(c\)-th class; \(J(x)\) represents the mean distance between the sample \(x\) and other outlier samples; represents the function of the sample \(x\), represents the sample \(x\) i 's function, The function represents the kernel space projection function determined by a pre-selected kernel function, indicating the representation form projected into the high-dimensional feature space; The distance between two different-class samples in the kernel space is converted into a kernel function calculation: where K() is a pre-selected kernel function, namely the RBF kernel function, K(x, x) indicates that the input of the kernel function K() is x, x, K(x, x i ) indicates that the input of the kernel function K() is x, x i , K(x i , x l ) indicates that the input of the kernel function K() is x i , x l , x l represents other heterogeneous samples of x except x i ; When the sample data volume is large, use formula (12): Among them, A c represents the set of outlier sample clustering centers of class c, and a i represents the i-th clustering center in A c .

8. The method for identifying abnormal operating conditions of a rotary kiln based on multi-modal data fusion according to claim 7, wherein: In step S7—2, the specific method for constructing a conformal transformation function based on the mean distance of different-class samples is as follows: When the sample data volume is small, use formula (13) to construct a conformal transformation function: Among them, N is a parameter reflecting the imbalance ratio IR of the training samples, and N is calculated by the following formula: where max(n c ) represents the number of samples in the class with the largest scale, and ρ is a parameter that controls the spatial expansion coefficient of different class regions in the kernel space. According to experience, it is optimized in [0, 0.1,..., 2]; When the sample data volume is large, use formula (15) to construct a conformal transformation function: The mean distance between different-class samples is replaced by the mean distance between the clustering centers of different-class samples. Therefore, the conformal transformation function of formula (13) is replaced by: where A c represents the set of heterogeneous sample clustering centers of class c, and a i represents the i-th clustering center in A c ; If the vector u(x) = (D1(x), D2(x),..., D d (x)) T is defined, where D i (x) represents the partial derivative of the conformal transformation function with respect to the i-th dimensional feature of the sample When using the RBF kernel function, the partial derivative of the conformal transformation function defined by Equation (13) with respect to the j-th dimension of the sample is calculated by the following formula: where x(j) represents the j-th dimensional element of the sample x, x i (j) represents the sample x i of the j-th dimensional element.

9. The method for identifying abnormal operating conditions of a rotary kiln based on multi-modal data fusion according to claim 8, characterized in that: In step S7-3, the corrected kernel matrix is expressed as: In the formula, represents the element in the i-th row and j-th column of the corrected kernel matrix , and k ij represents the element in the i-th row and j-th column of the kernel matrix K.

10. A system for an abnormal condition recognition method of a rotary kiln based on multi-modal data fusion according to any one of claims 1-9, characterized in that: It includes a flame video image sequence feature extraction module, a multi-thermodynamic variable data feature extraction module, a fusion feature extraction module, a composite loss function construction module, a feature concatenation module, and a kernel correction module. The flame video image sequence feature extraction module and the multi-thermodynamic variable data feature extraction module are respectively connected to the fusion feature extraction module. The feature concatenation module is respectively connected to the flame video image sequence feature extraction module, the multi-thermodynamic variable data feature extraction module, and the fusion feature extraction module. The composite loss function construction module is respectively connected to the flame video image sequence feature extraction module, the multi-thermodynamic variable data feature extraction module, and the fusion feature extraction module. The kernel correction module is connected to the feature concatenation module; The flame video image sequence feature extraction module is used to extract the features of the flame video image sequence, and obtain the dynamic feature F of the flame video fast ; The multi-thermodynamic variable data feature extraction module is used to extract multi-thermodynamic variable data features and obtain the multi-thermodynamic variable dynamic feature F slow ; The fusion feature extraction module is used to extract the fusion features of the flame video image sequence and multi-thermodynamic variable data, and obtain the multi-modal fusion feature F fusion ; The composite loss function construction module is used to construct a composite loss function; The feature cascading module is used to cascade the dynamic features F of the flame video fast , the dynamic features F of multiple thermal variables slow and the multi-modal fusion features F fusion to obtain the unified sintering condition feature F; The nuclear correction module is used to construct a conformal transformation function to correct the kernel function of the mcODM classifier, resulting in the KM-mcODM classifier; The KM-mcODM classifier is used to classify and identify the unified sintering condition feature F.

Images