Small sample sintering condition recognition method based on multi-scale attention modeling of flame video

Abstract

A small sample sintering condition recognition method based on flame video multi-scale attention modeling includes: collecting flame video condition data of a rotary kiln site, extracting global condition features and local condition features of flame video condition samples with known labels, and performing parameter update learning of the model; extracting global condition features and local condition features of unlabeled flame video condition samples; calculating the similarity of global condition features and local condition features of flame video abnormal condition samples with known labels and unlabeled flame video condition samples, and calibrating corresponding abnormal condition pseudo labels for unlabeled flame video condition samples according to feature similarity, or discarding the unlabeled flame video condition samples; adding flame video condition samples marked as abnormal condition pseudo labels to known label flame video condition samples to form a sample training sample set, and retraining the model using the training sample set. The present invention can improve the recognition accuracy of small sample abnormal conditions and reduce the computational complexity of the model.

Description

Technical Field

[0001] The present invention relates to a sintering condition recognition method, and in particular to a small sample sintering condition recognition method based on flame video multi-scale attention modeling. Background Art

[0002] Coal-fired high-temperature industrial processes are the main force of fossil energy consumption in my country and are widely used in traditional process industries such as power generation, cement, and metal smelting. Achieving stable control of high-temperature industrial production processes will help reduce carbon emissions, improve product quality, and reduce energy consumption. The temperature of the sintering zone in the kiln body is one of the most important parameters in the production process. According to the sintering temperature, the rotary kiln can be divided into three sintering conditions: normal, overburned, and underburned. Normal conditions indicate that the temperature of the sintering zone is within the set range. At this time, the material is fully sintered, the product sintering quality is high, and the carbon and nitrogen emissions are low. It is an ideal sintering condition. Overburned and underburned conditions indicate that the sintering temperature is higher and lower than the required sintering temperature, both of which are abnormal conditions. Overburning causes the material fluidity to deteriorate, and it is easy to form rings and nodules to damage the kiln body; underburning will lead to incomplete combustion of coal powder and insufficient material reaction, resulting in yellowing and high carbon and nitrogen oxide emissions. How to accurately identify the sintering conditions of the rotary kiln is of great significance for improving the sintering quality of materials, reducing carbon and nitrogen emissions, and improving energy utilization efficiency.

[0003] In recent years, with the development of big data, image processing technology and pattern recognition technology, many scholars have conducted extensive research on production conditions and product quality using flame video images. The main idea of ​​the working condition detection method based on coal flame images is to extract the two-dimensional or three-dimensional visual features of the edges, textures and contours of the flame images to identify related working conditions or predict key parameters. Early studies focused on using single-frame images to extract their local spatial visual features and extract global visual features such as image texture and contours for working condition identification. In recent years, some scholars have also used deep learning methods such as stacked autoencoder convolutional neural networks to automatically learn the visual features of flame images in an end-to-end manner to complete related working condition prediction work.

[0004] The above methods have achieved certain results in the sintering working condition recognition task by extracting more effective visual features from flame images and training on flame image datasets with working condition labels. There are still two problems that need to be solved in this type of method: First, this type of method first extracts frames from flame videos to form flame images, and then enhances the quality of flame images and recognizes working conditions. However, due to the characteristics of flame flickering, this type of working condition recognition method based on a single frame is difficult to capture the dynamic changes of flames, and the robustness and recognition accuracy are difficult to meet the needs of the site. Secondly, the corresponding data collected during the normal production process have problems such as large differences in the volume of normal working conditions and abnormal working conditions, that is, sample imbalance. Although patent CN117150345 A proposes a solution to improve the classifier for the problem of long-tail distribution of data, which is helpful for improving the recognition accuracy of minority abnormal working conditions, the impact of the working condition sample imbalance problem also includes the problem of overfitting of the feature learning network for the majority class samples. Simply improving the classifier cannot improve the data bias of the feature learning model, and it is difficult to learn the clear classification boundaries of different working condition features, which affects the working condition recognition accuracy. Summary of the invention

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the above-mentioned background technology and provide a small sample sintering condition recognition method based on flame video multi-scale attention modeling, so as to improve the recognition accuracy of small sample abnormal conditions, improve the robustness and accuracy of the recognition model, and reduce the computational complexity of the model.

[0006] The technical solution adopted by the present invention to solve the technical problem is a small sample sintering condition recognition method based on flame video multi-scale attention modeling, including the following methods:

[0007] Step S1: Collecting the on-site flame video working condition data of the rotary kiln, and preprocessing it, constructing a flame video working condition sample, and marking the working condition category label for the data of known working conditions;

[0008] Step S2: Use the known labeled flame video working condition samples to train the sintering working condition recognition model: Input the known labeled flame video working condition samples into the global working condition feature extraction network, extract the global working condition features of the flame video, and form a flame video feature sequence , the flame video feature sequence Input the local working condition feature extraction network to extract the local working condition features of the flame video and form a flame video feature sequence ; The flame video feature sequence Input the classifier to update the parameters of the sintering condition recognition model;

[0009] Step S3: After the sintering condition recognition model is trained using the flame video condition samples with known labels, the unlabeled flame video condition samples are input into the global condition feature extraction network to extract the global condition features of the flame video to form a flame video feature sequence , the flame video feature sequence Input the local working condition feature extraction network to extract the local working condition features of the flame video and form a flame video feature sequence ;

[0010] Step S4: Calculate the similarity of the global working condition features and local working condition features of the flame video abnormal working condition samples with known labels and the flame video working condition samples without labels, and calibrate the corresponding abnormal working condition pseudo labels for the flame video working condition samples without labels according to the feature similarity, or discard the flame video working condition samples without labels;

[0011] Step S5: adding the flame video condition samples marked with abnormal condition pseudo labels to the flame video condition samples with known labels to form a training sample set with a more balanced sample distribution, and using the training sample set to retrain the sintering condition recognition model.

[0012] Furthermore, in step S1, the method for preprocessing the rotary kiln on-site flame video working condition data is: extracting flame video frames, inputting them into the Transformer network, and the Patch Partition layer of the Transformer network blocks the input flame video frames.

[0013] Further, in step S2, the sintering condition recognition model includes a global condition feature extraction network, a local condition feature extraction network, a pseudo-label learning module and a classifier. The global condition feature extraction network is respectively connected to the local condition feature extraction network and the pseudo-label learning module, and the local condition feature extraction network is respectively connected to the pseudo-label learning module and the classifier; the global condition feature extraction network is used to extract the global condition features of the flame video, the local condition feature extraction network is used to extract the local condition features of the flame video, and the pseudo-label learning module is used to calculate the similarity of the global condition features and the local condition features of the abnormal condition samples of the flame video with known labels and the unlabeled flame video condition samples, and according to the feature similarity, the corresponding abnormal condition pseudo-label is calibrated for the unlabeled flame video condition samples, or the unlabeled flame video condition samples are discarded.

[0014] Further, in step S2, the global working condition feature extraction network includes a flame video linear embedding network, a first video linear attention network, a first spatial feature downsampling module, and a second video linear attention network, the flame video linear embedding network is connected to the first video linear attention network, the first video linear attention network is connected to the first spatial feature downsampling module, and the first spatial feature downsampling module is connected to the second video linear attention network; the first video linear attention network performs a first global working condition feature extraction on the output of the flame video linear embedding network, the extraction result is downsampled by the first spatial feature downsampling module, and then the second global working condition feature extraction is performed by the second video linear attention network;

[0015] The first video linear attention network is the same as the second video linear attention network, including a first normalization layer, a first 3D window focused linear multi-head self-attention module, a second normalization layer, a first multi-layer perceptron, a third normalization layer, a second 3D window focused linear multi-head self-attention module, a fourth normalization layer, and a second multi-layer perceptron. The first normalization layer is connected to the first 3D window focused linear multi-head self-attention module, the first 3D window focused linear multi-head self-attention module is connected to the second normalization layer, the second normalization layer is connected to the first multi-layer perceptron, the first multi-layer perceptron is connected to the third normalization layer, the third normalization layer is connected to the second 3D window focused linear multi-head self-attention module, the second 3D window focused linear multi-head self-attention module is connected to the fourth normalization layer, and the fourth normalization layer is connected to the second multi-layer perceptron. The first 3D window focused linear multi-head self-attention module is the same as the second 3D window focused linear multi-head self-attention module, and is used to calculate the self-attention with focused Linear Attention in a 3D window of a specified size to extract the global working condition characteristics of the flame video.

[0016] Input Features After the first normalization layer and the first 3D window focused linear multi-head self-attention module, the output features ,feature With features Sum and get the features ,feature Output features after the second normalization layer and the first multi-layer perceptron ,feature With features Sum and get the features ,feature After the third normalization layer and the second 3D window focused linear multi-head self-attention module, the output features ,feature With features Sum and get the features ,feature After the fourth normalization layer and the second multi-layer perceptron, the output feature ,feature With features Sum and get the features ; The specific process expression for extracting the global working condition features of flame video is:

[0017] The specific expression is:

[0018] (1)

[0019] (2)

[0020] (3)

[0021] (4)

[0022] For the first video linear attention network, the equations (1) to (4) , for the second video linear attention network, the equations (1) to (4) ,feature ,feature ,feature ,feature .

[0023] Furthermore, the first 3D window focused linear multi-head self-attention module and the second 3D window focused linear multi-head self-attention module obtain a Linear Attention self-attention weight matrix, specifically:

[0024] Through linear mapping, we can get matrix , matrix ， matrix ,Right now:

[0025] (5)

[0026] In formula (5), for the first 3D window focused linear multi-head self-attention module, represents the input features of the first 3D window focused linear multi-head self-attention module, that is, the features obtained after data normalization by the first normalization layer; for the second 3D window focused linear multi-head self-attention module, Represents the input features of the second 3D window focused linear multi-head self-attention module, that is, the features obtained after data normalization by the third normalization layer; express matrix The learnable parameters of express matrix The learnable parameters of express matrix The learnable parameters of , express Features or The dimension of the feature;

[0027] Using Focus Function as the kernel function right matrix Perform nonlinear feature mapping to obtain the focused feature matrix ,right matrix Perform nonlinear feature mapping to obtain the focused feature matrix ; Mathematical expression of Focus Function for:

[0028] (6)

[0029] (7)

[0030] The input in (6) and (7) refer to matrix or matrix , Represents the Rectified Linear Unit activation function; Express Level Power operation;

[0031] Get the focused feature matrix and After that, calculate the Linear Attention self-attention weight matrix , among which Linear Attention self-attention weights The calculation formula is:

[0032] (8)

[0033] In formula (8), express matrix Middle feature vectors, express matrix Middle feature vectors, express matrix Middle feature vectors, , Represents the focused feature matrix The focus feature in , Represents the focused feature matrix The focus feature; Indicates the number of block feature vectors;

[0034] Using 3D depth-wise separable convolutional modules with time dimension matrix Perform a three-dimensional convolution operation and then superimpose it on the Linear Attention self-attention weight matrix , get the final Linear Attention self-attention weight matrix The mathematical expression is:

[0035] (9)

[0036] Said and They represent the final Linear Attention self-attention weight matrices of the first 3D window focused linear multi-head self-attention module and the second 3D window focused linear multi-head self-attention module respectively.

[0037] Further, in step S2, the local working condition feature extraction network includes a second spatial feature downsampling module, a first video Swin Transformer network, a third spatial feature downsampling module, and a second video Swin Transformer network, the second spatial feature downsampling module is connected to the first video Swin Transformer network, the first video Swin Transformer network is connected to the third spatial feature downsampling module, and the third spatial feature downsampling module is connected to the second video Swin Transformer network; the second spatial feature downsampling module downsamples the output of the second video linear attention network, and the downsampling result is subjected to the first local working condition feature extraction through the first video Swin Transformer network, the extraction result is downsampled through the third spatial feature downsampling module, and the downsampling result is subjected to the second local working condition feature extraction through the second video Swin Transformer network;

[0038] The Swin Transformer network of the first video is the same as the Swin Transformer network of the second video, including a fifth normalization layer, a 3D window multi-head self-attention module, a sixth normalization layer, a third multi-layer perceptron, a seventh normalization layer, a 3D moving window multi-head self-attention module, an eighth normalization layer, and a fourth multi-layer perceptron. The fifth normalization layer is connected to the 3D window multi-head self-attention module, the 3D window multi-head self-attention module is connected to the sixth normalization layer, the sixth normalization layer is connected to the third multi-layer perceptron, the third multi-layer perceptron is connected to the seventh normalization layer, the seventh normalization layer is connected to the 3D moving window multi-head self-attention module, the 3D moving window multi-head self-attention module is connected to the eighth normalization layer, and the eighth normalization layer is connected to the fourth multi-layer perceptron; the 3D window multi-head self-attention module is used to calculate the SoftMax self-attention in a 3D window of a specified size, thereby extracting local working condition features; the 3D moving window multi-head self-attention module is used to calculate the SoftMax self-attention by moving the 3D window in a 3D window of a specified size, thereby extracting local working condition features; features The data is normalized by the fifth normalization layer and then passes through the 3D window multi-head self-attention module to output features. ,feature With features Sum and get the features ,feature Output features after the sixth normalization layer and the third multi-layer perceptron ,feature With features Sum and get the features ,feature After the seventh normalization layer and the 3D moving window multi-head self-attention module, the output features ,feature With features Sum and get the features ,feature After the eighth normalization layer and the fourth multi-layer perceptron, the output feature ,feature With features Sum and get the final output features of the Swin Transformer network of the first video ; The specific process expression for extracting local working condition features is:

[0039] (10)

[0040] (11)

[0041] (12)

[0042] (13)

[0043] For the first video Swin Transformer network, the equations (10) to (13) are , for the second video Swin Transformer network, the equations (10) to (13) are ,feature ,feature ,feature ,feature .

[0044] Furthermore, the specific method for the 3D window multi-head self-attention module to obtain the SoftMax self-attention matrix is:

[0045] The characteristics After inputting into the fifth normalization layer, the normalized features are obtained. , and then this feature Input into the 3D window multi-head self-attention module;

[0046] feature After entering the 3D window multi-head self-attention module, linear mapping is performed to obtain matrix , matrix and matrix ,Right now:

[0047] (14)

[0048] In formula (14), express matrix The learnable parameters of express matrix The learnable parameters of express matrix The learnable parameters of

[0049] get matrix , matrix and matrix After that, in the size of The SoftMax self-attention weight matrix is ​​calculated in the 3D window; the calculation expression of the SoftMax self-attention weight matrix is:

[0050] (15)

[0051] (16)

[0052] In formulas (15) and (16), represents the SoftMax self-attention weight matrix, express matrix , matrix The characteristic dimension of Indicates relative position deviation;

[0053] The specific method for the 3D moving window multi-head self-attention module to obtain the SoftMax self-attention weight matrix is:

[0054] get matrix , matrix and matrix Then, move the 3D window forward. P / 2 distance, move right M / 2 round down the distance, move downward M / 2 The distance rounded down, that is, the overall movement of the 3D window The distance rounded down; in the moved 3D window, the SoftMax self-attention weight matrix is ​​calculated.

[0055] Further, step S4 specifically includes the following steps:

[0056] Step S4-1: Calculate the global working condition feature similarity between the overburning flame video working condition sample with known label and the flame video working condition sample without label , set the global working condition feature similarity threshold between the overburned flame video working condition sample and the unlabeled flame video working condition sample , determine the similarity of global working condition characteristics Is it greater than the threshold? If it is greater than, go to step S5-2 to further calculate the similarity of the local working condition features of the flame video overburning working condition samples with known labels and the flame video working condition samples without labels. If not, proceed to step S4-3 to further calculate the global working condition feature similarity between the flame video underburned working condition sample with known labels and the flame video working condition sample without labels. ;

[0057] Step S4-2: Calculate the similarity of local working condition features between the flame video overburning working condition samples with known labels and the flame video working condition samples without labels , set the similarity threshold of the local working condition features between the flame video overburning working condition samples and the unlabeled flame video working condition samples , judge the similarity of local working condition characteristics Is it greater than the threshold? If it is greater than, the unlabeled flame video working condition sample is calibrated with an overburning working condition pseudo label; if it is not greater than, go to step S4-3 to further calculate the global working condition feature similarity between the flame video underburning working condition sample with a known label and the unlabeled flame video working condition sample ;

[0058] Step S4-3: Calculate the global working condition feature similarity between the flame video underburning working condition samples with known labels and the flame video working condition samples without labels , set the global working condition feature similarity threshold between the flame video underburned working condition sample and the unlabeled flame video working condition sample , determine the similarity of global working condition characteristics Is it greater than the threshold? If it is greater than, go to step S4-4 to further calculate the similarity of the local working condition features of the flame video underburning working condition samples with known labels and the flame video working condition samples without labels ; If not, discard the unlabeled flame video condition sample;

[0059] Step S4-4: Calculate the similarity of local working condition features between the flame video underburning working condition samples with known labels and the flame video working condition samples without labels , set the similarity threshold of the local working condition features between the flame video underburned working condition samples and the unlabeled flame video working condition samples , judge the similarity of local working condition characteristics Is it greater than the threshold? , if it is greater than, the under-burning condition pseudo label is calibrated for the unlabeled flame video condition sample; if it is not greater than, the unlabeled flame video condition sample is discarded.

[0060] Furthermore, the feature similarity is calculated as follows:

[0061] (17)

[0062] in, Flame video representing known tags Comparison of abnormal working condition samples and unlabeled flame video working condition samples Class feature similarity, Represents an unlabeled flame video sample Class features, Represents a flame video sample with known labels Class features, Flame video representing known tags Abnormal working condition samples Class features, Flame video representing known tags Abnormal working condition samples Class feature set, Represents a flame video sample with known labels Class feature set, Class characteristics are global operating condition characteristics or local operating condition characteristics. The abnormal operating conditions are over-burning or under-burning; is a constant, and its value is Square root of the mean.

[0063] Furthermore, the feature similarity threshold is set as follows: Abnormal working condition samples The class feature similarities are added together and the average value is used as the feature similarity threshold.

[0064] Compared with the prior art, the advantages of the present invention are as follows:

[0065] In order to solve the problem of small samples of abnormal working conditions caused by unbalanced working condition data, a pseudo-label learning method based on the similarity of global working condition features and local working condition features is adopted. On the basis of augmenting the data of small sample abnormal working conditions, the bias of feature learning model towards unbalanced data is alleviated, thereby improving the recognition accuracy of small sample abnormal working conditions. The same working conditions are judged based on the distribution of original data and the data distribution of features extracted by the network. Compared with direct judgment using classifiers, the working conditions are more credible, and more information is provided for network input, improving the accuracy of feature extraction network and classifier.

[0066] In order to solve the problem of unstable single-frame working condition feature information in the sintering process of the rotary kiln, the video Transformer method is used to model and analyze the flame video to improve the robustness and accuracy of the recognition model, and solve the problem of insufficient working condition features of single-frame flame video in the process of identifying abnormal working conditions of the rotary kiln. In order to solve the problem of high complexity in calculating the global working condition features of the flame video, the 3D window focused linear multi-head self-attention module is used to calculate the Linear Attention self-attention to obtain the global working condition features, and the 3D window multi-head self-attention module is used to calculate the SoftMax self-attention to obtain the local working condition features, which can effectively reduce the computational complexity of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a schematic diagram of a method according to an embodiment of the present invention.

[0068] Figure 2 yes Figure 1The illustrated embodiment is a schematic diagram of a sintering condition recognition model trained using flame video condition samples with known labels.

[0069] Figure 3 yes Figure 2 Schematic diagram of the structure of the video linear attention network of the illustrated embodiment.

[0070] Figure 4 yes Figure 1 A schematic diagram of the structure of the video Swin Transformer network of the illustrated embodiment.

[0071] Figure 5 yes Figure 1 The illustrated embodiment is a flow chart for calibrating pseudo labels for abnormal operating conditions. DETAILED DESCRIPTION

[0072] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0073] Reference Figure 1-5 In this embodiment, a small sample sintering condition recognition method based on flame video multi-scale attention modeling includes the following steps:

[0074] Step S1: Collecting the on-site flame video working condition data of the rotary kiln, and preprocessing it, constructing a flame video working condition sample, and marking the working condition category label for the data of known working conditions;

[0075] The flame video preprocessing module preprocesses the flame video: extracts the flame video frame and inputs it into the Transformer network. The Patch Partition layer of the Transformer network divides the input flame video frame into blocks; the extracted flame video frame is ,in Indicates the number of input frames, that is, the depth, each frame contains Pixels, is the height of the video image frame, is the width of the video image frame, 3 represents the three color channels of the video image frame, that is, the input flame video frame is , the Patch Partition layer of the Transformer network divides the input video frame into 3D blocks to obtain tokens, each of which is , with a total size of Non-overlapping tokens, each of which has a 96-dimensional feature vector, are used as the input of the subsequent working condition feature extraction module M2, and token is the unit of tokens.

[0076] Step S2: Use the known labeled flame video working condition samples to train the sintering working condition recognition model: Input the known labeled flame video working condition samples into the global working condition feature extraction network, extract the global working condition features of the flame video, and form a flame video feature sequence , the flame video feature sequence Input the local working condition feature extraction network to extract the local working condition features of the flame video and form a flame video feature sequence ; The flame video feature sequence Input the classifier to update the parameters of the sintering condition recognition model;

[0077] The sintering condition recognition model includes a global condition feature extraction network, a local condition feature extraction network, a pseudo-label learning module and a classifier. The global condition feature extraction network is connected to the local condition feature extraction network and the pseudo-label learning module respectively, and the local condition feature extraction network is connected to the pseudo-label learning module and the classifier respectively. The global condition feature extraction network is used to extract the global condition features of the flame video, the local condition feature extraction network is used to extract the local condition features of the flame video, and the pseudo-label learning module is used to calculate the similarity of the global condition features and local condition features of the flame video abnormal condition samples with known labels and the unlabeled flame video condition samples. According to the feature similarity, the unlabeled flame video condition samples are calibrated with the corresponding abnormal condition pseudo-labels, or the unlabeled flame video condition samples are discarded. The global condition feature extraction network and the local condition feature extraction network constitute a feature extraction module based on multi-scale attention.

[0078] The global working condition feature extraction network includes a flame video linear embedding network, a first video linear attention (VLA) network, a first spatial feature downsampling module, and a second video linear attention network. The flame video linear embedding network is connected to the first video linear attention network, the first video linear attention network is connected to the first spatial feature downsampling module, and the first spatial feature downsampling module is connected to the second video linear attention network.

[0079] To ensure that each dimension (T, H, W) is divisible by the corresponding value in the token, the Flame Video Linear Embedding Network first pads the length H, width W, and depth T of the input token, and then uses a 3D convolutional layer to map each token block to a C-dimensional space. The C-dimensional space is a vector space with C dimensions, where C is a positive integer. After linear embedding, tokens are represented by becomes .

[0080] The first video linear attention network is the same as the second video linear attention network, including a first normalization layer (LN), a first 3D window focus linear multi-head self-attention module (3D Focus Linear Attention Multi-headSelf-Attention: 3DLA3DFLA-MSA), a second normalization layer, a first multilayer perceptron (MultilayerPerceptron: MLP), a third normalization layer, a second 3D window focus linear multi-head self-attention module, a fourth normalization layer, and a second multilayer perceptron. The first normalization layer is connected to the first 3D window focus linear multi-head self-attention module, the first 3D window focus linear multi-head self-attention module is connected to the second normalization layer, the second normalization layer is connected to the first multilayer perceptron, the first multilayer perceptron is connected to the third normalization layer, the third normalization layer is connected to the second 3D window focus linear multi-head self-attention module, the second 3D window focus linear multi-head self-attention module is connected to the fourth normalization layer, and the fourth normalization layer is connected to the second multilayer perceptron.

[0081] The first 3D window focused linear multi-head self-attention module is the same as the second 3D window focused linear multi-head self-attention module, which is used to calculate the self-attention with focused Linear Attention in a 3D window of a specified size to extract the global working condition features of the flame video.

[0082] Input Features After the first normalization layer and the first 3D window focused linear multi-head self-attention module, the output features ,feature With features Sum and get the features ,feature Output features after the second normalization layer and the first multi-layer perceptron ,feature With features Sum and get the features ,feature After the third normalization layer and the second 3D window focused linear multi-head self-attention module, the output features ,feature With features Sum and get the features ,feature After the fourth normalization layer and the second multi-layer perceptron, the output feature ,feature With features Sum and get the features The specific process expression for extracting the global working condition features of flame video is:

[0083] The specific expression is:

[0084] (1)

[0085] (2)

[0086] (3)

[0087] (4)

[0088] For the first video linear attention network, the equations (1) to (4) , for the second video linear attention network, the equations (1) to (4) ,feature ,feature ,feature ,feature .

[0089] In order to extract the global working condition features, the Linear Attention self-attention weight matrix is ​​obtained by using the first 3D window focused linear multi-head self-attention module and the second 3D window focused linear multi-head self-attention module. The 3D window size is specified as 8×56×56 in the first video linear attention network, and the 3D window size is specified as 8×28×28 in the second video linear attention network. The specific operations of the first 3D window focused linear multi-head self-attention module and the second 3D window focused linear multi-head self-attention module are as follows:

[0090] Through linear mapping, we can get matrix , matrix ， matrix ,Right now:

[0091] (5)

[0092] In formula (5), for the first 3D window focused linear multi-head self-attention module, Represents the input features of the first 3D window focused linear multi-head self-attention module, that is, the features obtained after data normalization by the first normalization layer, as the starting input of the first 3D window focused linear multi-head self-attention module , , Indicates the number of tokens feature vectors, represents the feature dimension, Represents the matrix size and shape; for the second 3D window focused linear multi-head self-attention module, Represents the input features of the second 3D window focused linear multi-head self-attention module, that is, the features obtained after data normalization by the third normalization layer. express matrix The learnable parameters of express matrix The learnable parameters of express matrix The learnable parameters of , express Features or The dimension of the feature.

[0093] In order to improve the prominence of similar features, Focus Function is used as the kernel function right matrix Perform nonlinear feature mapping to obtain the focused feature matrix ,right matrix Perform nonlinear feature mapping to obtain the focused feature matrix ; Mathematical expression of Focus Function for:

[0094] (6)

[0095] (7)

[0096] The input in (6) and (7) refer to matrix or matrix , Represents the Rectified Linear Unit activation function; Express Level Power operation.

[0097] In getting the focused feature matrix and Then calculate the Linear Attention self-attention weight matrix , among which Linear Attention self-attention weights The calculation formula is:

[0098] (8)

[0099] In formula (8), express matrix Middle feature vectors, express matrix Middle feature vectors, express matrix Middle feature vectors, , Represents the focused feature matrix The focus feature in , Represents the focused feature matrix The focus feature; Indicates the number of tokens feature vectors.

[0100] Since the Linear Attention self-attention weight matrix is ​​obtained through Linear Attention self-attention calculation There is a low-rank problem, which leads to low feature diversity and limits the representation ability of the model. In order to improve feature diversity, the present invention uses a 3D depth-wise convolution module with a time dimension to matrix Perform a three-dimensional convolution operation and then superimpose it on the Linear Attention self-attention weight matrix , get the final Linear Attention self-attention weight matrix The mathematical expression is:

[0101] (9)

[0102] In the above and They represent the final Linear Attention self-attention weight matrices of the first 3D window focused linear multi-head self-attention module and the second 3D window focused linear multi-head self-attention module respectively.

[0103] The first spatial feature downsampling module achieves downsampling by splicing adjacent patches, reducing resolution, adjusting the number of channels, and saving computational effort while maintaining information integrity, and finally reducing the number of channels through a linear layer. Each downsampling is doubled, that is, elements are selected at intervals of 2 in the row and column directions to form a new patch, and then all patches are connected as a whole tensor. At this time, the channel dimension will become 4 times the original, and finally a fully connected layer is used to adjust the channel dimension to 2 times the original.

[0104] The local working condition feature extraction network includes a second spatial feature downsampling module, a first video Swin Transformer (Video Swin Transformer: VST) network, a third spatial feature downsampling module, and a second video Swin Transformer (Video Swin Transformer: VST) network. The second spatial feature downsampling module is connected to the first video Swin Transformer network, the first video Swin Transformer network is connected to the third spatial feature downsampling module, and the third spatial feature downsampling module is connected to the second video Swin Transformer network.

[0105] The Swin Transformer network of the first video is the same as the Swin Transformer network of the second video, including the fifth normalization layer (LN), a 3D window multi-head self-attention module (3DW-MSA), a sixth normalization layer, a third multilayer perceptron (Multilayer Perceptron: MLP), a seventh normalization layer, a 3D moving window multi-head self-attention module (3DSW-MSA), an eighth normalization layer, and a fourth multilayer perceptron. The fifth normalization layer is connected to the 3D window multi-head self-attention module, the 3D window multi-head self-attention module is connected to the sixth normalization layer, the sixth normalization layer is connected to the third multilayer perceptron, the third multilayer perceptron is connected to the seventh normalization layer, the seventh normalization layer is connected to the 3D moving window multi-head self-attention module, the 3D moving window multi-head self-attention module is connected to the eighth normalization layer, and the eighth normalization layer is connected to the fourth multilayer perceptron.

[0106] The input of the first video Swin Transformer network is the output feature of the second spatial feature downsampling module .feature The data is normalized by the fifth normalization layer and then passes through the 3D window multi-head self-attention module to output features. ,feature With features Sum and get the features ,feature Output features after the sixth normalization layer and the third multi-layer perceptron ,feature With features Sum and get the features ,feature After the seventh normalization layer and the 3D moving window multi-head self-attention module, the output features ,feature With features Sum and get the features ,feature After the eighth normalization layer and the fourth multi-layer perceptron, the output feature ,feature With features Sum and get the final output features of the Swin Transformer network of the first video The specific process expression of local working condition feature extraction is:

[0107] (10)

[0108] (11)

[0109] (12)

[0110] (13)

[0111] For the first video Swin Transformer network, the equations (10) to (13) are , for the second video Swin Transformer network, the equations (10) to (13) are ,feature ,feature ,feature ,feature .

[0112] In order to extract local working condition features, the 3D window multi-head self-attention module and the 3D moving window multi-head self-attention module are used to obtain the SoftMax self-attention weight matrix. The 3D window multi-head self-attention module is used to calculate the SoftMax self-attention in a 3D window of a specified size to extract local working condition features. The 3D moving window multi-head self-attention module is used to calculate the SoftMax self-attention by moving the 3D window in a 3D window of a specified size to extract local working condition features.

[0113] The 3D window size specified in the 3D window multi-head self-attention module and the 3D moving window multi-head self-attention module is (length × width × height). Compared with the 3D window multi-head self-attention module, the 3D moving window multi-head self-attention module requires one more step, namely the movement of the 3D window. The specific process of obtaining the SoftMax self-attention matrix through the 3D window multi-head self-attention module is as follows:

[0114] The characteristics After inputting into the fifth normalization layer, the normalized features are obtained. , and then this feature Input into the 3D window multi-head self-attention module;

[0115] feature After entering the 3D window multi-head self-attention module, linear mapping is performed to obtain matrix , matrix and matrix ,Right now:

[0116] (14)

[0117] In formula (14), express matrix The learnable parameters of express matrix The learnable parameters of express matrix The learnable parameters of .

[0118] get matrix , matrix and matrix After that, in the size of The SoftMax self-attention weight matrix is ​​calculated in the 3D window. The calculation expression of the SoftMax self-attention weight matrix is:

[0119] (15)

[0120] (16)

[0121] In formulas (15) and (16), represents the SoftMax self-attention weight matrix, express matrix , matrix The characteristic dimension of Indicates relative position deviation.

[0122] For the 3D moving window multi-head self-attention module, the specific process of obtaining the SoftMax self-attention matrix is ​​different from that of the 3D window multi-head self-attention module only in that:

[0123] get matrix , matrix and matrix After that, before calculating the SoftMax self-attention weight matrix, the 3D window is moved forward (in time and sequence frame direction) P / 2 distance, move to the right (in the right spatial direction) M / 2 The distance rounded down, moving downward (in the downward spatial direction) M / 2 The distance rounded down, that is, the overall movement of the 3D window The rest of the process is the same as the 3D window multi-head self-attention module to obtain the SoftMax self-attention matrix.

[0124] Assume that the input tokens size is , set the window size , then in the 3D window multi-head self-attention module, Then, in the 3D moving window multi-head self-attention module, the window moves to a position where the window is half the size, forming The window moves with the window, successfully solving the problem of the previous layer of tokens being hindered from interacting with each other due to 3D non-overlapping windows, and enhancing the spatiotemporal information exchange. The window calculates the SoftMax self-attention weight matrix within the window.

[0125] The second spatial feature downsampling module and the third spatial feature downsampling module are both designed to reduce the computational complexity of the model, improve the representation capability of the model's hierarchical features, and enrich the semantic features.

[0126] Step S3: After the sintering condition recognition model is trained using the flame video condition samples with known labels, the unlabeled flame video condition samples are input into the global condition feature extraction network to extract the global condition features of the flame video to form a flame video feature sequence , the flame video feature sequence Input the local working condition feature extraction network to extract the local working condition features of the flame video and form a flame video feature sequence ;

[0127] Step S4: Calculate the similarity of the global working condition features and local working condition features of the flame video abnormal working condition samples with known labels and the flame video working condition samples without labels, and calibrate the corresponding abnormal working condition pseudo labels for the flame video working condition samples without labels according to the feature similarity, or discard the flame video working condition samples without labels; specifically, the following steps are included:

[0128] Step S4-1: Calculate the global working condition feature similarity between the overburning flame video working condition sample with known label and the flame video working condition sample without label , set the global working condition feature similarity threshold between the overburned flame video working condition sample and the unlabeled flame video working condition sample , determine the similarity of global working condition characteristics Is it greater than the threshold? If it is greater than, go to step S5-2 to further calculate the similarity of the local working condition features of the flame video overburning working condition samples with known labels and the flame video working condition samples without labels. If not, proceed to step S4-3 to further calculate the global working condition feature similarity between the flame video underburned working condition sample with known labels and the flame video working condition sample without labels. ;

[0129] Step S4-2: Calculate the similarity of local working condition features between the flame video overburning working condition samples with known labels and the flame video working condition samples without labels , set the similarity threshold of the local working condition features between the flame video overburning working condition samples and the unlabeled flame video working condition samples , judge the similarity of local working condition characteristics Is it greater than the threshold? If it is greater than, the unlabeled flame video working condition sample is calibrated with an overburning working condition pseudo label; if it is not greater than, go to step S4-3 to further calculate the global working condition feature similarity between the flame video underburning working condition sample with a known label and the unlabeled flame video working condition sample ;

[0130] Step S4-3: Calculate the global working condition feature similarity between the flame video underburning working condition samples with known labels and the flame video working condition samples without labels , set the global working condition feature similarity threshold between the flame video underburned working condition sample and the unlabeled flame video working condition sample , determine the similarity of global working condition characteristics Is it greater than the threshold? If it is greater than, go to step S4-4 to further calculate the similarity of the local working condition features of the flame video underburning working condition samples with known labels and the flame video working condition samples without labels ; If not, discard the unlabeled flame video condition sample;

[0131] Step S4-4: Calculate the similarity of local working condition features between the flame video underburning working condition samples with known labels and the flame video working condition samples without labels , set the similarity threshold of the local working condition features between the flame video underburned working condition samples and the unlabeled flame video working condition samples , judge the similarity of local working condition characteristics Is it greater than the threshold? , if it is greater than, the under-burning condition pseudo label is calibrated for the unlabeled flame video condition sample; if it is not greater than, the unlabeled flame video condition sample is discarded.

[0132] The feature similarity is calculated as follows:

[0133] (17)

[0134] in, Flame video representing known tags Comparison of abnormal working condition samples and unlabeled flame video working condition samples Class feature similarity, Represents an unlabeled flame video sample Class features, Represents a flame video sample with known labels Class features, Flame video representing known tags Abnormal working condition samples Class features, Flame video representing known tags Abnormal working condition samples Class feature set, Represents a flame video sample with known labels Class feature set, Class characteristics are global operating condition characteristics or local operating condition characteristics. The abnormal operating conditions are over-burning condition or under-burning condition. is a constant, and its value is Square root of the mean.

[0135] The method for setting the feature similarity threshold is: Abnormal working condition samples The class feature similarities are added together and the average value is used as the feature similarity threshold.

[0136] Step S5: adding the flame video condition samples marked with abnormal condition pseudo labels to the flame video condition samples with known labels to form a training sample set with a more balanced sample distribution, and using the training sample set to retrain the sintering condition recognition model.

[0137] The experimental data of this example are collected from rotary kilns No. 5 and No. 6 of Aluminum Corporation of China Zhongzhou Branch. A color charge coupled device (CCD) camera is installed at the kiln head of the rotary kiln. The signal collected by the CCD is digitized to obtain a speed of 25 frames per second, and the size of each frame is , a flame video in RGB format with a precision of 24 bits. To test the performance of the algorithm, the experiment used flame video data recorded within 30 days, and a total of 5083 labeled working condition samples were obtained. The specific number of working condition category samples and data set construction information are shown in Table 1:

[0138] Table 1 .

[0139] Table 2 is the ablation experiment results of the working condition recognition of this embodiment. Table 2 uses C3D, SlowFast-R101, Video Swin Transformer and different working condition recognition methods of this embodiment respectively, where C3D refers to Convolutional 3D, that is, a three-dimensional convolution method, SlowFast-R101 refers to a method of combining the SlowFast model with the ResNet-101 network, and Video Swin Transformer is a Transformer model specially designed for video understanding tasks.

[0140] It can be seen from Table 2 that after the working condition samples of this example are processed by the above method, the classification accuracy under normal working condition, under-burning working condition and over-burning working condition reaches 98.65%, 98.74% and 92.38% respectively. Compared with other working condition identification methods, the working condition classification accuracy of the method of the present invention is the highest:

[0141] Table 2 .

[0142] The above experiment was simulated on a server. The server uses an NVIDIA RTX A6000 CPU, runs at 1455MHz, and has a memory of 48GB. The running software is Python. Those skilled in the art may make various modifications and variations to the present invention. If these modifications and variations are within the scope of the claims of the present invention and their equivalents, then these modifications and variations are also within the protection scope of the present invention.

[0143] Those skilled in the art may make various modifications and variations to the present invention. If these modifications and variations are within the scope of the claims of the present invention and their equivalents, then these modifications and variations are also within the protection scope of the present invention.

[0144] The contents not described in detail in the specification are prior art known to those skilled in the art.

Claims

1. A small sample sintering condition recognition method based on flame video multi-scale attention modeling, characterized in that: The following methods are included: Step S1: Collecting the on-site flame video working condition data of the rotary kiln, and preprocessing it, constructing a flame video working condition sample, and marking the working condition category label for the data of known working conditions; Step S2: Use the known labeled flame video working condition samples to train the sintering working condition recognition model: Input the known labeled flame video working condition samples into the global working condition feature extraction network, extract the global working condition features of the flame video, and form a flame video feature sequence , the flame video feature sequence Input the local working condition feature extraction network to extract the local working condition features of the flame video and form a flame video feature sequence ; The flame video feature sequence Input the classifier to update the parameters of the sintering condition recognition model; Step S3: After the sintering condition recognition model is trained using the flame video condition samples with known labels, the unlabeled flame video condition samples are input into the global condition feature extraction network to extract the global condition features of the flame video to form a flame video feature sequence , the flame video feature sequence Input the local working condition feature extraction network to extract the local working condition features of the flame video and form a flame video feature sequence ; Step S4: Calculate the similarity of the global working condition features and local working condition features of the flame video abnormal working condition samples with known labels and the flame video working condition samples without labels, and calibrate the corresponding abnormal working condition pseudo labels for the flame video working condition samples without labels according to the feature similarity, or discard the flame video working condition samples without labels; Step S5: adding the flame video condition samples marked with abnormal condition pseudo labels to the flame video condition samples with known labels to form a training sample set with a more balanced sample distribution, and using the training sample set to retrain the sintering condition recognition model.

2. The small sample sintering condition recognition method based on flame video multi-scale attention modeling according to claim 1 is characterized in that: In step S1, the method for preprocessing the rotary kiln on-site flame video working condition data is: extracting flame video frames, inputting them into the Transformer network, and the Patch Partition layer of the Transformer network divides the input flame video frames into blocks.

3. The small sample sintering condition recognition method based on flame video multi-scale attention modeling according to claim 1 is characterized in that: In step S2, the sintering condition recognition model includes a global condition feature extraction network, a local condition feature extraction network, a pseudo-label learning module and a classifier. The global condition feature extraction network is respectively connected to the local condition feature extraction network and the pseudo-label learning module, and the local condition feature extraction network is respectively connected to the pseudo-label learning module and the classifier; the global condition feature extraction network is used to extract the global condition features of the flame video, the local condition feature extraction network is used to extract the local condition features of the flame video, and the pseudo-label learning module is used to calculate the similarity of the global condition features and the local condition features of the abnormal condition samples of the flame video with known labels and the unlabeled flame video condition samples, and according to the feature similarity, the corresponding abnormal condition pseudo-label is calibrated for the unlabeled flame video condition samples, or the unlabeled flame video condition samples are discarded.

4. The small sample sintering condition recognition method based on flame video multi-scale attention modeling as claimed in claim 2 is characterized by: In step S2, the global working condition feature extraction network includes a flame video linear embedding network, a first video linear attention network, a first spatial feature downsampling module, and a second video linear attention network. The flame video linear embedding network is connected to the first video linear attention network, the first video linear attention network is connected to the first spatial feature downsampling module, and the first spatial feature downsampling module is connected to the second video linear attention network; the first video linear attention network performs a first global working condition feature extraction on the output of the flame video linear embedding network, the extraction result is downsampled by the first spatial feature downsampling module, and then the second global working condition feature extraction is performed by the second video linear attention network; The first video linear attention network is the same as the second video linear attention network, including a first normalization layer, a first 3D window focused linear multi-head self-attention module, a second normalization layer, a first multi-layer perceptron, a third normalization layer, a second 3D window focused linear multi-head self-attention module, a fourth normalization layer, and a second multi-layer perceptron. The first normalization layer is connected to the first 3D window focused linear multi-head self-attention module, the first 3D window focused linear multi-head self-attention module is connected to the second normalization layer, the second normalization layer is connected to the first multi-layer perceptron, the first multi-layer perceptron is connected to the third normalization layer, the third normalization layer is connected to the second 3D window focused linear multi-head self-attention module, the second 3D window focused linear multi-head self-attention module is connected to the fourth normalization layer, and the fourth normalization layer is connected to the second multi-layer perceptron. The first 3D window focused linear multi-head self-attention module is the same as the second 3D window focused linear multi-head self-attention module, and is used to calculate the self-attention with focused Linear Attention in a 3D window of a specified size to extract the global working condition characteristics of the flame video. Input Features After the first normalization layer and the first 3D window focused linear multi-head self-attention module, the output features ,feature With features Sum and get the features ,feature Output features after the second normalization layer and the first multi-layer perceptron ,feature With features Sum and get the features ,feature After the third normalization layer and the second 3D window focused linear multi-head self-attention module, the output features ,feature With features Sum and get the features ,feature After the fourth normalization layer and the second multi-layer perceptron, the output feature ,feature With features Sum and get the features ; The specific process expression for extracting the global working condition features of flame video is: The specific expression is: (1) (2) (3) (4) For the first video linear attention network, the equations (1) to (4) , for the second video linear attention network, the equations (1) to (4) ,feature ,feature ,feature ,feature .

5. The small sample sintering condition recognition method based on flame video multi-scale attention modeling as claimed in claim 4 is characterized by: The first 3D window focused linear multi-head self-attention module and the second 3D window focused linear multi-head self-attention module obtain a Linear Attention self-attention weight matrix, specifically: Through linear mapping, we can get matrix , matrix ， matrix ,Right now: (5) In formula (5), for the first 3D window focused linear multi-head self-attention module, represents the input features of the first 3D window focused linear multi-head self-attention module, that is, the features obtained after data normalization by the first normalization layer; for the second 3D window focused linear multi-head self-attention module, Represents the input features of the second 3D window focused linear multi-head self-attention module, that is, the features obtained after data normalization by the third normalization layer; express matrix The learnable parameters of express matrix The learnable parameters of express matrix The learnable parameters of , express Features or The dimension of the feature; Using Focus Function as the kernel function right matrix Perform nonlinear feature mapping to obtain the focused feature matrix ,right matrix Perform nonlinear feature mapping to obtain the focused feature matrix ; Mathematical expression of Focus Function for: (6) (7) The input in (6) and (7) refer to matrix or matrix , Represents the Rectified Linear Unit activation function; Express Level Power operation; Get the focused feature matrix and After that, calculate the Linear Attention self-attention weight matrix , among which Linear Attention self-attention weights The calculation formula is: (8) In formula (8), express matrix Middle feature vectors, express matrix Middle feature vectors, express matrix Middle feature vectors, , Represents the focused feature matrix The focus feature in , Represents the focused feature matrix The focus feature; Indicates the number of block feature vectors; Using 3D depthwise separable convolutional modules with time dimension matrix Perform a three-dimensional convolution operation and then superimpose it on the Linear Attention self-attention weight matrix , get the final Linear Attention self-attention weight matrix The mathematical expression is: (9) Said and They represent the final Linear Attention self-attention weight matrices of the first 3D window focused linear multi-head self-attention module and the second 3D window focused linear multi-head self-attention module respectively.

6. The small sample sintering condition recognition method based on flame video multi-scale attention modeling according to claim 1 is characterized in that: In step S2, the local working condition feature extraction network includes a second spatial feature downsampling module, a first video Swin Transformer network, a third spatial feature downsampling module, and a second video Swin Transformer network. The second spatial feature downsampling module is connected to the first video Swin Transformer network, the first video Swin Transformer network is connected to the third spatial feature downsampling module, and the third spatial feature downsampling module is connected to the second video Swin Transformer network; the second spatial feature downsampling module downsamples the output of the second video linear attention network, and the downsampling result is subjected to the first local working condition feature extraction through the first video Swin Transformer network, the extraction result is downsampled through the third spatial feature downsampling module, and the downsampling result is subjected to the second local working condition feature extraction through the second video Swin Transformer network; The Swin Transformer network of the first video is the same as the Swin Transformer network of the second video, including a fifth normalization layer, a 3D window multi-head self-attention module, a sixth normalization layer, a third multi-layer perceptron, a seventh normalization layer, a 3D moving window multi-head self-attention module, an eighth normalization layer, and a fourth multi-layer perceptron. The fifth normalization layer is connected to the 3D window multi-head self-attention module, the 3D window multi-head self-attention module is connected to the sixth normalization layer, the sixth normalization layer is connected to the third multi-layer perceptron, the third multi-layer perceptron is connected to the seventh normalization layer, the seventh normalization layer is connected to the 3D moving window multi-head self-attention module, the 3D moving window multi-head self-attention module is connected to the eighth normalization layer, and the eighth normalization layer is connected to the fourth multi-layer perceptron; the 3D window multi-head self-attention module is used to calculate SoftMax self-attention in a 3D window of a specified size to extract local working condition features; 3D moving window multi-head self-attention module, which is used to calculate SoftMax self-attention by moving the 3D window in a 3D window of a specified size, and then extract local working condition features; Features The data is normalized by the fifth normalization layer and then passes through the 3D window multi-head self-attention module to output features. ,feature With features Sum and get the features ,feature Output features after the sixth normalization layer and the third multi-layer perceptron ,feature With features Sum and get the features ,feature After the seventh normalization layer and the 3D moving window multi-head self-attention module, the output features ,feature With features Sum and get the features ,feature After the eighth normalization layer and the fourth multi-layer perceptron, the output feature ,feature With features Sum and get the final output features of the Swin Transformer network of the first video ; The specific process expression for extracting local working condition features is: (10) (11) (12) (13) For the first video Swin Transformer network, the equations (10) to (13) are , for the second video SwinTransformer network, the equations (10) to (13) are ,feature ,feature ,feature ,feature .

7. The small sample sintering condition recognition method based on flame video multi-scale attention modeling according to claim 6 is characterized in that: The specific method for the 3D window multi-head self-attention module to obtain the SoftMax self-attention matrix is: The characteristics After inputting into the fifth normalization layer, the normalized features are obtained. , and then this feature Input into the 3D window multi-head self-attention module; feature After entering the 3D window multi-head self-attention module, linear mapping is performed to obtain matrix , matrix and matrix ,Right now: (14) In formula (14), express matrix The learnable parameters of express matrix The learnable parameters of express matrix The learnable parameters of get matrix , matrix and matrix After that, in the size of The SoftMax self-attention weight matrix is ​​calculated in the 3D window; the calculation expression of the SoftMax self-attention weight matrix is: (15) (16) In formulas (15) and (16), represents the SoftMax self-attention weight matrix, express matrix , matrix The characteristic dimension of Indicates relative position deviation; The specific method for the 3D moving window multi-head self-attention module to obtain the SoftMax self-attention weight matrix is: get matrix , matrix and matrix Then, move the 3D window forward. P / 2 distance, move right M / 2 round down the distance, move downward M / 2 The distance rounded down, that is, the overall movement of the 3D window The distance rounded down; in the moved 3D window, the SoftMax self-attention weight matrix is ​​calculated.

8. The small sample sintering condition recognition method based on flame video multi-scale attention modeling according to claim 1 is characterized in that: Step S4 The specific steps include: Step S4-1: Calculate the global working condition feature similarity between the overburning flame video working condition sample with known label and the flame video working condition sample without label , set the global working condition feature similarity threshold between the overburned flame video working condition sample and the unlabeled flame video working condition sample , determine the similarity of global working condition characteristics Is it greater than the threshold? If it is greater than, go to step S5-2 to further calculate the similarity of the local working condition features of the flame video overburning working condition samples with known labels and the flame video working condition samples without labels. If not, proceed to step S4-3 to further calculate the global working condition feature similarity between the flame video underburned working condition sample with known labels and the flame video working condition sample without labels. ; Step S4-2: Calculate the similarity of local working condition features between the flame video overburning working condition samples with known labels and the flame video working condition samples without labels , set the similarity threshold of the local working condition features between the flame video overburning working condition samples and the unlabeled flame video working condition samples , judge the similarity of local working condition characteristics Is it greater than the threshold? If it is greater than, the unlabeled flame video working condition sample is calibrated with an overburning working condition pseudo label; if it is not greater than, go to step S4-3 to further calculate the global working condition feature similarity between the flame video underburning working condition sample with a known label and the unlabeled flame video working condition sample ; Step S4-3: Calculate the global working condition feature similarity between the flame video underburning working condition samples with known labels and the flame video working condition samples without labels , set the global working condition feature similarity threshold between the flame video underburned working condition sample and the unlabeled flame video working condition sample , determine the similarity of global working condition characteristics Is it greater than the threshold? If it is greater than, go to step S4-4 to further calculate the similarity of the local working condition features of the flame video underburning working condition samples with known labels and the flame video working condition samples without labels ; If not, discard the unlabeled flame video condition sample; Step S4-4: Calculate the similarity of local working condition features between the flame video underburning working condition samples with known labels and the flame video working condition samples without labels , set the similarity threshold of the local working condition features between the flame video underburned working condition samples and the unlabeled flame video working condition samples , judge the similarity of local working condition characteristics Is it greater than the threshold? , if it is greater than, the under-burning condition pseudo label is calibrated for the unlabeled flame video condition sample; if it is not greater than, the unlabeled flame video condition sample is discarded.

9. The small sample sintering condition recognition method based on flame video multi-scale attention modeling as claimed in claim 8, characterized in that: The feature similarity is calculated as follows: （17） in, Flame video representing known tags Comparison of abnormal working condition samples and unlabeled flame video working condition samples Class feature similarity, Represents an unlabeled flame video sample Class features, Represents a flame video sample with known labels Class features, Flame video representing known tags Abnormal working condition samples Class features, Flame video representing known tags Abnormal working condition samples Class feature set, Represents a flame video sample with known labels Class feature set, Class characteristics are global operating condition characteristics or local operating condition characteristics. The abnormal operating conditions are over-burning or under-burning; is a constant, and its value is Square root of the mean.

10. The small sample sintering condition recognition method based on flame video multi-scale attention modeling according to claim 9, characterized in that: The method for setting the feature similarity threshold is: Abnormal working condition samples The class feature similarities are added together and the average value is used as the feature similarity threshold.

Images