Coal-burning condition recognition method and system based on flame video multi-scale attention

Abstract

A method and system for identifying coal-burning conditions based on flame video multi-scale attention, the method includes flame video preprocessing: a flame video linear embedding network fills the blocks of the video frame, and uses a 3D convolution layer to map each block to a C-dimensional space; each mapped block is input into a global condition feature extraction network to extract the global condition features of the flame video sample; the global condition features are input into a local condition feature extraction network to extract the local condition features of the flame video sample; a classifier classifies the conditions according to the local condition features of the flame video to obtain a classification result. Also included is a coal-burning condition identification system based on flame video multi-scale attention. The present invention improves the robustness and accuracy of the recognition model while reducing the computational complexity of the model.

Description

Technical Field

[0001] The present invention relates to coal burning condition recognition, and in particular to a coal burning condition recognition method based on flame video multi-scale attention and a system thereof. Background Art

[0002] Traditional industries such as steel, nonferrous metals, building materials, and electricity often use coal to produce energy and calcine materials. They are the main energy-consuming industries and the main source of carbon emissions. Robust and accurate detection of the system's coal-burning conditions is an important part of achieving efficient system control and is the key to reducing carbon emissions, improving product quality, and reducing energy consumption. Due to the complex equipment and harsh on-site environment of coal-fired high-temperature industrial processes, the existing comprehensive production control level is relatively backward, making it difficult to ensure long-term stable and optimized operation of the entire production process.

[0003] Flame video images directly reflect the combustion conditions. In recent years, with the development of big data, image processing technology and pattern recognition technology, many scholars have used flame images to conduct extensive research on production conditions and product quality. This type of method first extracts frames from the flame video to form a flame image, and then enhances the quality of the flame image and identifies the working conditions. However, due to the flickering characteristics of flames, this type of working condition recognition method based on a single frame is difficult to capture the dynamic changes of the flame, and the robustness and recognition accuracy are difficult to meet the needs of the site. Compared with images, videos can analyze the correlation between multiple frames of images, use the information of multiple frames of images for mutual calibration and verification, reduce errors and uncertainties, and thus obtain richer and more robust working condition information. Secondly, CNN obtains local detail information through convolution images, making it particularly capable of processing the boundary texture information of objects in the image, but it is hindered when learning low-level features of flames. This is due to the complex environment in the rotary kiln. The flame video acquisition will be affected by dust, smoke and high temperature. The texture features of the flame in the sintering area captured are weak, and there is no obvious boundary information between the flame and the surrounding environment.

[0004] Aiming at the problem of unstable characteristic information of single frame working condition in the sintering process of rotary kiln, how to improve the robustness and accuracy of the recognition model while reducing the computational complexity of the model is an urgent problem to be solved in the current rotary kiln working condition identification. Summary of the invention

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the above-mentioned background technology and provide a coal-burning condition recognition method and system based on multi-scale attention of flame video, so as to improve the robustness and accuracy of the recognition model while reducing the computational complexity of the model.

[0006] The technical solution adopted by the present invention to solve the technical problem is a coal-burning working condition recognition system based on flame video multi-scale attention, comprising: a flame video preprocessing module, a working condition feature extraction module, and a classifier, wherein the flame video preprocessing module is connected to the working condition feature extraction module, and the working condition feature extraction module is connected to the classifier;

[0007] The working condition feature extraction module includes a global working condition feature extraction network and a local working condition feature extraction network, and the global working condition feature extraction network is connected to the local working condition feature extraction network;

[0008] The flame video preprocessing module preprocesses the flame video;

[0009] The global working condition feature extraction network realizes the extraction of global working condition features of the flame video; the local working condition feature extraction network realizes the extraction of local working condition features of the flame video;

[0010] The classifier classifies the working conditions according to the local working condition characteristics of the flame video and obtains the classification results;

[0011] The global working condition feature extraction network includes a flame video linear embedding network, a first video linear attention network, a first spatial feature downsampling module, and a second video linear attention network. The flame video linear embedding network is connected to the first video linear attention network, the first video linear attention network is connected to the first spatial feature downsampling module, and the first spatial feature downsampling module is connected to the second video linear attention network.

[0012] The local working condition feature extraction network includes a second spatial feature downsampling module, a first video Swin Transformer network, a third spatial feature downsampling module, and a second video Swin Transformer network. The second spatial feature downsampling module is connected to the first video Swin Transformer network, the first video Swin Transformer network is connected to the third spatial feature downsampling module, and the third spatial feature downsampling module is connected to the second video Swin Transformer network.

[0013] Furthermore, the first video linear attention network is the same as the second video linear attention network, including a first normalization layer, a first 3D window focused linear multi-head self-attention module, a second normalization layer, a first multi-layer perceptron, a third normalization layer, a second 3D window focused linear multi-head self-attention module, a fourth normalization layer, and a second multi-layer perceptron. The first normalization layer is connected to the first 3D window focused linear multi-head self-attention module, the first 3D window focused linear multi-head self-attention module is connected to the second normalization layer, the second normalization layer is connected to the first multi-layer perceptron, the first multi-layer perceptron is connected to the third normalization layer, the third normalization layer is connected to the second 3D window focused linear multi-head self-attention module, the second 3D window focused linear multi-head self-attention module is connected to the fourth normalization layer, and the fourth normalization layer is connected to the second multi-layer perceptron. The first 3D window focused linear multi-head self-attention module is the same as the second 3D window focused linear multi-head self-attention module, and is used to calculate self-attention with focused Linear Attention in a 3D window of a specified size to extract global working condition features of the flame video.

[0014] Further, the first video Swin Transformer network is the same as the second video Swin Transformer network, including a fifth normalization layer, a 3D window multi-head self-attention module, a sixth normalization layer, a third multi-layer perceptron, a seventh normalization layer, a 3D moving window multi-head self-attention module, an eighth normalization layer, and a fourth multi-layer perceptron. The fifth normalization layer is connected to the 3D window multi-head self-attention module, the 3D window multi-head self-attention module is connected to the sixth normalization layer, the sixth normalization layer is connected to the third multi-layer perceptron, the third multi-layer perceptron is connected to the seventh normalization layer, and the seventh normalization layer is connected to the 3D moving window multi-head self-attention module. The 3D moving window multi-head self-attention module is connected to the eighth normalization layer, and the eighth normalization layer is connected to the fourth multi-layer perceptron; the 3D window multi-head self-attention module is used to calculate the SoftMax self-attention in a 3D window of a specified size, and then extract local working condition characteristics; the 3D moving window multi-head self-attention module is used to calculate the SoftMax self-attention by moving the 3D window in a 3D window of a specified size, and then extract local working condition characteristics.

[0015] A method for identifying coal-burning conditions based on flame video multi-scale attention includes the following steps:

[0016] Step S1: flame video preprocessing: extract flame video frames and input them into the Transformer network. The Patch Partition layer of the Transformer network divides the input flame video frames into blocks.

[0017] Step S2: The flame video linear embedding network fills the blocks of the video frame and uses a 3D convolutional layer to map each block into a C-dimensional space;

[0018] Step S3: input each mapped block into a global working condition feature extraction network to extract the global working condition features of the flame video sample;

[0019] Step S4: inputting the global working condition features into the local working condition feature extraction network to extract the local working condition features of the flame video sample;

[0020] Step S5: The classifier classifies the working conditions according to the local working condition characteristics of the flame video to obtain a classification result.

[0021] Further, in step S1, the flame video frame extracted is ,in T Indicates the number of input frames, that is, the depth, each frame contains Pixels, H is the height of the video image frame, W is the width of the video image frame, 3 represents the three color channels of the video image frame, that is, the input flame video frame is ,The PatchPartition layer of the Transformer network performs 3D block partitioning on the input video frame to obtain tokens.

[0022] Furthermore, in step S2, the flame video linear embedding network first pads the length H, width W, and depth T of the input token, and then uses a 3D convolutional layer to map each token block into a C-dimensional space. The C-dimensional space is a vector space with C dimensions, where C is a positive integer.

[0023] Further, in step S3, the first video linear attention network performs a first global working condition feature extraction on the output of the flame video linear embedding network, the extraction result is downsampled by the first spatial feature downsampling module, and then the second video linear attention network performs a second global working condition feature extraction; the method for extracting global working condition features by the first video linear attention network and the second video linear attention network is as follows:

[0024] Input Features After the first normalization layer and the first 3D window focused linear multi-head self-attention module, the output features ,feature With features Sum and get the features ,feature Output features after the second normalization layer and the first multi-layer perceptron ,feature With features Sum and get the features ,feature After the third normalization layer and the second 3D window focused linear multi-head self-attention module, the output features ,feature With features Sum and get the features ,feature After the fourth normalization layer and the second multi-layer perceptron, the output feature ,feature With features Sum and get the features , the specific expression is:

[0025] (1)

[0026] (2)

[0027] (3)

[0028] (4)

[0029] For the first video linear attention network, L = 1 in equations (1) to (4), and for the second video linear attention network, L = 3 in equations (1) to (4). ,feature ,feature ,feature .

[0030] Further, in step S3, the first 3D window focused linear multi-head self-attention module and the second 3D window focused linear multi-head self-attention module obtain a Linear Attention self-attention weight matrix, and the specific method is as follows:

[0031] Through linear mapping, we can get Query matrix Q , Key matrix K, Value matrix V ,Right now:

[0032] (5)

[0033] In formula (5), for the first 3D window focused linear multi-head self-attention module, represents the input features of the first 3D window focused linear multi-head self-attention module, that is, the features obtained after data normalization by the first normalization layer; for the second 3D window focused linear multi-head self-attention module, Represents the input features of the second 3D window focused linear multi-head self-attention module, that is, the features obtained after data normalization by the third normalization layer; express Query matrix Q The learnable parameters of express Key matrix K The learnable parameters of express Value matrix V The learnable parameters of , d express Query Features or Key The dimension of the feature;

[0034] Using Focus Function as the kernel function right Query matrix Q Perform nonlinear feature mapping to obtain the focused feature matrix ,right Key matrix K Perform nonlinear feature mapping to obtain the focused feature matrix ; Mathematical expression of FocusFunction for:

[0035] (6)

[0036] (7)

[0037] The input in (6) and (7) x refer to Query matrix Q or Key matrix K , ReLU represents the Rectified LinearUnit activation function; Express Level c Power operation;

[0038] Get the focused feature matrix and After that, calculate the Linear Attention self-attention weight matrix A , among which i Linear Attention self-attention weights The calculation formula is:

[0039] (8)

[0040] In formula (8), express Query matrix Q Middle i feature vectors, express Key matrix K Middle j feature vectors, express Value matrix V Middle j feature vectors, , Represents the focused feature matrix The focus feature in , Represents the focused feature matrix The focus feature; N Indicates the number of block feature vectors;

[0041] Using 3D depth-wise separable convolutional modules with time dimension Value matrix V Perform a three-dimensional convolution operation and then superimpose it on the Linear Attention self-attention weight matrix A , get the final Linear Attention self-attention weight matrix T :

[0042] (9)

[0043] Said and They represent the final Linear Attention self-attention weight matrices of the first 3D window focused linear multi-head self-attention module and the second 3D window focused linear multi-head self-attention module respectively.

[0044] Further, in step S4, the second spatial feature downsampling module downsamples the output of the second video linear attention network, the downsampling result is subjected to the first video Swin Transformer network for the first local working condition feature extraction, the extraction result is downsampled by the third spatial feature downsampling module, and the downsampling result is subjected to the second local working condition feature extraction by the second video Swin Transformer network; the method for extracting local working condition features by the first video Swin Transformer network and the second video Swin Transformer network is as follows:

[0045] feature The data is normalized by the fifth normalization layer and then passes through the 3D window multi-head self-attention module to output features. ,feature With features Sum and get the features ,feature Output features after the sixth normalization layer and the third multi-layer perceptron ,feature With features Sum and get the features ,feature After the seventh normalization layer and the 3D moving window multi-head self-attention module, the output features ,feature With features Sum and get the features ,feature After the eighth normalization layer and the fourth multi-layer perceptron, the output feature ,feature With features Sum and get the final output features of the Swin Transformer network of the first video ; The specific process expression for extracting local working condition features is:

[0046] (10)

[0047] (11)

[0048] (12)

[0049] (13)

[0050] For the first video Swin Transformer network, L = 5 in equations (10) to (13), and for the second video Swin Transformer network, L = 9 in equations (10) to (13).

[0051] ,feature ,feature ,feature .

[0052] Further, in step S4, the 3D window multi-head self-attention module and the 3D moving window multi-head self-attention module obtain a SoftMax self-attention weight matrix;

[0053] The specific method of obtaining the SoftMax self-attention matrix by the 3D window multi-head self-attention module is as follows:

[0054] The characteristics After inputting into the fifth normalization layer, the normalized features are obtained. , and then this feature Input into the 3D window multi-head self-attention module;

[0055] feature After entering the 3D window multi-head self-attention module, linear mapping is performed to obtain Query matrix , Key matrix and Value matrix ,Right now:

[0056] (14)

[0057] In formula (14), express Query matrix The learnable parameters of express Key matrix The learnable parameters of express Value matrix The learnable parameters of

[0058] get Query matrix , Key matrix and Value matrix After that, in the size of The SoftMax self-attention weight matrix is ​​calculated in the 3D window; the calculation expression of the SoftMax self-attention weight matrix is:

[0059] (15)

[0060] (16)

[0061] In formulas (15) and (16), represents the SoftMax self-attention weight matrix, D express Query matrix , Key matrix The characteristic dimension of B Indicates relative position deviation;

[0062] The method for the 3D moving window multi-head self-attention module to obtain the SoftMax self-attention weight matrix is:

[0063] get Query matrix , Key matrix and Valuematrix Then, move the 3D window forward. P / 2 distance, move right M / 2 round down the distance, move downward M / 2 The distance rounded down, that is, the overall movement of the 3D window The distance rounded down; in the moved 3D window, the SoftMax self-attention weight matrix is ​​calculated.

[0064] Compared with the prior art, the advantages of the present invention are as follows:

[0065] In order to solve the problem of unstable single-frame working condition feature information in the sintering process of the rotary kiln, the video Transformer method is used to model and analyze the flame video to improve the robustness and accuracy of the recognition model, and solve the problem of insufficient working condition features of single-frame flame video in the process of identifying abnormal working conditions of the rotary kiln. In order to solve the problem of high complexity in calculating the global working condition features of the flame video, the 3D window focused linear multi-head self-attention module is used to calculate the Linear Attention self-attention to obtain the global working condition features, and the 3D window multi-head self-attention module is used to calculate the SoftMax self-attention to obtain the local working condition features, which can effectively reduce the computational complexity of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a structural schematic diagram of a coal-burning condition recognition system based on flame video multi-scale attention according to an embodiment of the present invention.

[0067] Figure 2 yes Figure 1 Schematic diagram of the structure of the video linear attention network of the illustrated embodiment.

[0068] Figure 3 yes Figure 1 A schematic diagram of the structure of the video Swin Transformer network of the illustrated embodiment. DETAILED DESCRIPTION

[0069] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0070] Reference Figures 1 - 3 The coal-burning operating condition recognition system based on flame video multi-scale attention of this embodiment includes a flame video preprocessing module, an operating condition feature extraction module, and a classifier. The flame video preprocessing module is connected to the operating condition feature extraction module, and the operating condition feature extraction module is connected to the classifier.

[0071] The working condition feature extraction module includes a global working condition feature extraction network and a local working condition feature extraction network, and the global working condition feature extraction network is connected to the local working condition feature extraction network.

[0072] The global working condition feature extraction network realizes the extraction of global working condition features of flame video. The local working condition feature extraction network realizes the extraction of local working condition features of flame video.

[0073] The global working condition feature extraction network includes a flame video linear embedding network, a first video linear attention (VLA) network, a first spatial feature downsampling module, and a second video linear attention network. The flame video linear embedding network is connected to the first video linear attention network, the first video linear attention network is connected to the first spatial feature downsampling module, and the first spatial feature downsampling module is connected to the second video linear attention network.

[0074] The flame video preprocessing module preprocesses the flame video: extracts the flame video frame and inputs it into the Transformer network. The Patch Partition layer of the Transformer network divides the input flame video frame into blocks; the extracted flame video frame is ,in T Indicates the number of input frames, that is, the depth, each frame contains Pixels, H is the height of the video image frame, W is the width of the video image frame, 3 represents the three color channels of the video image frame, that is, the input flame video frame is , the Patch Partition layer of the Transformer network divides the input video frame into 3D blocks to obtain tokens, each of which is , with a total size of Non-overlapping tokens, each of which has a 96-dimensional feature vector, are used as the input of the subsequent working condition feature extraction module, and token is the unit of tokens.

[0075] To ensure that each dimension (T, H, W) is divisible by the corresponding value in the token, the Flame Video Linear Embedding Network first pads the length H, width W, and depth T of the input token, and then uses a 3D convolutional layer to map each token block to a C-dimensional space. The C-dimensional space is a vector space with C dimensions, where C is a positive integer. After linear embedding, tokens are represented by becomes .

[0076] The first video linear attention network is the same as the second video linear attention network, including a first normalization layer (LN), a first 3D window focus linear multi-head self-attention module (3D Focus Linear Attention Multi-headSelf-Attention: 3DLA3DFLA-MSA), a second normalization layer, a first multilayer perceptron (MultilayerPerceptron: MLP), a third normalization layer, a second 3D window focus linear multi-head self-attention module, a fourth normalization layer, and a second multilayer perceptron. The first normalization layer is connected to the first 3D window focus linear multi-head self-attention module, the first 3D window focus linear multi-head self-attention module is connected to the second normalization layer, the second normalization layer is connected to the first multilayer perceptron, the first multilayer perceptron is connected to the third normalization layer, the third normalization layer is connected to the second 3D window focus linear multi-head self-attention module, the second 3D window focus linear multi-head self-attention module is connected to the fourth normalization layer, and the fourth normalization layer is connected to the second multilayer perceptron.

[0077] The first 3D window focused linear multi-head self-attention module is the same as the second 3D window focused linear multi-head self-attention module, which is used to calculate the self-attention with focused Linear Attention in a 3D window of a specified size to extract the global working condition features of the flame video.

[0078] Input Features After the first normalization layer and the first 3D window focused linear multi-head self-attention module, the output features ,feature With features Sum and get the features ,feature Output features after the second normalization layer and the first multi-layer perceptron ,feature With features Sum and get the features ,feature After the third normalization layer and the second 3D window focused linear multi-head self-attention module, the output features ,feature With features Sum and get the features ,feature After the fourth normalization layer and the second multi-layer perceptron, the output feature ,feature With features Sum and get the features ,The specific process expression of global working condition feature extraction of flame video is:

[0079] (1)

[0080] (2)

[0081] (3)

[0082] (4)

[0083] For the first video linear attention network, L = 1 in equations (1) to (4), and for the second video linear attention network, L = 3 in equations (1) to (4). ,feature ,feature ,feature .

[0084] In order to extract the global working condition features, the Linear Attention self-attention weight matrix is ​​obtained by using the first 3D window focused linear multi-head self-attention module and the second 3D window focused linear multi-head self-attention module. The 3D window size is specified as 8×56×56 in the first video linear attention network, and the 3D window size is specified as 8×28×28 in the second video linear attention network. The specific operations of the first 3D window focused linear multi-head self-attention module and the second 3D window focused linear multi-head self-attention module are as follows:

[0085] Through linear mapping, we can get Query matrix Q , Key matrix K, Value matrix V ,Right now:

[0086] (5)

[0087] In formula (5), for the first 3D window focused linear multi-head self-attention module, Represents the input features of the first 3D window focused linear multi-head self-attention module, that is, the features obtained after data normalization by the first normalization layer, as the starting input of the first 3D window focused linear multi-head self-attention module , , N Indicates the number of tokens feature vectors, C represents the feature dimension, Represents the matrix size and shape; for the second 3D window focused linear multi-head self-attention module, Represents the input features of the second 3D window focused linear multi-head self-attention module, that is, the features obtained after data normalization by the third normalization layer; expressQuery matrix Q The learnable parameters of express Key matrix K The learnable parameters of express Value matrix V The learnable parameters of , d express Query Features or Key The dimension of the feature.

[0088] In order to improve the prominence of similar features, Focus Function is used as the kernel function right Query matrix Q Perform nonlinear feature mapping to obtain the focused feature matrix ,right Key matrix K Perform nonlinear feature mapping to obtain the focused feature matrix ; Mathematical expression of Focus Function for:

[0089] (6)

[0090] (7)

[0091] The input in (6) and (7) x refer to Query matrix Q or Key matrix K , ReLU represents the Rectified LinearUnit activation function; Express Level c Power operation.

[0092] In getting the focused feature matrix and Then calculate the Linear Attention self-attention weight matrix A , among which i Linear Attention self-attention weights The calculation formula is:

[0093] (8)

[0094] In formula (8), express Query matrix Q Middlei feature vectors, express Key matrix K Middle j feature vectors, express Value matrix V Middle j feature vectors, , Represents the focused feature matrix The focus feature in , Represents the focused feature matrix The focus feature; N Indicates the number of tokens feature vectors.

[0095] Since the Linear Attention self-attention weight matrix is ​​obtained through Linear Attention self-attention calculation A There is a low-rank problem, which leads to low feature diversity and limits the representation ability of the model. In order to improve feature diversity, the present invention uses a 3D depth-wise convolution module with a time dimension to Value matrix V Perform a three-dimensional convolution operation and then superimpose it on the Linear Attention self-attention weight matrix A , get the final Linear Attention self-attention weight matrix T The mathematical expression is:

[0096] (9)

[0097] In the above and They represent the final Linear Attention self-attention weight matrices of the first 3D window focused linear multi-head self-attention module and the second 3D window focused linear multi-head self-attention module respectively.

[0098] The first spatial feature downsampling module achieves downsampling by splicing adjacent patches, reducing resolution, adjusting the number of channels, and saving computational effort while maintaining information integrity, and finally reducing the number of channels through a linear layer. Each downsampling is doubled, that is, elements are selected at intervals of 2 in the row and column directions to form a new patch, and then all patches are connected as a whole tensor. At this time, the channel dimension will become 4 times the original, and finally a fully connected layer is used to adjust the channel dimension to 2 times the original.

[0099] The local working condition feature extraction network includes a second spatial feature downsampling module, a first video Swin Transformer (Video Swin Transformer: VST) network, a third spatial feature downsampling module, and a second video Swin Transformer (Video Swin Transformer: VST) network. The second spatial feature downsampling module is connected to the first video Swin Transformer network, the first video Swin Transformer network is connected to the third spatial feature downsampling module, and the third spatial feature downsampling module is connected to the second video Swin Transformer network.

[0100] The Swin Transformer network of the first video is the same as the Swin Transformer network of the second video, including the fifth normalization layer (LN), a 3D window multi-head self-attention module (3DW-MSA), a sixth normalization layer, a third multilayer perceptron (Multilayer Perceptron: MLP), a seventh normalization layer, a 3D moving window multi-head self-attention module (3DSW-MSA), an eighth normalization layer, and a fourth multilayer perceptron. The fifth normalization layer is connected to the 3D window multi-head self-attention module, the 3D window multi-head self-attention module is connected to the sixth normalization layer, the sixth normalization layer is connected to the third multilayer perceptron, the third multilayer perceptron is connected to the seventh normalization layer, the seventh normalization layer is connected to the 3D moving window multi-head self-attention module, the 3D moving window multi-head self-attention module is connected to the eighth normalization layer, and the eighth normalization layer is connected to the fourth multilayer perceptron.

[0101] The input of the first video Swin Transformer network is the output feature of the second spatial feature downsampling module .feature The data is normalized by the fifth normalization layer and then passes through the 3D window multi-head self-attention module to output features. ,feature With features Sum and get the features ,feature Output features after the sixth normalization layer and the third multi-layer perceptron ,feature With features Sum and get the features ,feature After the seventh normalization layer and the 3D moving window multi-head self-attention module, the output features ,feature With features Sum and get the features ,feature After the eighth normalization layer and the fourth multi-layer perceptron, the output feature ,feature With features Sum and get the final output features of the Swin Transformer network of the first video The specific process expression of local working condition feature extraction is:

[0102] (10)

[0103] (11)

[0104] (12)

[0105] (13)

[0106] For the first video Swin Transformer network, L = 5 in equations (10) to (13), and for the second video Swin Transformer network, L = 9 in equations (10) to (13). ,feature ,feature ,feature .

[0107] In order to extract local working condition features, the 3D window multi-head self-attention module and the 3D moving window multi-head self-attention module are used to obtain the SoftMax self-attention weight matrix. The 3D window multi-head self-attention module is used to calculate the SoftMax self-attention in a 3D window of a specified size, thereby extracting local working condition features. The 3D moving window multi-head self-attention module is used to calculate the SoftMax self-attention by moving the 3D window in a 3D window of a specified size, thereby extracting local working condition features.

[0108] The 3D window size specified in the 3D window multi-head self-attention module and the 3D moving window multi-head self-attention module is (length × width × height). Compared with the 3D window multi-head self-attention module, the 3D moving window multi-head self-attention module requires one more step, namely the movement of the 3D window. The specific process of obtaining the SoftMax self-attention matrix through the 3D window multi-head self-attention module is as follows:

[0109] The characteristics After inputting into the fifth normalization layer, the normalized features are obtained. , and then this feature Input into the 3D window multi-head self-attention module;

[0110] feature After entering the 3D window multi-head self-attention module, linear mapping is performed to obtain Query matrix , Key matrix and Value matrix ,Right now:

[0111] (14)

[0112] In formula (14), express Query matrix The learnable parameters of express Key matrix The learnable parameters of express Value matrix The learnable parameters of .

[0113] get Query matrix , Key matrix and Value matrix After that, in the size of The SoftMax self-attention weight matrix is ​​calculated in the 3D window. The calculation expression of the SoftMax self-attention weight matrix is:

[0114] (15)

[0115] (16)

[0116] In formulas (15) and (16), represents the SoftMax self-attention weight matrix, D express Query matrix , Key matrix The characteristic dimension of B Indicates relative position deviation.

[0117] For the 3D moving window multi-head self-attention module, the specific process of obtaining the SoftMax self-attention matrix is ​​different from that of the 3D window multi-head self-attention module only in that:

[0118] get Query matrix , Key matrix and Value matrix After that, before calculating the SoftMax self-attention weight matrix, the 3D window is moved forward (in the direction of time and sequence frame). P / 2 distance, move to the right (in the right spatial direction) M / 2 The distance rounded down, moving downward (in the downward spatial direction) M / 2 The distance rounded down, that is, the overall movement of the 3D window The rest of the process is the same as the 3D window multi-head self-attention module to obtain the SoftMax self-attention matrix.

[0119] Assume that the input tokens size is , set the window size , then in the 3D window multi-head self-attention module, Then, in the 3D moving window multi-head self-attention module, the window moves to a position where the window is half the size, forming The window moves with the window, successfully solving the problem of the previous layer of tokens being blocked from interacting with each other due to 3D non-overlapping windows, and enhancing the spatiotemporal information exchange. The window calculates the SoftMax self-attention weight matrix within the window.

[0120] The second spatial feature downsampling module and the third spatial feature downsampling module are both designed to reduce the computational complexity of the model, improve the representation capability of the model's hierarchical features, and enrich the semantic features.

[0121] The classifier includes a fully connected layer and a Softmax activation function. There is a connection between each input node and output node of the fully connected layer, which is used to map features to the category space. Suppose there is an input vector x and an output vector y , the calculation process of the fully connected layer can be expressed as:

[0122] (17)

[0123] in, is the weight matrix, which contains the weights of all connections, b is the bias vector, which is used to adjust the linear transformation of the output, and the input vector x The output of the second video Swin Transformer network, the output vector y is the sample category score vector, and the Softmax activation function is used to generate a probability distribution so that the probability value of each sample category is between 0 and 1. The sample categories are divided into normal, overburned, and underburned.

[0124] For the sample category score vector , the Softmax function converts the sample category score vector into the sample category probability distribution matrix , calculated as follows:

[0125] (18)

[0126] In the formula, .

[0127] The coal burning condition recognition method based on flame video multi-scale attention in this embodiment includes the following steps:

[0128] Step S1: flame video preprocessing: extract flame video frames and input them into the Transformer network. The Patch Partition layer of the Transformer network divides the input flame video frames into blocks; the extracted flame video frames are ,in T Indicates the number of input frames, that is, the depth, each frame contains Pixels, H is the height of the video image frame, W is the width of the video image frame, 3 represents the three color channels of the video image frame, that is, the input flame video frame is , the Patch Partition layer of the Transformer network divides the input video frame into 3D blocks to obtain tokens, each of which is , with a total size of Non-overlapping tokens, each of which has a 96-dimensional feature vector, serve as the input of the subsequent working condition feature extraction module.

[0129] Step S2: The flame video linear embedding network fills the blocks of the video frame and uses a 3D convolutional layer to map each block into a C-dimensional space;

[0130] To ensure that each dimension (T, H, W) is divisible by the corresponding value in the token, the Flame Video Linear Embedding Network first pads the length H, width W, and depth T of the input token, and then uses a 3D convolutional layer to map each token block to a C-dimensional space. The C-dimensional space is a vector space with C dimensions, where C is a positive integer. After linear embedding, tokens are represented by becomes .

[0131] Step S3: Input each mapped block into the global working condition feature extraction network to extract the global working condition features of the flame video; the specific method is as follows:

[0132] The first video linear attention network performs the first global working condition feature extraction on the output of the flame video linear embedding network, and the extraction result is downsampled by the first spatial feature downsampling module, and then the second video linear attention network performs the second global working condition feature extraction. The method for extracting global working condition features by the first video linear attention network and the second video linear attention network is as follows:

[0133] Input Features After the first normalization layer and the first 3D window focused linear multi-head self-attention module, the output features ,feature With features Sum and get the features ,feature Output features after the second normalization layer and the first multi-layer perceptron ,feature With features Sum and get the features ,feature After the third normalization layer and the second 3D window focused linear multi-head self-attention module, the output features ,feature With features Sum and get the features ,feature After the fourth normalization layer and the second multi-layer perceptron, the output feature ,feature With features Sum and get the features , the specific expression is:

[0134] (1)

[0135] (2)

[0136] (3)

[0137] (4)

[0138] For the first video linear attention network, L = 1 in equations (1) to (4), and for the second video linear attention network, L = 3 in equations (1) to (4). ,feature ,feature ,feature .

[0139] The first 3D window focused linear multi-head self-attention module and the second 3D window focused linear multi-head self-attention module obtain the Linear Attention self-attention weight matrix as follows:

[0140] Through linear mapping, we can get Query matrix Q , Key matrix K, Value matrix V ,Right now:

[0141] (5)

[0142] In formula (5), for the first 3D window focused linear multi-head self-attention module, Represents the input features of the first 3D window focused linear multi-head self-attention module, that is, the features obtained after data normalization by the first normalization layer, as the starting input of the first 3D window focused linear multi-head self-attention module , , N Indicates the number of tokens feature vectors, C represents the feature dimension, Represents the matrix size and shape; for the second 3D window focused linear multi-head self-attention module, Represents the input features of the second 3D window focused linear multi-head self-attention module, that is, the features obtained after data normalization by the third normalization layer.

[0143] express Query matrix Q The learnable parameters of express Key matrix K The learnable parameters of express Value matrix V The learnable parameters of , d express Query Features or Key The dimension of the feature.

[0144] In order to improve the prominence of similar features, Focus Function is used as the kernel function right Query matrix Q Perform nonlinear feature mapping to obtain the focused feature matrix ,right Key matrix K Perform nonlinear feature mapping to obtain the focused feature matrix ; Mathematical expression of Focus Function for:

[0145] (6)

[0146] (7)

[0147] The input in (6) and (7) x refer to Query matrix Q or Key matrix K , ReLU represents the Rectified LinearUnit activation function; Express Level c Power operation.

[0148] In getting the focused feature matrix and Then calculate the Linear Attention self-attention weight matrix A , among which i Linear Attention self-attention weights The calculation formula is:

[0149] (8)

[0150] In formula (8), express Query matrix Q Middle i feature vectors, express Key matrix K Middle j feature vectors, express Value matrix V Middle j feature vectors, , Represents the focused feature matrix The focus feature in , Represents the focused feature matrix The focus feature; N Indicates the number of tokens feature vectors.

[0151] Since the Linear Attention self-attention weight matrix is ​​obtained through Linear Attention self-attention calculation A There is a low-rank problem, which leads to low feature diversity and limits the representation ability of the model. In order to improve feature diversity, a 3D depth-wise convolution module with a time dimension is used to Value matrixV Perform a three-dimensional convolution operation and then superimpose it on the Linear Attention self-attention weight matrix A , get the final LinearAttention self-attention weight matrix T The mathematical expression is:

[0152] (9)

[0153] Said and They represent the final Linear Attention self-attention weight matrices of the first 3D window focused linear multi-head self-attention module and the second 3D window focused linear multi-head self-attention module respectively.

[0154] Step S4: input the global working condition features into the local working condition feature extraction network to extract the local working condition features of the flame video; the specific method is as follows:

[0155] The second spatial feature downsampling module downsamples the output of the second video linear attention network, and the downsampling result is subjected to the first video Swin Transformer network for the first local working condition feature extraction. The extraction result is downsampled by the third spatial feature downsampling module, and the downsampling result is subjected to the second local working condition feature extraction by the second video Swin Transformer network. The method for extracting local working condition features by the first video Swin Transformer network and the second video Swin Transformer network is as follows:

[0156] feature The data is normalized by the fifth normalization layer and then passes through the 3D window multi-head self-attention module to output features. ,feature With features Sum and get the features ,feature Output features after the sixth normalization layer and the third multi-layer perceptron ,feature With features Sum and get the features ,feature After the seventh normalization layer and the 3D moving window multi-head self-attention module, the output features ,feature With features Sum and get the features ,feature After the eighth normalization layer and the fourth multi-layer perceptron, the output feature ,feature With features Sum and get the final output features of the Swin Transformer network of the first video ; The specific process expression for extracting local working condition features is:

[0157] (10)

[0158] (11)

[0159] (12)

[0160] (13)

[0161] For the first video Swin Transformer network, L = 5 in equations (10) to (13), and for the second video Swin Transformer network, L = 9 in equations (10) to (13).

[0162] ,feature ,feature ,feature .

[0163] The 3D window multi-head self-attention module and the 3D moving window multi-head self-attention module obtain the SoftMax self-attention weight matrix.

[0164] The specific method of obtaining the SoftMax self-attention matrix by the 3D window multi-head self-attention module is as follows:

[0165] The characteristics After inputting into the fifth normalization layer, the normalized features are obtained. , and then this feature Input into the 3D window multi-head self-attention module;

[0166] feature After entering the 3D window multi-head self-attention module, linear mapping is performed to obtain Query matrix , Key matrix and Value matrix ,Right now:

[0167] (14)

[0168] In formula (14), express Query matrix The learnable parameters of express Keymatrix The learnable parameters of express Value matrix The learnable parameters of .

[0169] get Query matrix , Key matrix and Value matrix After that, in the size of The SoftMax self-attention weight matrix is ​​calculated in the 3D window; the calculation expression of the SoftMax self-attention weight matrix is:

[0170] (15)

[0171] (16)

[0172] In formulas (15) and (16), represents the SoftMax self-attention weight matrix, D express Query matrix , Key matrix The characteristic dimension of B Indicates relative position deviation.

[0173] The method for the 3D moving window multi-head self-attention module to obtain the SoftMax self-attention weight matrix is:

[0174] get Query matrix , Key matrix and Value matrix Then, the 3D window is moved forward (in the direction of time and sequence frames). P / 2 distance, move to the right (in the right spatial direction) M / 2 The distance rounded down, moving downward (in the downward spatial direction) M / 2 The distance rounded down, that is, the overall movement of the 3D window The distance rounded down; in the moved 3D window, the SoftMax self-attention weight matrix is ​​calculated.

[0175] Step S5: The classifier classifies the working conditions according to the local working condition characteristics of the flame video to obtain a classification result.

[0176] The classifier includes a fully connected layer and a Softmax activation function. There is a connection between each input node and output node of the fully connected layer, which is used to map features to the category space. Suppose there is an input vector x and an output vector y , the calculation process of the fully connected layer can be expressed as:

[0177] (17)

[0178] in, is the weight matrix, which contains the weights of all connections, b is the bias vector, which is used to adjust the linear transformation of the output, and the input vector x The output of the second video Swin Transformer network, the output vector y is the sample category score vector, and the Softmax activation function is used to generate a probability distribution so that the probability value of each sample category is between 0 and 1. The sample categories are divided into normal, overburned, and underburned.

[0179] For the sample category score vector , the Softmax function converts the sample category score vector into the sample category probability distribution matrix , calculated as follows: (18)

[0180] In the formula, .

[0181] In this embodiment, a CCD camera is used to record the flame combustion condition in the sintering zone at the kiln head of the rotary kiln of Zhongzhou Aluminum Plant in China, and a flame video with a frame rate of 25 is collected. A 10-second flame video is taken as a sample, and each sample has Frame images, before the video samples are input into the model, 32 frames of non-overlapping RGB images are extracted from them for preprocessing as samples for the input model. The experimental samples are divided into three categories: normal, overburned, and underburned. There are 599 normal samples, 137 overburned samples, and 547 underburned samples, for a total of 1283 samples. 80% of the data is used for training and 20% of the data is used for testing.

[0182] The coal-fired operating condition recognition method based on flame video multi-scale attention of the present invention can effectively extract the operating condition features in the flame video, and the recognition rate of abnormal operating conditions such as overburning and underburning of the rotary kiln can reach more than 90%. Compared with the operating condition recognition method using a single-frame flame image and the existing video-based operating condition recognition method, the performance is greatly improved. Table 1 shows the prediction accuracy experimental results of the method of the present invention for abnormal operating condition recognition of coal-fired rotary kilns.

[0183]

[0184] In Table 1, C3D refers to Convolutional 3D, that is, the three-dimensional convolution method, SlowFast-R101 refers to the method of combining the SlowFast model with the ResNet-101 network, and Video Swin Transformer is a Transformer model specially designed for video understanding tasks.

[0185] Those skilled in the art may make various modifications and variations to the present invention. If these modifications and variations are within the scope of the claims of the present invention and their equivalents, then these modifications and variations are also within the protection scope of the present invention.

[0186] The contents not described in detail in the specification are prior art known to those skilled in the art.

Claims

1. A coal-burning condition recognition system based on flame video multi-scale attention, characterized in that: include: A flame video preprocessing module, a working condition feature extraction module, and a classifier, wherein the flame video preprocessing module is connected to the working condition feature extraction module, and the working condition feature extraction module is connected to the classifier; The working condition feature extraction module includes a global working condition feature extraction network and a local working condition feature extraction network, and the global working condition feature extraction network is connected to the local working condition feature extraction network; The flame video preprocessing module preprocesses the flame video; The global working condition feature extraction network realizes the extraction of global working condition features of the flame video; the local working condition feature extraction network realizes the extraction of local working condition features of the flame video; The classifier classifies the working conditions according to the local working condition characteristics of the flame video and obtains the classification results; The global working condition feature extraction network includes a flame video linear embedding network, a first video linear attention network, a first spatial feature downsampling module, and a second video linear attention network. The flame video linear embedding network is connected to the first video linear attention network, the first video linear attention network is connected to the first spatial feature downsampling module, and the first spatial feature downsampling module is connected to the second video linear attention network. The local working condition feature extraction network includes a second spatial feature downsampling module, a first video Swin Transformer network, a third spatial feature downsampling module, and a second video Swin Transformer network. The second spatial feature downsampling module is connected to the first video Swin Transformer network, the first video Swin Transformer network is connected to the third spatial feature downsampling module, and the third spatial feature downsampling module is connected to the second video Swin Transformer network.

2. The coal burning condition recognition system based on flame video multi-scale attention according to claim 1, characterized in that: The first video linear attention network is the same as the second video linear attention network, including a first normalization layer, a first 3D window focused linear multi-head self-attention module, a second normalization layer, a first multi-layer perceptron, a third normalization layer, a second 3D window focused linear multi-head self-attention module, a fourth normalization layer, and a second multi-layer perceptron. The first normalization layer is connected to the first 3D window focused linear multi-head self-attention module, the first 3D window focused linear multi-head self-attention module is connected to the second normalization layer, the second normalization layer is connected to the first multi-layer perceptron, the first multi-layer perceptron is connected to the third normalization layer, the third normalization layer is connected to the second 3D window focused linear multi-head self-attention module, the second 3D window focused linear multi-head self-attention module is connected to the fourth normalization layer, and the fourth normalization layer is connected to the second multi-layer perceptron. The first 3D window focused linear multi-head self-attention module is the same as the second 3D window focused linear multi-head self-attention module, and is used to calculate self-attention with focused Linear Attention in a 3D window of a specified size to extract global working condition features of the flame video.

3. The coal burning condition recognition system based on flame video multi-scale attention according to claim 1, characterized in that: The first video Swin Transformer network is the same as the second video Swin Transformer network, including a fifth normalization layer, a 3D window multi-head self-attention module, a sixth normalization layer, a third multi-layer perceptron, a seventh normalization layer, a 3D moving window multi-head self-attention module, an eighth normalization layer, and a fourth multi-layer perceptron. The fifth normalization layer is connected to the 3D window multi-head self-attention module, the 3D window multi-head self-attention module is connected to the sixth normalization layer, the sixth normalization layer is connected to the third multi-layer perceptron, the third multi-layer perceptron is connected to the seventh normalization layer, and the seventh normalization layer is connected to the 3D moving window multi-head self-attention module. The 3D moving window multi-head self-attention module is connected to the eighth normalization layer, and the eighth normalization layer is connected to the fourth multi-layer perceptron; the 3D window multi-head self-attention module is used to calculate the SoftMax self-attention in a 3D window of a specified size, and then extract local working condition characteristics; the 3D moving window multi-head self-attention module is used to calculate the SoftMax self-attention by moving the 3D window in a 3D window of a specified size, and then extract local working condition characteristics.

4. A method for identifying coal-burning conditions based on flame video multi-scale attention, characterized in that: The coal burning condition recognition system based on flame video multi-scale attention as described in any one of claims 1 to 3 comprises the following steps: Step S1: flame video preprocessing: extract flame video frames and input them into the Transformer network. The Patch Partition layer of the Transformer network divides the input flame video frames into blocks. Step S2: The flame video linear embedding network fills the blocks of the video frame and uses a 3D convolutional layer to map each block into a C-dimensional space; Step S3: input each mapped block into a global working condition feature extraction network to extract the global working condition features of the flame video sample; Step S4: inputting the global working condition features into the local working condition feature extraction network to extract the local working condition features of the flame video sample; Step S5: The classifier classifies the working conditions according to the local working condition characteristics of the flame video to obtain a classification result.

5. The method for identifying coal-burning conditions based on flame video multi-scale attention according to claim 4, characterized in that: In step S1, the flame video frame extracted is ,in T Indicates the number of input frames, that is, the depth, each frame contains Pixels, H is the height of the video image frame, W is the width of the video image frame, 3 represents the three color channels of the video image frame, that is, the input flame video frame is ,The Patch Partition layer of the Transformer network performs 3D block partitioning on the input video frame to obtain tokens.

6. The method for identifying coal-burning conditions based on flame video multi-scale attention according to claim 5, characterized in that: In step S2, the flame video linear embedding network first pads the length H, width W, and depth T of the input token, and then uses a 3D convolutional layer to map each token block into a C-dimensional space. The C-dimensional space is a vector space with C dimensions, where C is a positive integer.

7. The method for identifying coal-burning conditions based on flame video multi-scale attention according to claim 4, characterized in that: In step S3, the first video linear attention network performs a first global working condition feature extraction on the output of the flame video linear embedding network, the extraction result is downsampled by the first spatial feature downsampling module, and then the second video linear attention network performs a second global working condition feature extraction; the method for extracting global working condition features by the first video linear attention network and the second video linear attention network is as follows: Input Features After the first normalization layer and the first 3D window focused linear multi-head self-attention module, the output features ,feature With features Sum and get the features ,feature Output features after the second normalization layer and the first multi-layer perceptron ,feature With features Sum and get the features ,feature After the third normalization layer and the second 3D window focused linear multi-head self-attention module, the output features ,feature With features Sum and get the features ,feature After the fourth normalization layer and the second multi-layer perceptron, the output feature ,feature With features Sum and get the features , the specific expression is: (1) (2) (3) (4) For the first video linear attention network, L = 1 in equations (1) to (4), and for the second video linear attention network, L = 3 in equations (1) to (4). ,feature ,feature ,feature .

8. The method for identifying coal-burning conditions based on flame video multi-scale attention according to claim 7, characterized in that: In step S3, the first 3D window focused linear multi-head self-attention module and the second 3D window focused linear multi-head self-attention module obtain the Linear Attention self-attention weight matrix, and the specific method is as follows: Through linear mapping, we can get Query matrix Q , Key matrix K,Value matrix V ,Right now: (5) In formula (5), for the first 3D window focused linear multi-head self-attention module, represents the input features of the first 3D window focused linear multi-head self-attention module, that is, the features obtained after data normalization by the first normalization layer; for the second 3D window focused linear multi-head self-attention module, Represents the input features of the second 3D window focused linear multi-head self-attention module, that is, the features obtained after data normalization by the third normalization layer; express Query matrix Q The learnable parameters of express Key matrix K The learnable parameters of express Value matrix V The learnable parameters of , d express Query Features or Key The dimension of the feature; Using Focus Function as the kernel function right Query matrix Q Perform nonlinear feature mapping to obtain the focused feature matrix ,right Key matrix K Perform nonlinear feature mapping to obtain the focused feature matrix ; Mathematical expression of Focus Function for: (6) (7) The input in (6) and (7) x refer to Query matrix Q or Key matrix K , ReLU represents the Rectified Linear Unit activation function; Express Level c Power operation; Get the focused feature matrix and After that, calculate the Linear Attention self-attention weight matrix A , among which i Linear Attention self-attention weights The calculation formula is: (8) In formula (8), express Query matrix Q Middle i feature vectors, express Key matrix K Middle j feature vectors, express Value matrix V Middle j feature vectors, , Represents the focused feature matrix The focus feature in , Represents the focused feature matrix The focus feature; N Indicates the number of block feature vectors; Using 3D depth-wise separable convolutional modules with time dimension Value matrix V Perform a three-dimensional convolution operation and then superimpose it on the Linear Attention self-attention weight matrix A , get the final Linear Attention self-attention weight matrix T : (9) Said and They represent the final Linear Attention self-attention weight matrices of the first 3D window focused linear multi-head self-attention module and the second 3D window focused linear multi-head self-attention module respectively.

9. The method for identifying coal-burning conditions based on flame video multi-scale attention according to claim 4, characterized in that: In step S4, the second spatial feature downsampling module downsamples the output of the second video linear attention network, and the downsampling result is subjected to the first video Swin Transformer network for the first local working condition feature extraction, and the extraction result is downsampled by the third spatial feature downsampling module, and the downsampling result is subjected to the second local working condition feature extraction by the second video Swin Transformer network; the method for extracting local working condition features by the first video Swin Transformer network and the second video Swin Transformer network is as follows: feature The data is normalized by the fifth normalization layer and then passes through the 3D window multi-head self-attention module to output features. ,feature With features Sum and get the features ,feature Output features after the sixth normalization layer and the third multi-layer perceptron ,feature With features Sum and get the features ,feature After the seventh normalization layer and the 3D moving window multi-head self-attention module, the output features ,feature With features Sum and get the features ,feature After the eighth normalization layer and the fourth multi-layer perceptron, the output feature ,feature With features Sum and get the final output features of the Swin Transformer network of the first video ; The specific process expression for extracting local working condition features is: (10) (11) (12) (13) For the first video Swin Transformer network, L = 5 in equations (10) to (13), and for the second video Swin Transformer network, L = 9 in equations (10) to (13). ,feature ,feature ,feature .

10. The method for identifying coal-burning conditions based on flame video multi-scale attention according to claim 9, characterized in that: In step S4, the 3D window multi-head self-attention module and the 3D moving window multi-head self-attention module obtain the SoftMax self-attention weight matrix; The specific method of obtaining the SoftMax self-attention matrix by the 3D window multi-head self-attention module is as follows: The characteristics After inputting into the fifth normalization layer, the normalized features are obtained. , and then this feature Input into the 3D window multi-head self-attention module; feature After entering the 3D window multi-head self-attention module, linear mapping is performed to obtain Query matrix , Key matrix and Value matrix ,Right now: (14) In formula (14), express Query matrix The learnable parameters of express Key matrix The learnable parameters of express Value matrix The learnable parameters of get Query matrix , Key matrix and Value matrix After that, in the size of The SoftMax self-attention weight matrix is ​​calculated in the 3D window; the calculation expression of the SoftMax self-attention weight matrix is: (15) (16) In formulas (15) and (16), represents the SoftMax self-attention weight matrix, D express Query matrix , Key matrix The characteristic dimension of B Indicates relative position deviation; The method for the 3D moving window multi-head self-attention module to obtain the SoftMax self-attention weight matrix is: get Query matrix , Key matrix and Value matrix Then, move the 3D window forward. P / 2 distance, move right M / 2 round down the distance, move downward M / 2 The distance rounded down, that is, the overall movement of the 3D window The distance rounded down; in the moved 3D window, the SoftMax self-attention weight matrix is ​​calculated.

Images