Floatation froth stability estimation method based on infrared time-series salient object segmentation

Abstract

The application provides a flotation froth stability estimation method based on infrared time sequence significant target segmentation. First, a U-shaped network based on a ConvNeXt network design is used, a ConvLSTM network is embedded in the encoder to realize time sequence infrared saliency information extraction, and a cross attention mechanism is added to the ConvNeXt network to realize initial positioning of the infrared saliency area. Second, a residual refinement network with a U-shaped encoder-decoder structure is constructed, the residual between the deep learning saliency map and the true value is learned to improve the edge details of the saliency area, and fine segmentation of the time sequence significant target is realized. Finally, according to the significant target segmentation result, the froth stability is calculated, the deviation and abnormal threshold of the froth stability in the time sequence under different working conditions are counted, the significant area of the froth infrared video image is detected to realize the segmentation of the merged and broken bubbles, and the froth stability is evaluated according to the segmentation result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of foam image processing, in particular to a flotation foam stability estimation method based on infrared time sequence salient target segmentation. BACKGROUND

[0002] The generation and collapse of bubbles in the flotation process can effectively reflect the mineral content. Through image segmentation technology, newly generated bubbles and collapsed bubbles in the flotation process can be detected. This technology has wide application prospects in the fields of computer vision and industrial production. Foam stability is defined as the ability of bubbles in the foam to resist merging and collapse, which is a key driving factor of flotation performance and helps to determine the grade and recovery rate of the flotation process.

[0003] In recent years, there are many methods for detecting unstable bubbles. Based on image analysis and neural networks, according to the difference between the shadows formed during the generation and dissipation of bubbles, and combined with the speed information of bubble movement, the collapse of the foam is distinguished to evaluate the stability. However, the detection accuracy of this method is affected by bubble deformation and light, causing misjudgment of the bright spots or high-light edges of the bubbles; using the watershed algorithm, the deformation law of flotation foam under different process parameters is studied to segment the size of the bubbles, but this method is also affected by light, and over-segmentation or under-segmentation may occur when merging or collapsing bubbles occur; flotation condition recognition based on a dual-mode convolutional neural network, researchers studied the infrared thermal imaging of foam and found that bubbles release heat when they collapse or merge, and the temperature is higher than that of the surrounding bubbles, which presents a high-light salient region after thermal imaging, providing an intuitive display effect for unstable bubbles.

[0004] Due to the stirring action of the flotation machine and the extrusion condition of the mineralized bubbles, the flotation bubbles inevitably undergo geometric shape changes such as angular rotation and size scaling during the flow (flowing towards the scraper). Bubble merging, collapse, and fragmentation occur, accompanied by the generation of new bubbles. Therefore, it is difficult to use conventional image processing methods to segment the bubbles in the foam image, and it is also difficult to track and identify independent bubbles with severe deformation and high collapse and fragmentation rates, making it impossible to accurately extract and quantify the features of the flotation foam image. SUMMARY

[0005] Therefore, the purpose of the present application is to provide a flotation foam stability estimation method based on infrared time sequence salient target segmentation, which realizes the segmentation of merging and collapsing bubbles by detecting the salient region of the foam infrared video image, and evaluates the stability of the foam according to the segmentation results.

[0006] To achieve the above purpose, the present application adopts the following technical solution: a flotation foam stability estimation method based on infrared time sequence salient target segmentation, comprising the following steps:

[0007] Step 1: The infrared foam video collected by the thermal infrared imager is extracted as images by frame number and divided into training set and test set, and the saliency region is manually segmented using Lableme;

[0008] Step 2: The image of the test set is used as input to train the network model, and the ConvNeXt network with added double cross attention module and ConvLSTM is used to realize the initial positioning of the time series infrared saliency region;

[0009] Step 3: A residual refinement network with U-shaped encoder-decoder structure is constructed to deep learn the residual between the saliency map and the true value to improve the edge details of the saliency region and realize fine segmentation of the time series saliency target;

[0010] Step 4: According to the saliency target segmentation result, the foam stability is calculated, and the deviation degree r of the foam stability in the time series under different working conditions is calculated. i and the abnormal threshold T.

[0011] In a preferred embodiment, in step 2, a U-shaped network based on ConvNeXt network design is used, and a ConvLSTM network is embedded in the encoder to realize time series infrared saliency information extraction, and a cross attention mechanism is added to the ConvNeXt network to realize initial positioning of the infrared saliency region.

[0012] In a preferred embodiment, in step 2, a “decoder” part is first added to the original ConvNeXt network, and the original structure is used as the “encoder” part; the redesigned ConvNeXt network satisfies the structure of “encoder-decoder”; in the “decoder” part, each upsampling block contains a 3x3 convolution, a BN layer, a GELU layer and a 1x1 convolution;

[0013] A double cross attention module DCA is used to extract the mutual dependence of the channel dimension and the spatial dimension of the multi-scale features in the “encoder”; the double cross attention module DCA takes the multi-scale features of the first three stages as input, generates enhanced representations, and connects them to the corresponding three decoder stages; the double cross attention module DCA module is divided into a first stage and a second stage; the first stage contains a multi-scale Patch Embedding for obtaining “encoder” tokens; in the second stage, the “encoder” tokens are processed using the DCA mechanism, and the channel cross attention module CCA and the spatial cross attention module SCA are used to capture the relationship between different scale features; finally, LayerNorm and GeLU are applied, and these tokens are upsampled to connect them to the corresponding “decoder” part;

[0014] The channel cross-attention module CCA concatenates the tokens T i along the channel dimension to create keys and values T c Simultaneously, T c is transformed using a 1 x 1 depth convolution as queries;

[0015]

[0016] K = DConv1D K (T c ) (2)

[0017] V = DConv1D V (T c ) (3)

[0018]

[0019] where Q i , K, V represent the queries, keys, values matrices, respectively, is a scaling factor, and DConv1D x () represents a 1 x 1 depth convolution.

[0020] The SCA module concatenates the output from the CCA module along the channel dimension to create tokens as queries and keys, and uses each token as a value, transforming the tokens using a 1 x 1 depth convolution.

[0021]

[0022]

[0023]

[0024]

[0025] where Q , K, V represent the queries, keys, values matrices, respectively, is a scaling factor, and DConv1D x () represents a 1 x 1 depth convolution.

[0026] In a preferred embodiment, in step 2, the last stage of the ConvNeXt network encoder is inserted with a ConvLSTM; the ConvLSTM converts the 2D input in the LSTM into a 3D tensor, and the ConvLSTM uses three convolution gates to control the signal flow within the cell, and the ConvLSTM outputs a hidden state H t and uses a memory cell C t to control the update and output of the state:

[0027]

[0028]

[0029]

[0030]

[0031]

[0032] where i t ,f t ,o t are the input gate, the forget gate and the output gate of the ConvLSTM model respectively, σ = 1 / [1+exp(-x)] (sigmoid function) is the activation function of i t ,f t ,o t , and the hyperbolic tangent function tanh(x) = [exp(x)-exp(-x)] / [exp(x)+exp(-x)] is the activation function of c t ; '*' represents the convolution operator, and represents the Hadamard product; the initial saliency information extracted from the last layer of the ConvNeXt network encoder is input into the ConvLSTM, so as to realize the extraction of the time sequence saliency information.

[0033] In a preferred embodiment, in step 3, a residual refinement network is constructed to further optimize the edges of the salient region, and the residual s coarse between the saliency map s resdual and the GT is further optimized to obtain a fine segmentation map:

[0034] s refined = s coarse + s resdual (14)

[0035] The residual refinement network structure uses a U-shaped encoder-decoder structure and is connected using a bridge stage; each stage of the encoder and the decoder contains a 64-core 3*3 convolution layer, and each convolution layer is followed by a BatchNorm layer and a RELU activation function; the bridge stage has the same structure as a stage, that is, it contains a 64-core 3*3 convolution layer followed by a BatchNorm layer and a RELU activation function; after the time sequence significant region map is subjected to significant target edge optimization by the residual refinement network, the final segmentation map is output.

[0036] In a preferred embodiment, the step 4 comprises:

[0037] 1) Calculate stability

[0038] In order to fully utilize the size information of the merging and breaking area of the flotation foam, the foam stability S is defined, and the calculation formula is as follows, wherein n is 100:

[0039]

[0040]

[0041] In the formula: X i represents the number of pixels of each frame of the flotation foam image; x i is the number of pixel points of the unstable foam region segmented in the infrared flotation video frame; is the average number of pixel points of the segmented region of the current stage of the foam sequence; σ is the standard deviation of the number of pixel points of the segmented region of the current stage of the foam sequence; i=1, 2,..., n;

[0042] 2) Calculate deviation

[0043] The deviation can well reflect the distribution of the data, and the deviation can be used to evaluate the fluctuation of the foam stability, and the deviation r of the foam sequence is obtained by dividing the absolute value of the difference between the stability corresponding to m frames of infrared foam video frames and the average value by the standard deviation of the stability S of the flotation foam sequence in the section: i , and the calculation formulas are shown in formulas (9) and (10):

[0044]

[0045]

[0046] In the formula: s i is the stability S corresponding to the flotation foam sequence in the section; is the average value of the stability S of the flotation foam sequence in the section; η is the standard deviation of the flotation foam stability S; i=1, 2,..., m;

[0047] 3) Abnormal threshold setting

[0048] Setting an abnormal deviation threshold for early warning directly impacts the effectiveness of the warning. The sliding window method not only covers all data but also uses pre-calculated results to overwrite other data, avoiding redundant calculations and reducing runtime. During the sliding process, the average deviation T of windows with different widths is calculated to determine the unstable abnormal deviation threshold. The maximum average deviation is denoted as T. N This is the abnormal threshold:

[0049]

[0050] In the formula: N is the width of the sliding window.

[0051] Compared with the prior art, the present invention has the following beneficial effects: The method of the present invention can achieve good significant target segmentation results on both the flotation foam infrared video dataset and the public dataset RGB-T234. The segmentation accuracy of unstable bubbles under different working conditions is improved compared with the existing methods. It realizes accurate quantitative description of foam stability and lays the foundation for subsequent flotation production condition identification and early warning of faults. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of a U-shaped network design based on ConvNeXt according to a preferred embodiment of the present invention;

[0053] Figure 2 This is a schematic diagram of the dual cross-attention module according to a preferred embodiment of the present invention;

[0054] Figure 3 This is a schematic diagram of the residual refinement network structure of a preferred embodiment of the present invention;

[0055] Figure 4 This is a schematic diagram of the experimental process of a preferred embodiment of the present invention;

[0056] Figure 5 The image shows the RGB-T234 dataset segmentation results of a preferred embodiment of the present invention, wherein (a) is the original video frame; (b) is the ground truth label; (c) is the salient region of the time series; and (d) is the method of the present invention.

[0057] Figure 6 The above is a temporal salient target segmentation result of a preferred embodiment of the present invention, wherein (a) is the original float infrared foam video frame, (b) is the temporal salient region, and (c) is the spatial salient target optimization effect;

[0058] Figure 7 This is the result of over-flotation infrared foam segmentation in a preferred embodiment of the present invention;

[0059] Figure 8 Segmentation result of normal flotation froth infrared image for preferred embodiments of the present application;

[0060] Figure 9 Segmentation result of under-froth infrared image for preferred embodiments of the present application;

[0061] Figure 10 Different working condition froth stability display for preferred embodiments of the present application, wherein (a) is normal flotation stability, (b) is under-froth stability, and (c) is over-froth stability;

[0062] Figure 11 Deviation curve and anomaly judgment schematic diagram for preferred embodiments of the present application. DETAILED DESCRIPTION

[0063] The present application will be further described below in conjunction with the accompanying drawings and embodiments.

[0064] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application pertains.

[0065] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and furthermore, it should be understood that when the terms "comprise" and / or "include" are used in the present specification, they indicate the presence of a feature, step, operation, device, component, and / or combination thereof.

[0066] The present application proposes a flotation froth stability estimation method based on time series infrared salient target deep learning segmentation, which converts the flotation froth stability evaluation into an infrared salient target segmentation problem. First, a U-shaped network based on the ConvNeXt network design is adopted, a ConvLSTM network is embedded in the encoder to realize time series infrared saliency information extraction, and a cross-attention mechanism is added to the ConvNeXt network to realize initial positioning of the infrared salient region; second, a residual refinement network with a U-shaped encoder-decoder structure is constructed, the residual between the deep learning saliency map and the true value is used to improve the edge details of the salient region, and fine segmentation of the time series salient target is realized; finally, the froth stability is calculated according to the salient target segmentation result, and the deviation and abnormal threshold of the froth stability in the time series under different working conditions are counted.

[0067] The detailed technical solutions are as follows:

[0068] 1. ConvNeXt-based U-shaped network design

[0069] ConvNeXt is not an encoder-decoder model, only used for image classification and object detection. In order to complete the segmentation of infrared foam images, the present invention first adds a "decoder" part to the original ConvNeXt network, while the original structure is used as an "encoder" part. As shown in Figure 2 , the redesigned ConvNeXt network can meet the structure of "encoder-decoder". In the "decoder" part. Each upsampling block contains a 3x3 convolution, a BN layer, a GELU layer and a 1x1 convolution.

[0070] The present invention adopts a dual cross-attention module (DCA) to extract the mutual dependence of the channel dimension and the spatial dimension of the multi-scale features in the "encoder", thereby enhancing the ability of the model to segment different size infrared saliency regions. As shown in Figure 1 , the DCA used in the present invention takes the multi-scale features of the first three stages (the output of the last convolution layer of each stage) as input, generates enhanced representations, and connects them to the corresponding three decoder stages. As shown in Figure 2 , the DCA module is divided into two main stages. The first stage contains a multi-scale Patch Embedding to obtain "encoder" tokens. In the second stage, DCA mechanism is used on these "encoder" tokens to capture the relationship between different scale features through channel cross-attention (CCA) and spatial cross-attention (SCA) modules. Finally, LayerNorm and GeLU are applied, and these tokens are upsampled to connect them to the corresponding "decoder" part.

[0071] The CCA module concatenates tokens T i (i = 1, 2,..., n) along the channel dimension to create our keys and values T c , while using T c as queries, a 1x1 depth convolution is used to transform the tokens.

[0072]

[0073] K = DConv1D K (T c ) # (2)

[0074] V = DConv1D V (T c ) # (3)

[0075]

[0076] where Q, K, V represent queries, keys, values matrices respectively, i is a scaling factor, DConv1D x () represents a 1x1 depth convolution.

[0077] The SCA module takes the output from the CCA module The tokens after concatenation along the channel dimension as queries and keys, while using each tokens as values, transforms the tokens using a 1x1 depth convolution.

[0078]

[0079]

[0080]

[0081]

[0082] where Q, K, V represent queries, keys, values matrices respectively, is a scaling factor, DConv1D x () represents a 1x1 depth convolution.

[0083] 2. ConvLSTM-based time series salient region detection

[0084] To capture the dynamic feature information in time series, a ConvLSTM is inserted into the last stage of the ConvNeXt network encoder in the present application, as shown in Figure 4 ConvLSTM converts the 2D input in LSTM into a 3D tensor in order to overcome the shortcomings of LSTM in processing three-dimensional information, and ConvLSTM uses three convolution gates (input, output, and forget) to control the signal flow within the cell, ConvLSTM outputs a hidden state H t and uses a memory cell C t to control the update and output of the state:

[0085]

[0086]

[0087]

[0088] ​

[0089]

[0090] where i t ,f t ,o t are the input gate, forget gate and output gate in the ConvLSTM model respectively, and sigma = 1 / [1+exp(-x)] (sigmoid function) is the activation function of i t ,f t ,o t ; the hyperbolic tangent function tanh(x) = [exp(x)-exp(-x)] / [exp(x)+exp(-x)] is the activation function of c t ; '*' represents the convolution operator, and represents the Hadamard product.

[0091] The initial saliency information extracted from the encoder of the last layer of the ConvNeXt network is input into the ConvLSTM, so as to realize extraction of time sequence saliency information.

[0092] 3. Salient object segmentation based on residual refinement network

[0093] The salient region detected by the ConvNeXt network has problems such as fuzzy boundary, and the residual refinement network is constructed to further optimize the salient region edge, and the residual s coarse between the salient map s resdual and GT is further optimized to obtain a fine segmentation map:

[0094] s refined = s coarse + s resdual (14)

[0095] The residual refinement network structure is shown in Figure 3 , which adopts a U-shaped encoder-decoder structure and uses a bridge stage for connection. Each stage of the encoder and the decoder contains a 64-core 3x3 convolution layer, and each convolution layer is followed by a BatchNorm layer and a RELU activation function. The difference lies in that the encoder uses a max-pooling layer for down-sampling, while the decoder uses bilinear interpolation for up-sampling. The bridge stage is the same as a stage structure, that is, it contains a 64-core 3x3 convolution layer, followed by a BatchNorm layer and a RELU activation function. After the time sequence salient region map is optimized by the residual refinement network, the final segmentation map is output.

[0096] 4. Foam stability estimation and anomaly judgment

[0097] The present application estimates the foam stability S by using the time sequence infrared salient target segmentation result, and the yellow highlighted part in the infrared foam video, i.e. the broken and combined bubble, is segmented out, and then the number of pixel points in the segmented target region is counted to evaluate the stability, when some abnormality occurs in the flotation process, causing the related monitoring variables to deviate from the normal foam stability, when the deviation exceeds the set threshold, it is determined that the flotation foam stability is abnormal.

[0098] (1) Calculation of stability

[0099] In order to make full use of the size information of the flotation foam merging and breaking area, the foam stability S is defined as follows, where n is 100:

[0100]

[0101]

[0102] In the formula: X i represents the number of pixels of each frame of flotation foam image; x i is the number of pixel points of the unstable foam region segmented in the infrared flotation video frame; is the average number of pixel points of the segmented region of the current stage foam sequence; σ is the standard deviation of the number of pixel points of the segmented region of the current stage foam sequence; i = 1, 2, …, n.

[0103] (2) Calculation of deviation

[0104] The deviation can well reflect the distribution of the data, and the deviation can be used to evaluate the fluctuation of the foam stability, and the deviation r of the foam sequence is obtained by calculating the absolute value of the difference between the stability corresponding to m frames of infrared foam video frames and the average value divided by the standard deviation of the stability S of the flotation foam sequence in this section i , and the calculation formulas are shown in formulas (9) and (10):

[0105]

[0106]

[0107] In the formula: s i is the stability S corresponding to the flotation foam sequence in this section; is the average value of the stability S of the flotation foam sequence in this section; η is the standard deviation of the flotation foam stability S; i = 1, 2, …, m.

[0108] (3) Abnormal threshold setting

[0109] To judge the fluctuation anomaly of the foam stability under different working conditions, an offset degree anomaly threshold is set for early warning, and the setting of the anomaly threshold will directly affect the effect of early warning, and the sliding window method can not only cover all the data, but also can cover other data through the calculated results, avoid repeated calculation and reduce the running time. In the sliding process, the offset degree mean T of different width windows is calculated to determine the unstable anomaly threshold, and the maximum value of the offset degree mean is recorded as T N , that is, the anomaly threshold:

[0110]

[0111] In the formula: N is the width of the sliding window.

[0112] In the RGB-T234 data set, 234 infrared video sequences and their corresponding Ground Truth (GT) reference images are included, in the experiment, four infrared video sequences with different backgrounds are selected. Figure 7 The qualitative results of the four selected infrared video sequences segmented by the method of the present application are as follows: the first row is the original video frame image, the second row is the true label of the frame, the third row is the time sequence saliency region, and the fourth row is the result obtained after optimization by the residual refinement network. Selecting the people3, car41, jump and inglassandmobile sequences, it can be seen that, Figure 5 (c) The time saliency region sequence can roughly extract the target region, but the edge is very fuzzy and the segmentation is not detailed enough, but after optimization by the residual correction network, the final segmentation result as shown in Figure 5 (d) is obtained, which performs excellently and the edge is clear and detailed.

[0113] The effectiveness of the time sequence saliency target segmentation algorithm of the present application is experimentally tested, and the results are as shown in Figure 6 Figure 6 (a) is the original infrared foam video sequence frame; Figure 6 (b) is the time sequence saliency rough map extracted after the ConvNeXt network, which is the saliency detection result before the residual refinement module; Figure 6 (c) is the fine segmentation map, which is the saliency detection result after the residual refinement module. From the experimental results, it can be seen that the optimization result by the residual refinement module is better than that before optimization, and the residual refinement module can effectively handle the boundary blur condition, and the saliency region not only becomes smoother, but the background suppression is more thorough.

[0114] ​To verify the universality of the timing authorship target segmentation algorithm in different working conditions and different size types of bubbles, three groups of bubble infrared video sequences of over-flotation, normal flotation and under-flotation are used to experiment the proposed segmentation method (Ours) and compare it with other four algorithms qualitatively and quantitatively to verify the effectiveness of the algorithm proposed in this paper, as shown in Figs. Figure 7 , 8 , 9, which are the qualitative experimental results of the method in this paper and other comparison methods, where GroudTruth (GT) is the real value manually labeled. As can be seen from the figure, the segmentation effect of the method in this paper is closest to GT, the edge is smooth and the segmentation is more detailed; segformer can be seen in the over-flotation image that the segmentation result is not complete enough, and the internal region of the bubble is incorrectly identified as background; transunet is prone to identify the yellowish edge region as a merged or broken region, resulting in over-segmentation. At the same time, it can be seen from the over-flotation image that the segmentation result is not complete enough, and the internal region of the bubble is incorrectly identified as background.

[0115] In order to objectively and quantitatively measure the performance of various algorithms, the segmentation results are evaluated, and 50 normal flotation bubble images, 50 under-flotation bubble images and 50 over-flotation bubble images are selected as experimental objects for quantitative evaluation. Among the commonly used segmentation indicators, mean IoU (average value of IoU of all classes in the image), mean Dice (average value of Dice of all classes in the image) and mean absolute error (MAE) are selected to evaluate the experimental results:

[0116]

[0117]

[0118] In the formula: TP represents the overlapping area of the standard cut and the segmentation result of the algorithm; FN represents the area of the standard cut after removing the overlapping part TP; FP represents the area of the algorithm segmentation result after removing the overlapping part TP.

[0119]

[0120] In the formula: P and G are the predicted segmentation map and the real segmentation map of the image respectively, (H, W) and (r, c) are (height, width) and pixel coordinates respectively.

[0121] The quantitative comparison of each algorithm segmentation result is shown in Table 3: For the over-froth foam image segmentation result, transunet and segformer will incorrectly identify the pixels in the foam internal region as background, resulting in over-segmentation and reducing the segmentation accuracy. The algorithm in the present application can accurately segment the gap between the bubbles, and to some extent, improve the segmentation accuracy; for the under-froth foam image segmentation result, the segformer algorithm misidentifies and missegments small bubbles, resulting in over-segmentation and reducing the segmentation accuracy. For the normal flotation bubble image, the seven methods all show high segmentation accuracy. In summary, the method of the present application for segmenting synthesized and collapsed bubbles can exhibit high segmentation accuracy and good robustness under different flotation conditions, meeting the needs of online detection and dynamic changes in flotation production.

[0122] Table 1 Comparison of significant target segmentation results of foam infrared video

[0123]

[0124] In order to further verify the effectiveness of each module in the proposed segmentation algorithm, the present application uses normal flotation foam in the flotation foam infrared video as the verification data set, and adds ConvLSTM, double cross attention DCA module and residual refinement module to the ConvNeXt network as the baseline, respectively, to perform ablation experiments. As shown in Table 2, by comparing the four evaluation indexes of ConvNeXt, ConvNeXt+ConvLSTM, ConvNeXt+ConvLSTM+DCA, ConvNeXt+ConvLSTM+DCA+Residualrefinement module, the effectiveness of the ConvLSTM module, the double cross attention DCA module and the residual refinement module can be verified

[0125] Table 2 Ablation experiment

[0126]

[0127] The present application selects 10 infrared video samples of three over-froth flotation, normal flotation and under-froth flotation foams respectively, each video sample is 2s, performs time sequence significant target segmentation on the merged and broken foam, counts the number of pixels in the segmented region, calculates the corresponding stability S according to formula (7), and the stability curve of the statistics is as shown in Figure 10 As can be seen from Figure 10 (c) the stability of the over-froth foam is the highest, Figure 10 (b) the stability of the under-froth foam is the lowest, and Figure 10 (a) the stability of the normal flotation foam is moderate, which is consistent with the stability of the foam in the actual working condition.

[0128] To further verify the accuracy of stability calculation, 10 minutes of running data in the normal flotation, over-flotation and under-flotation states of the flotation plant were selected as experimental data, respectively, wherein the data sampling interval was 3s. The experimental data were input into the built segmentation model as a test set, the predicted value of stability was obtained by the stability calculation formula, and the RMSE, MAE and R 2 As an evaluation index, the average value of each parameter was taken as the evaluation result, as shown in Table 3.

[0129] Table 3 Prediction performance statistics of foam stability under three working conditions

[0130]

[0131] To verify the abnormal judgment effect of foam stability, infrared video images of the surface of the lead flotation tank in the flotation plant of Fujian Jindong Mining Co., Ltd. were collected in real time for experiment, the flotation reagent dosage was adjusted to be in the normal flotation state, the deviation degree of the foam stability was calculated according to formula (9), a sequence of deviation degrees was obtained, the deviation degree curve was as shown in Figure 11 The width N of the sliding window was selected to be 10, and T N =1.6145 could be obtained according to formula (11), and the red solid line was the abnormal threshold line. The reagent dosage was suddenly adjusted to be reduced by 20% at the 120th minute, so that the flotation tank slurry was in the under-flotation state, and the stability of the foam was reduced. As can be seen from the figure, the deviation degree suddenly increased and exceeded the abnormal threshold value after the 127th minute, and the reagent dosage was adjusted back to the original value at the 180th minute. As can be seen from the figure, the deviation degree gradually decreased and returned to the normal range after the 192nd minute. It can be seen that the change of the deviation degree can sensitively reflect the dynamic change of the foam stability, and the foam stability and the working condition abnormality can be effectively judged in time.

Claims

1. A method for froth stability estimation based on infrared temporal salient object segmentation, characterized in that, The method comprises the following steps: Step 1: The infrared foam video collected by the thermal infrared imager is extracted into images according to the frame number and is divided into a training set and a test set, and a significant area is manually segmented by using Lableme; Step 2: The image of the test set is input into the trained network model, and the ConvNeXt network with added double cross attention modules and ConvLSTM is used to realize the preliminary positioning of the time sequence infrared significant area; Step 3: A residual refinement network with a U-shaped encoder-decoder structure is constructed, and the residual between the deep learning significant map and the true value is refined to improve the edge details of the significant area, so as to realize the fine segmentation of the time sequence significant target; Step 4: Calculate the stability of the foam according to the segmentation result of the prominent target, and count the deviation degree of the stability of the foam in the time sequence under different working conditions and an anomaly threshold T; In step 2, a "decoder" part is added to the original ConvNeXt network, and the original structure is used as an "encoder" part; the redesigned ConvNeXt network meets the structure of "encoder-decoder"; in the "decoder" part, each upsampling block contains a 3x3 convolution, a BN layer, a GELU layer and a 1x1 convolution; The double cross attention module DCA is used to extract the mutual dependence of the channel dimension and the spatial dimension of the multi-scale features in the "encoder"; the double cross attention module DCA takes the multi-scale features of the previous three stages as input, generates enhanced representations, and connects them to the corresponding three decoder stages; the double cross attention module DCA module is divided into a first stage and a second stage; the first stage contains a multi-scale Patch Embedding for obtaining "encoder" tokens; in the second stage, the "encoder" tokens are processed by the DCA mechanism, and the relationship between different scale features is captured through the channel cross attention module CCA and the spatial cross attention module SCA; finally, Layer Norm and GeLU are applied, and the tokens are upsampled to connect them to the corresponding "decoder" part; The channel cross-attention module CCA concatenates tokens (i = 1, 2,..., n) along the channel dimension to create keys and values At the same time, using 1 x 1 depth convolution to transform tokens as queries; wherein, , , , respectively represent queries, keys, values matrices, is a scaling factor, represents a 1 x 1 depth convolution; The SCA module transforms the output from the CCA module tokens after concatenation along the channel dimension each tokens are used as queries and keys tokens are transformed using a 1 x 1 depthwise convolution as values; wherein, , , , respectively represent queries, keys, values matrices, is a scaling factor, represents a 1 x 1 depth convolution; In step 2, the last stage of the ConvNeXt network encoder is inserted with a ConvLSTM; the ConvLSTM converts the 2D input in the LSTM into a 3D tensor, and the ConvLSTM uses three convolution gates to control the signal flow within the cell, and the ConvLSTM outputs a hidden state and uses a memory cell to control the update and output of the state: (9) (10) (11) (12) (13) In the formula: These are the input gate, forget gate, and output gate in the ConvLSTM model. The sigmoid function is Activation function, hyperbolic tangent function yes The activation function; '∗' represents the convolution operator and ' 'Represents the Hadamard product; the initial saliency information extracted by the encoder of the last layer in the ConvNeXt network is input into the ConvLSTM to achieve the extraction of temporal saliency information.

2. The froth stability estimation method based on infrared time-of-flight salient object segmentation of claim 1, wherein, In step 2, the U-shaped network based on the ConvNeXt network design is used, the ConvLSTM network is embedded in the encoder to realize the time sequence infrared significant information extraction, and the cross attention mechanism is added to the ConvNeXt network to realize the preliminary positioning of the infrared significant area.

3. The froth stability estimation method based on infrared time-of-flight salient object segmentation of claim 1, wherein, In step 3, the residual refinement network is constructed to further optimize the edge of the salient region, and the saliency map is obtained through deep learning Residual between GT and GT Further optimize the salient edge to obtain a refined segmentation map: (14) The residual refinement network structure uses a U-shaped encoder-decoder structure and uses a bridge layer for connection; each stage of the encoder and the decoder contains a 3x3 convolution layer with 64 cores, and each convolution layer is followed by a Batch Norm layer and a RELU activation function; the bridge layer has the same structure as a stage, that is, it contains a 3x3 convolution layer with 64 cores, followed by a Batch Norm layer and a RELU activation function; the time sequence significant area graph is output after the significant target edge optimization by the residual refinement network.

4. The froth stability estimation method based on infrared time-of-flight salient object segmentation of claim 1, wherein, Step 4 comprises: 1) Calculate the stability To make full use of the information of the size of the merging and breaking regions of the flotation froth, the froth stability S is defined as follows: Take 100: (15) (16) wherein: represents the number of pixels of each frame of flotation froth image; represents the number of pixels of the unstable froth region segmented in the infrared flotation video frame; represents the average number of pixels of the segmented region of the froth sequence in the current stage; represents the standard deviation of the number of pixels of the segmented region of the froth sequence in the current stage; ; 2) Calculate the deviation The deviation degree can well reflect the distribution of data, and the fluctuation of the stability of the foam can be evaluated by using the deviation degree, and the deviation degree of the foam sequence is calculated by The deviation degree of the foam sequence is calculated by dividing the absolute value of the difference between the stability of the infrared foam video frame corresponding to the frame and the average value by the standard deviation of the stability S of the foam sequence , and the calculation formula is shown in formulas (9) and (10): (17) (18) wherein: S is the stability of the sequence of flotation froth for this section; S is the average value of the stability S of the sequence of flotation froth for this section; S is the standard deviation of the stability S of the flotation froth; ; 3) Set the abnormal threshold The deviation degree abnormal threshold is set to give a warning, and the setting of the abnormal threshold will directly affect the effect of the warning. The sliding window method can cover all data, and the results calculated are used to cover other data, so as to avoid repeated calculation and reduce running time. In the sliding process, the deviation degree mean T of different width windows is calculated to determine the unstable abnormal threshold, and the maximum value of the deviation degree mean is recorded as , which is the abnormal threshold. (19) In the formula, N is the width of the sliding window.

Images