Flotation froth stability estimation method based on infrared time sequence salient target segmentation

Abstract

The invention provides a flotation froth stability estimation method based on infrared time sequence salient target segmentation, which comprises the following steps of: firstly, embedding a ConvLSTM network into an encoder by adopting a U-shaped network designed based on a ConvNeXt network so as to realize time sequence infrared salient information extraction; a cross attention mechanism is added into the ConvNeXt network to realize the initial positioning of the infrared saliency region; secondly, constructing a residual error refining network of a U-shaped encoder-decoder structure, deeply learning a residual error between a saliency map and a truth value so as to improve edge details of a saliency region and realize refined segmentation of a time sequence saliency target, and finally, calculating foam stability according to a saliency target segmentation result so as to realize the segmentation of the time sequence saliency target. The deviation degree and the abnormal threshold value of the foam stability in the time sequence under different working conditions are counted, salient region detection is carried out on the foam infrared video image to realize segmentation of merged and broken bubbles, and the foam stability is evaluated according to the segmentation result.

Description

Technical Field

[0001] The invention relates to the technical field of foam image processing, in particular to a flotation foam stability estimation method based on infrared time series significant target segmentation. Background Art

[0002] The generation and collapse of bubbles during flotation can effectively reflect the mineral content. Image segmentation technology can detect newly generated bubbles and collapsed bubbles during flotation. This technology has broad application prospects in computer vision and industrial production. Foam stability is defined as the ability of bubbles in the foam to resist merging and rupture. It is a key driver of flotation performance and helps determine the grade and recovery rate of the flotation process.

[0003] In recent years, there are many methods for detecting unstable bubbles. Based on image analysis and neural networks, according to the difference between the shadows formed during the generation and dissipation of bubbles, combined with the speed information of bubble movement, the collapsed foam can be identified to evaluate the stability, but the detection accuracy of this method will be affected by bubble deformation and illumination, resulting in misjudgment of the bright spots or bright edges of bubbles; the watershed algorithm is used to study the deformation law of flotation foam under different process parameters to segment the size of bubbles, but this method is also affected by illumination, and is prone to over-segmentation or under-segmentation when merging or breaking bubbles; based on the flotation condition recognition of dual-modal convolutional neural networks, researchers conducted infrared thermal imaging research on foam and found that bubbles release heat after collapse or merging, which is higher than the temperature of surrounding bubbles. After thermal imaging, a significant area of ​​highlight appears, which has an intuitive display effect on unstable bubbles.

[0004] Due to the stirring effect of the flotation machine and the squeezing of mineralized bubbles, the flotation bubbles inevitably undergo geometric changes such as angle rotation and size scaling during the flow (flow to the scraper), and the bubbles merge, collapse, and break, accompanied by the generation of new bubbles. Therefore, it is difficult to use conventional image processing methods to segment bubbles in foam images, and it is also difficult to track and identify independent bubbles with severe deformation and high collapse and breakage rates, so that the flotation foam image features cannot be accurately extracted and quantified. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a flotation foam stability estimation method based on infrared time series salient target segmentation, by performing salient area detection on foam infrared video images to achieve segmentation of merged and broken bubbles, and evaluating the foam stability based on the segmentation results.

[0006] To achieve the above object, the present invention adopts the following technical solution: a flotation foam stability estimation method based on infrared time series significant target segmentation, comprising the following steps:

[0007] Step 1: Extract the infrared foam video captured by the thermal infrared imager into images according to the number of frames and divide them into training sets and test sets, and manually segment the salient areas using Labelme;

[0008] Step 2: Use the images in the test set as input to the trained network model, and use the ConvNeXt network with double cross attention module and ConvLSTM to achieve the initial positioning of the infrared salient area of ​​the time series;

[0009] Step 3: Construct a residual refinement network with a U-shaped encoder-decoder structure, deeply learn the residual between the salient map and the true value to improve the edge details of the salient area and achieve refined segmentation of temporal salient objects;

[0010] Step 4: Calculate the foam stability based on the salient target segmentation results, and count the deviation r of the foam stability in the time series under different working conditions. i and the anomaly threshold T.

[0011] In a preferred embodiment, in step 2, a U-type network based on ConvNeXt network design is used, a ConvLSTM network is embedded in the encoder to realize time series infrared saliency information extraction, and a cross-attention mechanism is added to the ConvNeXt network to realize the initial positioning of infrared saliency areas.

[0012] In a preferred embodiment, in step 2, a "decoder" part is first added to the original ConvNeXt network, and the original structure is used as the "encoder" part; the redesigned ConvNeXt network satisfies the "codec" structure; in the "decoder" part, each upsampling block contains a 3×3 convolution, a BN layer, a GELU layer and a 1×1 convolution;

[0013] The dual cross attention module DCA is used to extract the interdependence of the channel dimension and spatial dimension of the multi-scale features in the "encoder"; the dual cross attention module DCA uses the multi-scale features of the first three stages as input, generates enhanced representations, and connects them to the corresponding three decoder stages; the dual cross attention module DCA module is divided into the first stage and the second stage; the first stage contains a multi-scale Patch Embedding to obtain the "encoder" tokens; in the second stage, the DCA mechanism is used on the "encoder" tokens to capture the relationship between features of different scales through the channel cross attention module CCA and the spatial cross attention module SCA module; finally, LayerNorm and GeLU are applied, and these tokens are upsampled so that they can be connected to the corresponding "decoder" part;

[0014] The channel cross attention module CCA converts tokens T i (i=1, 2, ..., n) concatenate along the channel dimension to create keys and values ​​T c , while using T c As queries, tokens are transformed using 1×1 depthwise convolution;

[0015]

[0016] K = DConv1D K (T c ) (2)

[0017] V = DConv1D V (T c ) (3)

[0018]

[0019] Among them, Q i ,K,V,represent the query,keys,value matrices respectively. is the scaling factor, DConv1D x () represents 1×1 depthwise convolution;

[0020] The SCA module converts the output from the CCA module Tokens concatenated along the channel dimension As queries and keys, use each token As values, tokens are transformed using 1×1 depthwise convolution;

[0021]

[0022]

[0023]

[0024]

[0025] Among them, Q, K, V, respectively represent the query, key, and value matrices. is the scaling factor, DConv1D x () represents a 1×1 depthwise convolution.

[0026] In a preferred embodiment, in step 2, the last stage of the ConvNeXt network encoder is inserted into the ConvLSTM; the ConvLSTM converts the 2D input in the LSTM into a 3D tensor, and the ConvLSTM uses three convolution gates to control the signal flow in the unit. The ConvLSTM outputs a hidden state H t , and uses a memory cell C t To control the update and output of the status:

[0027]

[0028]

[0029]

[0030]

[0031]

[0032] Where: i t ,f t ,o t They are the input gate, forget gate and output gate in the ConvLSTM model, and σ=1 / [1+exp(-x)] (sigmoid function) is i t ,f t ,o t The activation function, the hyperbolic tangent function tanh(x) = [exp(x) - exp(-x)] / [exp(x) + exp(-x)] is c t The activation function of ; '*' indicates the convolution operator and Represents Hadamard product; the initial saliency information extracted by the encoder of the last layer in the ConvNeXt network is input into ConvLSTM to realize the extraction of temporal saliency information.

[0033] In a preferred embodiment, in step 3, a residual refinement network is constructed to further optimize the edges of the salient regions, and the salient maps are learned through deep learning. coarse The residual between GT resdual Further optimize the salient edges to obtain a refined segmentation map:

[0034] s refined =s coarse +s resdual (14)

[0035] The residual refinement network structure uses a U-shaped encoder-decoder structure and is connected using a bridge layer. Each stage of the encoder and decoder contains a 3×3 convolution layer with 64 cores, and each convolution layer is followed by a BatchNorm layer and a RELU activation function. The bridge layer has the same structure as a stage, that is, it contains a 3×3 convolution layer with 64 cores, followed by a BatchNorm layer and a RELU activation function. The time series salient area map is optimized through the residual refinement network for salient target edges and the final segmentation map is output.

[0036] In a preferred embodiment, step 4 includes:

[0037] 1) Calculate stability

[0038] In order to make full use of the information on the size of the flotation foam merging and rupture areas, the calculation formula of the foam stability S is defined as follows, where n is 100:

[0039]

[0040]

[0041] Where: X i Indicates the number of pixels in each frame of flotation foam image; x i is the number of pixels of the segmented unstable foam area in the infrared flotation video frame; is the average number of pixels in the segmentation area of ​​the foam sequence at the current stage; σ is the standard deviation of the number of pixels in the segmentation area of ​​the foam sequence at the current stage; i=1,2,......,n;

[0042] 2) Calculate the deviation

[0043] The deviation can well reflect the distribution of data. The deviation can be used to evaluate the fluctuation of foam stability. The deviation r of the foam sequence is obtained by calculating the absolute value of the difference between the stability corresponding to m frames of infrared foam video and its average value and dividing it by the standard deviation of the stability S of the flotation foam sequence. i , the calculation formula is shown in formula (9) and (10):

[0044]

[0045]

[0046] Where: s i is the stability S corresponding to the flotation foam sequence of this section; is the average value of the flotation foam sequence stability S of this section; η is the standard deviation of the flotation foam stability S; i=1,2,......,m;

[0047] 3) Abnormal threshold setting

[0048] Set the deviation abnormality threshold for early warning. The setting of the abnormality threshold will directly affect the effect of the early warning. The sliding window method can not only cover all the data, but also cover other data with the calculated results to avoid repeated calculations and reduce runtime. During the sliding process, calculate the deviation mean T of windows of different widths to determine the unstable abnormality threshold, and record the maximum deviation mean as T. N , which is the abnormal threshold:

[0049]

[0050] Where: N is the width of the sliding window.

[0051] Compared with the prior art, the present invention has the following beneficial effects: the method of the present invention can achieve good significant target segmentation effect on both the flotation foam infrared video dataset and the public dataset RGB-T234, and the unstable bubble segmentation accuracy under different working conditions is improved compared with the prior art method, realizing an accurate quantitative description of the foam stability, laying the foundation for subsequent flotation production condition identification and early fault warning. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A schematic diagram of a U-shaped network design based on ConvNeXt according to a preferred embodiment of the present invention;

[0053] Figure 2 A schematic diagram of a dual cross attention module according to a preferred embodiment of the present invention;

[0054] Figure 3 A schematic diagram of a residual refinement network structure in a preferred embodiment of the present invention;

[0055] Figure 4 It is a schematic diagram of the experimental process of a preferred embodiment of the present invention;

[0056] Figure 5 The RGB-T234 dataset segmentation result of the preferred embodiment of the present invention, wherein (a) is the original video frame; (b) is the true label; (c) is the time series salient region; (d) is the method of the present invention;

[0057] Figure 6 The temporal salient target segmentation result of the preferred embodiment of the present invention, wherein (a) is the original flotation infrared foam video frame, (b) is the temporal salient region, and (c) is the spatial salient target optimization effect;

[0058] Figure 7 This is the infrared foam segmentation result of the flotation in the preferred embodiment of the present invention;

[0059] Figure 8 This is the normal flotation foam infrared image segmentation result of the preferred embodiment of the present invention;

[0060] Fig. 9 This is the infrared image segmentation result of under-flotation foam in the preferred embodiment of the present invention;

[0061] Fig.10 The foam stability under different working conditions of the preferred embodiment of the present invention is shown in FIG. 1 , wherein (a) is normal flotation stability, (b) is under-flotation stability, and (c) is over-flotation stability;

[0062] Fig.11 Schematic diagram of the deviation curve and abnormality judgment of the preferred embodiment of the present invention. DETAILED DESCRIPTION

[0063] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0064] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0065] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0066] The present invention proposes a flotation foam stability estimation method based on deep learning segmentation of time series infrared salient targets, which transforms the flotation foam stability assessment into the infrared salient target segmentation problem. Firstly, a U-type network based on ConvNeXt network design is adopted to embed the ConvLSTM network in the encoder to realize the extraction of time series infrared salient information, and the cross attention mechanism is added to the ConvNeXt network to realize the initial positioning of infrared salient areas; secondly, a residual refinement network of U-type encoder-decoder structure is constructed, and the residual between the salient map and the true value is deeply learned to improve the edge details of the salient area, so as to realize the refined segmentation of time series salient targets. Finally, the foam stability is calculated according to the salient target segmentation result, and the deviation and abnormal threshold of the foam stability in the time series under different working conditions are statistically analyzed.

[0067] The detailed technical solution is as follows:

[0068] 1. U-shaped network design based on ConvNeXt

[0069] ConvNeXt is not an encoder-decoder model and is only used for image classification and object detection. In order to complete the segmentation of infrared foam images, the present invention first adds a "decoder" part to the original ConvNeXt network, and the original structure is used as the "encoder" part. Figure 2 As shown in Figure 1, the redesigned ConvNeXt network can satisfy the structure of "encoder-decoder". In the "decoder" part, each upsampling block contains a 3×3 convolution, a BN layer, a GELU layer, and a 1×1 convolution.

[0070] The present invention adopts a dual cross attention module (DCA) to extract the interdependence of the channel dimension and spatial dimension of multi-scale features in the "encoder", thereby enhancing the model's ability to segment infrared salient regions of different sizes. Figure 1 , the DCA adopted by the present invention takes the multi-scale features of the first three stages (the output of the last convolutional layer of each stage) as input, generates enhanced representations, and connects them to the corresponding three decoder stages. Figure 2 As shown in the figure, the DCA module is divided into two main stages. The first stage contains a multi-scale Patch Embedding to obtain the "encoder" tokens. In the second stage, the DCA mechanism is used on these "encoder" tokens to capture the relationship between features at different scales through the channel cross attention (CCA) and spatial cross attention (SCA) modules. Finally, LayerNorm and GeLU are applied, and these tokens are upsampled so that they can be connected to the corresponding "decoder" part.

[0071] The CCA module converts tokensT i (i=1, 2, ..., n) concatenate along the channel dimension to create our keys and values ​​T c , while using T c As queries, tokens are transformed using a 1×1 depthwise convolution.

[0072]

[0073] K = DConv1D K (T c )# (2)

[0074] V = DConv1D V (T c )# (3)

[0075]

[0076] Among them, Q i ,K,V,represent the query,keys,value matrices respectively. is the scaling factor, DConv1D x () represents a 1×1 depthwise convolution.

[0077] The SCA module converts the output from the CCA module Tokens concatenated along the channel dimension As queries and keys, use each token As values, tokens are transformed using a 1×1 depthwise convolution.

[0078]

[0079]

[0080]

[0081]

[0082] Among them, Q, K, V, respectively represent the query, key, and value matrices. is the scaling factor, DConv1D x () represents a 1×1 depthwise convolution.

[0083] 2. Time series salient region detection based on ConvLSTM

[0084] In order to capture the dynamic feature information in time series, ConvLSTM is inserted into the last stage of the ConvNeXt network encoder in the present invention, such as Figure 4 To overcome the shortcomings of LSTM in processing three-dimensional information, ConvLSTM converts the 2D input in LSTM into a 3D tensor. ConvLSTM uses three convolution gates (input, output, and forget) to control the signal flow within the unit. ConvLSTM outputs a hidden state H t , and uses a memory cell C t To control the update and output of the status:

[0085]

[0086]

[0087]

[0088]

[0089]

[0090] Where: i t ,f t ,o t They are the input gate, forget gate and output gate in the ConvLSTM model, and σ=1 / [1+exp(-x)] (sigmoid function) is i t ,f t ,o t The activation function, the hyperbolic tangent function tanh(x) = [exp(x)-exp(-x)] / [exp(x)+exp(-x)] is c t The activation function of ; '*' indicates the convolution operator and Represents Hadamard.

[0091] The initial saliency information extracted by the encoder of the last layer in the ConvNeXt network is input into the ConvLSTM to extract temporal saliency information.

[0092] 3. Salient object segmentation based on residual refinement network

[0093] The salient regions detected by the ConvNeXt network have problems such as blurred boundaries. The present invention constructs a residual refinement network to further optimize the edges of the salient regions. coarse The residual between GT and resdual Further optimize the salient edges to obtain a refined segmentation map:

[0094] s refined =s coarse +s resdual (14)

[0095] The residual refinement network structure is as follows Figure 3 As shown in the figure, a U-shaped encoder-decoder structure is adopted and connected using a bridge layer. Each stage of the encoder and decoder contains a 3×3 convolution layer with 64 cores, and each convolution layer is followed by a BatchNorm layer and a RELU activation function. The difference is that the encoder uses a maximum pooling layer for downsampling, while the decoder uses bilinear interpolation for upsampling. The bridge layer has the same structure as a stage, that is, it contains a 3×3 convolution layer with 64 cores, followed by a BatchNorm layer and a RELU activation function. The time series salient area map is passed through the residual refinement network to optimize the salient target edge and output the final segmentation map.

[0096] 4. Foam stability estimation and abnormality assessment

[0097] The present invention estimates the foam stability S by using the time-series infrared significant target segmentation results. After the yellow highlighted part in the infrared foam video, i.e., the ruptured and merged bubbles, is segmented out, the number of pixels in the segmented target area is counted to perform stability assessment. When some abnormality occurs in the flotation process, causing the relevant monitoring variables to deviate from the normal foam stability, when this deviation exceeds the set threshold, the flotation foam stability is judged to be abnormal.

[0098] (1) Calculation of stability

[0099] In order to make full use of the information on the size of the flotation foam merging and rupture areas, the calculation formula of the foam stability S is defined as follows, where n is 100:

[0100]

[0101]

[0102] Where: X i Indicates the number of pixels in each frame of flotation foam image; x i is the number of pixels of the segmented unstable foam area in the infrared flotation video frame; is the average number of pixels in the bubble sequence segmentation area at the current stage; σ is the standard deviation of the number of pixels in the bubble sequence segmentation area at the current stage; i=1,2,......,n;.

[0103] (2) Calculate the deviation

[0104] The deviation can well reflect the distribution of data. The deviation can be used to evaluate the fluctuation of foam stability. The deviation r of the foam sequence is obtained by calculating the absolute value of the difference between the stability corresponding to m frames of infrared foam video and its average value and dividing it by the standard deviation of the stability S of the flotation foam sequence. i , the calculation formula is shown in formula (9) and (10):

[0105]

[0106]

[0107] Where: s i is the stability S corresponding to the flotation foam sequence of this section; is the average value of the flotation foam sequence stability S of this section; η is the standard deviation of the flotation foam stability S; i=1,2,......,m.

[0108] (3) Abnormal threshold setting

[0109] In order to judge the abnormal fluctuation of foam stability under different working conditions, the deviation abnormal threshold is set for early warning. The setting of the abnormal threshold will directly affect the effect of the early warning. The sliding window method can not only cover all data, but also cover other data with the calculated results to avoid repeated calculations and reduce runtime. During the sliding process, the deviation mean T of windows of different widths is calculated to determine the unstable abnormal threshold, and the maximum deviation mean is recorded as T N , which is the abnormal threshold:

[0110]

[0111] Where: N is the width of the sliding window.

[0112] The RGB-T234 dataset includes 234 infrared video sequences and their corresponding Ground Truth (GT) benchmark images. In this experiment, four infrared video sequences with different backgrounds are selected. Figure 7 The qualitative results of the segmentation of the four selected infrared video sequences by the method of the present invention are as follows: the first line is the original video frame image, the second line is the real label of the frame, the third line is the time series salient area, and the fourth line is the result obtained after the residual refinement network optimization. Selecting the sequences of people3, car41, jump, inglassandmobile, it can be seen that Figure 5 (c) The temporal salient region sequence can roughly extract the target area, but the edge is fuzzy and the segmentation is not detailed enough. However, after the residual correction optimization network is used, the following is obtained: Figure 5 The final segmentation result shown in (d) has excellent performance and clear edges.

[0113] The effectiveness of the temporal significant target segmentation algorithm of the present invention is experimentally tested, and the results are as follows: Figure 6 As shown: Figure 6 (a) is the original infrared foam video sequence frame; Figure 6 (b) is the temporal saliency rough map extracted after the ConvNeXt network and the saliency detection result before the residual refinement module; Figure 6 (c) is the refined segmentation map and the saliency detection result after the residual refinement module. From the experimental results, it can be seen that the optimization result after the residual refinement module is better than that before optimization. The residual refinement module can effectively handle the situation of blurred boundaries. The salient area not only becomes smoother, but also the background suppression is more thorough.

[0114] In order to verify the universality of the temporal significant target segmentation algorithm of the present invention under different working conditions and bubbles of different sizes and types, the proposed segmentation method (Ours) was experimented with three groups of bubble infrared video sequences of over-flotation, normal flotation and under-flotation, and qualitatively and quantitatively compared with the other four algorithms to verify the effectiveness of the algorithm proposed in this paper. Figure 7 , 8 9 show the qualitative experimental results of the proposed method and other comparison methods, where GroudTruth (GT) is the true value of manual annotation. As can be seen from the figure, the segmentation effect of the proposed method is closest to GT, with smooth edges and more detailed segmentation; segformer can be seen in the over-flotation image that the segmentation result is not complete, and the pixels in the foam internal area are mistakenly identified as background; transunet easily identifies the yellow bright area on the edge as a merged or broken area, resulting in over-segmentation. At the same time, it can be seen in the over-flotation image that the segmentation result is not complete, and the pixels in the foam internal area are mistakenly identified as background.

[0115] In order to objectively and quantitatively measure the performance of various algorithms, the segmentation results were evaluated. 50 normal flotation bubble images, under-floatation bubble images, and over-floatation bubble images were selected as experimental objects for quantitative evaluation. Mean IoU (the average value of IoU of all categories in the image), mean Dice (the average value of Dice of all categories in the image) and mean absolute error (MAE) were selected from the commonly used segmentation indicators to evaluate the experimental results:

[0116]

[0117]

[0118] Where TP represents the area where the standard cut and the algorithm segmentation result overlap; FN represents the area where the standard cut removes the overlapping part TP; FP represents the area where the algorithm segmentation result removes the overlapping part TP.

[0119]

[0120] Where: P and G are the predicted segmentation map and the real segmentation map of the image, respectively, (H, W) and (r, c) are (height, width) and pixel coordinates, respectively.

[0121] The quantitative comparison of the segmentation results of each algorithm is shown in Table 3: For the segmentation results of over-flotation foam images, transunet and segformer will mistakenly identify the pixels in the foam internal area as background, resulting in over-segmentation and thus reduced segmentation accuracy. The algorithm in this paper can accurately segment the gaps between bubbles, which improves the segmentation accuracy to a certain extent; for the segmentation results of under-flotation foam images, the segformer algorithm misdetects and mis-segments small bubbles, resulting in over-segmentation and thus reduced segmentation accuracy. For normal flotation bubble images, all seven methods show high segmentation accuracy. In summary, the method of the present invention is used to segment synthetic and collapsed bubbles, and can show high segmentation accuracy. It shows good robustness under different flotation conditions, meeting the needs of online detection and dynamic changes in flotation production.

[0122] Table 1 Comparison of salient target segmentation results in foam infrared video

[0123]

[0124] In order to further verify the effectiveness of each module in the proposed segmentation algorithm, the present invention uses the normal flotation foam in the flotation foam infrared video as the verification data set, takes the ConvNeXt network as the baseline, and adds ConvLSTM, dual cross attention DCA module and residual refinement module respectively to perform ablation experiments. As shown in Table 2, by comparing the four evaluation indicators of ConvNeXt, ConvNeXt+ConvLSTM, ConvNeXt+ConvLSTM+DCA, and ConvNeXt+ConvLSTM+DCA+residual refinement, the effectiveness of the ConvLSTM module, dual cross attention DCA module and residual refinement module can be verified.

[0125] Table 2 Ablation experiment

[0126]

[0127] The present invention selects 10 infrared video samples of three sections of over-flotation, normal flotation and under-flotation foam, each of which is 2s, and performs temporal significant target segmentation on the merged and broken foams. The number of pixels in the segmented areas is counted, and the corresponding stability S is calculated according to formula (7). The statistical stability curve is shown in the figure. Fig.10 As shown, it can be seen Fig.10 (c) The stability of flotation foam is the highest. Fig.10 (b) Underflotation foam has the lowest stability, while Fig.10 (a) The stability of normal flotation foam is moderate, which is consistent with the stability of foam under actual working conditions.

[0128] In order to further verify the accuracy of stability calculation, 10 minutes of operation data of the flotation plant under normal flotation, over-flotation and under-flotation conditions were selected as experimental data, where the data sampling interval was 3s. The experimental data was input into the built segmentation model as a test set, and the predicted value of stability was obtained through the stability calculation formula, and RMSE, MAE and R were selected. 2 As the evaluation index, the average value of each parameter is taken as the evaluation result, as shown in Table 3.

[0129] Table 3 Foam stability prediction performance statistics under three working conditions

[0130]

[0131] In order to verify the judgment effect of abnormal foam stability, the infrared video image of the foam surface in the lead flotation tank was collected in real time in the flotation plant of Fujian Jindong Mining Co., Ltd. to conduct experiments. The flotation agent dosage was adjusted to a normal flotation state. The deviation of the foam stability was calculated according to formula (9) to obtain a deviation sequence. The deviation curve is shown in Fig.11 As shown, the width N of the sliding window is selected as 10. According to formula (11), T N =1.6145, the solid red line is the abnormal threshold line. At the 120th minute, the dosage of the reagent was suddenly adjusted to decrease by 20%, so that the flotation tank pulp was in an under-flotation state, reducing the stability of the foam. It can be seen from the figure that the deviation suddenly increased and exceeded the abnormal threshold after the 127th minute. At the 180th minute, the dosage of the reagent was adjusted back to the original value. It can be seen from the figure that after the 192th minute, the deviation gradually decreased back to the original normal range. It can be seen that the change in deviation can sensitively reflect the dynamic change of foam stability, and can make effective judgments on foam stability and abnormal working conditions in a timely manner.

Claims

1. A flotation froth stability estimation method based on infrared time series salient target segmentation, characterized in that: The following steps are involved: Step 1: Extract the infrared foam video captured by the thermal infrared imager into images according to the number of frames and divide them into training sets and test sets, and manually segment the salient areas using Labelme; Step 2: Use the images in the test set as input to the trained network model, and use the ConvNeXt network with double cross attention module and ConvLSTM to achieve the initial positioning of the infrared salient area of ​​the time series; Step 3: Construct a residual refinement network with a U-shaped encoder-decoder structure, deeply learn the residual between the salient map and the true value to improve the edge details of the salient area and achieve refined segmentation of temporal salient objects; Step 4: Calculate the foam stability based on the salient target segmentation results, and count the deviation r of the foam stability in the time series under different working conditions. i and the anomaly threshold T.

2. The method for estimating froth stability of flotation based on infrared time series salient target segmentation according to claim 1, characterized in that: In the step 2, a U-type network designed based on the ConvNeXt network is adopted, and a ConvLSTM network is embedded in the encoder to realize the extraction of time series infrared salient information, and a cross attention mechanism is added to the ConvNeXt network to realize the initial positioning of the infrared salient area.

3. The method for estimating froth stability of flotation based on infrared time series salient target segmentation according to claim 2, characterized in that: In step 2, a "decoder" part is first added to the original ConvNeXt network, and the original structure is used as the "encoder" part; the redesigned ConvNeXt network satisfies the "encoder-decoder" structure; in the "decoder" part, each upsampling block contains a 3×3 convolution, a BN layer, a GELU layer and a 1×1 convolution; The dual cross attention module DCA is used to extract the interdependence of the channel dimension and spatial dimension of the multi-scale features in the "encoder". The dual cross attention module DCA takes the multi-scale features of the first three stages as input, generates enhanced representations, and connects them to the corresponding three decoder stages. The dual cross attention module DCA module is divided into the first stage and the second stage. The first stage contains a multi-scale Patch Embedding to obtain the "encoder" tokens. In the second stage, the DCA mechanism is used on the "encoder" tokens to capture the relationship between features of different scales through the channel cross attention module CCA and the spatial cross attention module SCA module. Finally, LayerNorm and GeLU are applied, and these tokens are upsampled so that they can be connected to the corresponding "decoder" part. The channel cross attention module CCA converts tokens T i (i=1, 2, ..., n) concatenate along the channel dimension to create keys and values ​​T c , while using T c As queries, tokens are transformed using 1×1 depthwise convolution; K=DConv1D K (T c ) (2) V=DConv1D V (T c ) (3) Among them, Q i ,K,V,represent the query,keys,value matrices respectively. is the scaling factor, DConv1D x () represents 1×1 depthwise convolution; The SCA module converts the output from the CCA module Tokens concatenated along the channel dimension As queries and keys, use each token As values, tokens are transformed using 1×1 depthwise convolution; Among them, Q, K, V, respectively represent the query, key, and value matrices. is the scaling factor, DConv1D x () represents a 1×1 depthwise convolution.

4. The method for estimating froth stability of flotation based on infrared time series salient target segmentation according to claim 2, characterized in that: In step 2, the last stage of the ConvNeXt network encoder is inserted into the ConvLSTM; the ConvLSTM converts the 2D input in the LSTM into a 3D tensor. The ConvLSTM uses three convolution gates to control the signal flow within the unit. The ConvLSTM outputs a hidden state H t , and uses a memory cell C t To control the update and output of the status: Where: i t ,f t ,o t They are the input gate, forget gate and output gate in the ConvLSTM model, and σ=1 / [1+exp(-x)] (sigmoid function) is i t ,f t ,o t The activation function, the hyperbolic tangent function tanh(x) = [exp(x)-exp(-x)] / [exp(x)+exp(-x)] is c t The activation function of ; '*' indicates the convolution operator and Represents Hadamard; The initial saliency information extracted by the encoder of the last layer in the ConvNeXt network is input into the ConvLSTM to extract temporal saliency information.

5. The method for estimating froth stability of flotation based on infrared time series salient target segmentation according to claim 1, characterized in that: In step 3, a residual refinement network is constructed to further optimize the edge of the salient area, and the salient map s is obtained by deep learning. coarse The residual between GT resdual Further optimize the salient edges to obtain a refined segmentation map: s refined =s coarse +s resdual (14) The residual refinement network structure uses a U-shaped encoder-decoder structure and is connected using a bridge layer. Each stage of the encoder and decoder contains a 3×3 convolution layer with 64 cores, and each convolution layer is followed by a BatchNorm layer and a RELU activation function. The bridge layer has the same structure as a stage, that is, it contains a 3×3 convolution layer with 64 cores, followed by a BatchNorm layer and a RELU activation function. The time series salient area map is optimized through the residual refinement network for salient target edges and the final segmentation map is output.

6. The method for estimating froth stability of flotation based on infrared time series salient target segmentation according to claim 1, characterized in that: The step 4 comprises: 1) Calculate stability In order to make full use of the information on the size of the flotation foam merging and rupture areas, the calculation formula of the foam stability S is defined as follows, where n is 100: Where: X i Indicates the number of pixels in each frame of flotation foam image; x i is the number of pixels of the segmented unstable foam area in the infrared flotation video frame; is the average number of pixels in the segmentation area of ​​the foam sequence at the current stage; σ is the standard deviation of the number of pixels in the segmentation area of ​​the foam sequence at the current stage; i=1,2,......,n; 2) Calculate the deviation The deviation can well reflect the distribution of data. The deviation can be used to evaluate the fluctuation of foam stability. The deviation r of the foam sequence is obtained by calculating the absolute value of the difference between the stability corresponding to m frames of infrared foam video and its average value and dividing it by the standard deviation of the stability S of the flotation foam sequence. i , the calculation formula is shown in formula (9) and (10): Where: s i is the stability S corresponding to the flotation foam sequence of this section; is the average value of the flotation foam sequence stability S of this section; η is the standard deviation of the flotation foam stability S; i=1,2,......,m; 3) Abnormal threshold setting Set the deviation abnormality threshold for early warning. The setting of the abnormality threshold will directly affect the effect of the early warning. The sliding window method can not only cover all the data, but also cover other data with the calculated results to avoid repeated calculations and reduce runtime. During the sliding process, calculate the deviation mean T of windows of different widths to determine the unstable abnormality threshold, and record the maximum deviation mean as T. N , which is the abnormal threshold: Where: N is the width of the sliding window.

Images