Potash flotation foam image segmentation method based on multi-scale feature extraction and fusion
Through a deep neural network method based on multi-scale feature extraction and fusion, the problem of low accuracy in potassium flotation foam image segmentation is solved, and more accurate foam edge detection and morphological feature extraction are achieved, which enhances the robustness of the segmentation network sex.
Patent Information
- Application Number
- CN202210779475.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-04
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-07-04
AI Technical Summary
The prior art has problems of low accuracy and weak robustness in the image segmentation of potassium salt flotation foam, especially due to the blurred edges caused by low color contrast between the foam and the background, large morphology changes, and equipment jitter, making it difficult to accurately segment. .
Using a deep neural network method based on multi-scale feature extraction and fusion, a deep neural network of flotation foam image segmentation is constructed by alternately stacking parallel multi-scale feature extraction submodules and multi-scale feature fusion submodules based on attention mechanism, combining semantic segmentation Label and data expansion technology to achieve more accurate foam edge detection.
The segmentation accuracy of potassium salt flotation foam images is improved, and the size and morphological characteristics of the foam can be accurately captured under different operating conditions, reducing image information loss, and enhancing the robustness of the segmentation network.
Smart Images

Figure CN115131561B_ABST
Abstract
Description
Technical Field
[0001] The disclosed embodiments relate to the field of image processing technology, and in particular to a method for segmenting potassium salt flotation foam images based on multi-scale feature extraction and fusion. Background Art
[0002] At present, flotation is an important part of the production and processing of potassium salt. Its purpose is to separate useful minerals from impurities according to the physical and chemical properties of the surface of minerals such as carnallite generated by sun-drying in salt lakes. The useful minerals are selectively attached to the air bubbles in the pulp and then float to the surface of the pulp, so as to achieve the purpose of separating useful minerals from impurities. In the actual production process, useful minerals will form a foam layer with certain visual characteristics after floating to the surface of the pulp with the bubbles. These visual characteristics depend on many actual production factors such as the amount of foam scraped by the current flotation tank scraper, the amount of flotation reagent added, the grade of the mineral, etc., so they can reflect the flotation working conditions at the current moment to a certain extent.
[0003] Among them, flotation foam size characteristics, such as the number of bubbles, average size, etc., are important reference indicators for evaluating the current flotation conditions and flotation ore grade. Most of the existing flotation foam size characteristics rely on the naked eye observation of on-site workers. This operation adjustment based on naked eye observation has certain limitations: first, the operator cannot monitor the changes in the foam at all times, which will lead to delayed operation adjustments; second, accurate judgment of operation adjustments requires long-term experience accumulation, which requires high operator ability; third, the worker's observation and judgment are highly subjective, qualitative rather than quantitative, and there is no unified evaluation standard, which leads to randomness and variability in production process control, which may lead to fluctuations in flotation conditions.
[0004] Applying computer vision technology and image segmentation methods as the basis, each bubble is separated from the collected potassium salt flotation tank foam image, and then further quantitative and accurate size feature parameter extraction is performed, which plays a key role in the subsequent flotation process operation optimization and other applications. Traditional image segmentation methods, such as threshold segmentation method, region growing method, and watershed algorithm, which is the most widely used in flotation foam segmentation, mostly rely on the key step of finding a suitable threshold. However, potassium salt flotation foam image segmentation faces the following difficulties: First, the color contrast between the foam and the foam in the potassium salt flotation tank, and between the foam and the background pulp is low, resulting in blurred foam edges; second, due to the fluctuation of flotation conditions, there are large differences in foam morphology, size and other information at different times; third, the rising air and water mist on the surface of the foam layer, the camera shake caused by the operation of the on-site equipment, and the movement of the foam itself, further aggravate the blur of the collected image and weaken the edge of the foam. The above problems make the threshold selection extremely difficult for potassium salt foam images, which also leads to low accuracy and weak robustness of traditional methods. At present, deep learning is rarely used in flotation foam image segmentation, and most of them are based on the U-Net network. Although these methods or some other common deep learning segmentation methods have achieved excellent segmentation effects in some fields, they have not been combined with the actual characteristics of potassium salt flotation foam images for network design, and directly applied to potassium salt flotation foam image segmentation, the effect is not good.
[0005] It can be seen that there is an urgent need for a potassium salt flotation foam image segmentation method based on multi-scale feature extraction and fusion that can accurately segment potassium salt flotation foam images. Summary of the invention
[0006] In view of this, the embodiments of the present disclosure provide a potassium salt flotation foam image segmentation method based on multi-scale feature extraction and fusion, which at least partially solves the problem of poor segmentation accuracy in the prior art.
[0007] The present disclosure provides a method for segmenting potassium salt flotation foam images based on multi-scale feature extraction and fusion, including:
[0008] Step 1, collecting foam images of the potash roughing process, marking multiple pixel points at the edges of each foam in the foam image to obtain the foam area, and then obtaining a semantic segmentation label image, constructing an original data set from the foam image and the semantic segmentation label image, and expanding the samples of the original data set to divide it into a training set and a test set;
[0009] Step 2, constructing a flotation foam image segmentation deep neural network model, wherein the flotation foam image segmentation deep neural network model includes a segmentation network backbone consisting of alternately stacked parallel multi-scale feature extraction submodules and a multi-scale feature fusion submodule based on an attention mechanism, an input adjustment module, and an output adjustment module;
[0010] Step 3, using the training set as input, training the flotation foam image segmentation deep neural network model;
[0011] Step 4: Obtain the flotation foam image to be segmented and crop it to a specified size, and then input it into the trained flotation foam image segmentation deep neural network model to obtain the segmentation result.
[0012] According to a specific implementation of the embodiment of the present disclosure, step 1 specifically includes:
[0013] Step 1.1, collect the froth image of the potash rougher flotation cell and crop it to 730×730 pixels;
[0014] Step 1.2, using image annotation software to make fine segmentation labels for the image obtained in step 1.1, by annotating several pixel points on the edge of each bubble in the image and connecting them into a closed figure, obtaining the area where the bubble is located, and then obtaining a semantic segmentation label image, and constructing an original data set from the bubble image and the semantic segmentation label image;
[0015] Step 1.3, expand the original data set by means of translation, rotation, noise addition, affine transformation, etc., crop the expanded data set, and obtain an image with a size of 512×512 pixels in the central area, which is divided into a training set and a test set.
[0016] According to a specific implementation of the embodiment of the present disclosure, step 2 specifically includes:
[0017] Step 2.1, construct a dual convolution submodule with residual connection;
[0018] Step 2.2, construct a parallel multi-scale feature extraction submodule, including multiple scale feature extraction branches, each scale feature extraction branch is composed of stacked double convolution submodules, the input and output feature map size and number of channels of a single scale feature extraction branch are the same, and the input feature maps of adjacent scale feature extraction branches are in a relationship of halving the size and doubling the number of channels;
[0019] Step 2.3, construct a multi-scale feature fusion submodule based on the attention mechanism, take multiple scale image feature maps as input, adjust the input to the specified scale and perform weighted summation to obtain a single output feature map of the specified scale;
[0020] Step 2.4, alternately stacking multiple parallel multi-scale feature extraction sub-modules and multi-scale feature fusion sub-modules based on attention mechanism;
[0021] Step 2.5, make preliminary adjustments to the original input image, input it into the segmentation network backbone, and obtain the segmentation network backbone output;
[0022] In step 2.6, after upsampling the segmentation network backbone output of step 2.5, the features are further extracted through the dual convolution submodule and the number of feature map channels is transformed through 1×1 convolution to obtain the output of the flotation foam image segmentation deep neural network model.
[0023] According to a specific implementation of the embodiment of the present disclosure, the double convolution submodule of step 2.1 is performed by the following formula:
[0024] Y=ReLU(BN(W2(ReLU(BN(W1X)))))+X)
[0025] Among them, X and Y represent the input and output feature maps respectively, W1 and W2 represent the convolution kernel weights of the first and second 3×3 convolution respectively, BN represents batch normalization, and ReLU represents the pixel-by-pixel execution of the ReLU activation function. The expression is
[0026] According to a specific implementation of an embodiment of the present disclosure, in step 2.2, the parallel multi-scale feature extraction submodule has a jump connection from the input feature map to the output feature map on the branch with the largest image scale.
[0027] According to a specific implementation of the embodiment of the present disclosure, step 2.3 specifically includes:
[0028] Step 2.3.1: Adjust the input feature maps of each scale to the target scale, and the adjusted feature map is recorded as X' i ,i=1…K, K is the number of input feature maps of different scales;
[0029] Step 2.3.2, perform maximum pooling and average pooling operations in the channel dimension on the feature maps of each scale adjusted in step 2.3.1 to obtain a feature map of 2×H×W dimensions, where H and W are the height and width of the feature map, respectively. This is followed by a dilated convolution of a single convolution kernel to expand the receptive field and further fuse information to obtain a 1×H×W feature map representing single-scale spatial information;
[0030] Step 2.3.3: Concatenate the K single-channel feature maps extracted in step 2.3.2 in the channel dimension, and then perform a 1×1 convolution to obtain a multi-scale attention feature map W. i ,i=1...K;
[0031] Step 2.3.4: Adjust the feature map X' after step 2.3.1 i , i=1...K through the multi-scale attention feature map W proposed in step 2.3.3 i , i=1...K perform weighted summation to obtain the output feature map Y, the formula is as follows
[0032]
[0033] Where the superscript j represents the jth channel of the feature map, ⊙ represents the Hadamard product, and C represents the number of channels of the output feature map.
[0034] According to a specific implementation of the embodiment of the present disclosure, step 2.4 specifically includes:
[0035] Step 2.4.1, for the Sth stacking, the feature extraction submodule of this stage contains S+1 branches, followed by S+1 feature fusion submodules. Each feature fusion submodule receives the output of all S+1 branches of the feature extraction submodule as input, and obtains outputs of S+1 scales that are the same as the input. For the last stacking, it contains 1 feature fusion submodule, the input is the output of all branches of the feature extraction submodule, and the output scale is the largest input feature map scale;
[0036] In step 2.4.2, each time stacking is performed, a branch is added to the feature extraction submodule of the next stage. The input of this branch is obtained by convolution and downsampling the output of the minimum-scale branch of the feature extraction submodule of the current stage.
[0037] According to a specific implementation of the embodiment of the present disclosure, step 2.5 specifically includes:
[0038] Step 2.5.1, preliminary feature extraction is performed through the double convolution submodule;
[0039] Step 2.5.2, a 3×3 convolution with a stride of 2 is performed to halve the feature map size and double the number of channels, thus reducing the overall network parameters.
[0040] Step 2.5.3: Apply a 3×3 convolution with a step size of 1 to the output feature map of step 2.5.2. The feature map size is not changed, and the number of feature map channels is adjusted to the reference channel number C. base , and, connect the output feature map of step 2.5.2 to a 3×3 convolution with a stride of 2 to halve the feature map size, adjust the number of channels to twice the number of reference channels, and obtain the input feature maps of the two branches of the segmentation network trunk;
[0041] Step 2.5.4, input the two feature maps of different scales adjusted in step 2.5.3 into the segmentation network backbone to obtain the segmentation network backbone output.
[0042] According to a specific implementation of the embodiment of the present disclosure, step 3 specifically includes:
[0043] Taking the training set as input, the flotation foam image segmentation deep neural network model is stacked with feature extraction and feature fusion submodules three times. The number of benchmark channels C base is 64, and trained as follows:
[0044] Using the Dice Loss loss function, the expression is as follows:
[0045]
[0046] Among them, P represents the predicted label, R represents the true label, |P∩R| represents the sum of the corresponding elements after dot multiplication, |·| represents the sum of each element, and the Adam optimizer is used to update the training weights. The learning rate is adjusted according to the following formula:
[0047]
[0048] Among them, lr(epoch) represents the learning rate of iteration to epoch, Indicates rounding down, n max Indicates the maximum number of iterations.
[0049] The potassium salt flotation foam image segmentation scheme based on multi-scale feature extraction and fusion in the embodiment of the present disclosure includes: step 1, collecting foam images of the potassium salt rough selection process, marking multiple pixel points of each foam edge in the foam image to obtain the foam area, and then obtaining a semantic segmentation label image, constructing an original data set by the foam image and the semantic segmentation label image, and expanding the samples of the original data set to divide it into a training set and a test set; step 2, constructing a flotation foam image segmentation deep neural network model, wherein the flotation foam image segmentation deep neural network model includes a segmentation network backbone composed of alternately stacked parallel multi-scale feature extraction submodules and a multi-scale feature fusion submodule based on an attention mechanism, an input adjustment module, and an output adjustment module; step 3, using the training set as input, training the flotation foam image segmentation deep neural network model; step 4, obtaining the flotation foam image to be segmented and cropping it to a specified size, and then inputting it into the trained flotation foam image segmentation deep neural network model to obtain a segmentation result.
[0050] The beneficial effects of the embodiments of the present disclosure are as follows: (1) the present invention combines the multi-scale and weak-edge morphological features in the potassium salt flotation foam image to construct a foam segmentation deep network, and achieves the foam image segmentation more accurately when the segmentation label preparation is difficult and the label data set is small;
[0051] (2) The present invention designs a parallel multi-scale feature extraction submodule composed of stacked residual blocks. During the feature extraction process, the feature map size is gradually halved and the number of channels is doubled, so that the network's receptive field is gradually expanded to better capture the larger bubble edges. While downsampling for feature extraction, feature extraction on the high-resolution feature map is still synchronized to avoid the loss of small bubble edge information. By extracting features at multiple scales in parallel, the network can obtain a larger receptive field while better reducing the loss of image information, taking into account the learning of bubble features of different scales;
[0052] (3) The present invention aims at extracting multiple scale feature maps. Considering that there are certain differences in the spatial information concerned at different scales, the feature maps with larger scales contain more detailed information, which is more conducive to identifying small objects. Conversely, the feature maps with smaller scales can better identify large objects. A multi-scale feature fusion submodule based on the attention mechanism is designed to allow the network to learn the spatial information concerned at each scale and assign different weights and then perform weighted summation to achieve feature fusion and complementarity. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0054] Figure 1 A schematic flow chart of a potassium salt flotation foam image segmentation method based on multi-scale feature extraction and fusion provided in an embodiment of the present disclosure;
[0055] Figure 2 A schematic flow chart of another method for segmenting potassium salt flotation foam images based on multi-scale feature extraction and fusion provided in an embodiment of the present disclosure;
[0056] Figure 3 A schematic diagram of the structure of a deep neural network model for flotation foam image segmentation provided in an embodiment of the present disclosure;
[0057] Figure 4 A structural diagram of a multi-scale feature fusion submodule based on an attention mechanism provided in an embodiment of the present disclosure;
[0058] Figure 5 A comparison chart of the segmentation effects of a potassium salt flotation foam image segmentation method based on multi-scale feature extraction and fusion and various methods provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0059] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.
[0060] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.
[0061] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.
[0062] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.
[0063] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.
[0064] The disclosed embodiment provides a potassium salt flotation foam image segmentation method based on multi-scale feature extraction and fusion, and the method can be applied to the potassium salt flotation foam image analysis process in the potassium salt production industrial scene.
[0065] See also Figure 1 , is a flow chart of a potassium salt flotation foam image segmentation method based on multi-scale feature extraction and fusion provided by an embodiment of the present disclosure. Figure 1 and Figure 2As shown, the method mainly comprises the following steps:
[0066] Step 1, collecting foam images of the potash roughing process, marking multiple pixel points at the edges of each foam in the foam image to obtain the foam area, and then obtaining a semantic segmentation label image, constructing an original data set from the foam image and the semantic segmentation label image, and expanding the samples of the original data set to divide it into a training set and a test set;
[0067] Optionally, the step 1 specifically includes:
[0068] Step 1.1, collect the froth image of the potash rougher flotation cell and crop it to 730×730 pixels;
[0069] Step 1.2, using image annotation software to make fine segmentation labels for the image obtained in step 1.1, by annotating several pixel points on the edge of each bubble in the image and connecting them into a closed figure, obtaining the area where the bubble is located, and then obtaining a semantic segmentation label image, and constructing an original data set from the bubble image and the label image;
[0070] Step 1.3, expand the original data set by means of translation, rotation, noise addition, affine transformation, etc., crop the expanded data set, and obtain an image with a size of 512×512 pixels in the central area, which is divided into a training set and a test set.
[0071] In the specific implementation, data preparation and data preprocessing can be performed first, including collecting foam images of the potash roughing process, obtaining the foam area by marking several pixel points at the edges of each foam in the image, and then obtaining semantic segmentation labels, and constructing the original data set from the foam image and the label image; expanding the original data set samples by image processing means, and dividing them into training sets and test sets, including:
[0072] The foam images of the potash rougher flotation tank were collected by an industrial camera and cropped to a size of 730 × 730 pixels;
[0073] The obtained image is finely segmented and labeled by image annotation software. The foam area is obtained by annotating several pixel points on the edge of each foam in the image and connecting them into a closed figure, and then a semantic segmentation label image is obtained. The original image segmentation data set is constructed from the foam image and the semantic segmentation label image.
[0074] The original data set is expanded with the help of translation, rotation, noise addition, affine transformation and other means, so that the subsequent segmentation network can fully learn the position, area and brightness differences of the foam under different working conditions. The expanded data set is cropped to obtain images with a size of 512×512 pixels in the central area, which are divided into training and test sets.
[0075] Step 2, constructing a flotation foam image segmentation deep neural network model, wherein the flotation foam image segmentation deep neural network model includes a segmentation network backbone consisting of alternately stacked parallel multi-scale feature extraction submodules and a multi-scale feature fusion submodule based on an attention mechanism, an input adjustment module, and an output adjustment module;
[0076] Furthermore, the step 2 specifically includes:
[0077] Step 2.1, construct a dual convolution submodule with residual connection;
[0078] Step 2.2, construct a parallel multi-scale feature extraction submodule, including multiple scale feature extraction branches, each scale feature extraction branch is composed of stacked double convolution submodules, the input and output feature map size and number of channels of a single scale feature extraction branch are the same, and the input feature maps of adjacent scale feature extraction branches are in a relationship of halving the size and doubling the number of channels;
[0079] Step 2.3, construct a multi-scale feature fusion submodule based on the attention mechanism, take multiple scale image feature maps as input, adjust the input to the specified scale and perform weighted summation to obtain a single output feature map of the specified scale;
[0080] Step 2.4, alternately stacking multiple parallel multi-scale feature extraction sub-modules and multi-scale feature fusion sub-modules based on attention mechanism;
[0081] Step 2.5, make preliminary adjustments to the original input image, input it into the segmentation network backbone, and obtain the segmentation network backbone output;
[0082] In step 2.6, after upsampling the segmentation network backbone output of step 2.5, the features are further extracted through the dual convolution submodule and the number of feature map channels is transformed through 1×1 convolution to obtain the output of the flotation foam image segmentation deep neural network model.
[0083] Optionally, the double convolution submodule of step 2.1 is performed by the following formula:
[0084] Y=ReLU(BN(W2(ReLU(BN(W1X)))))+X)
[0085] Among them, X and Y represent the input and output feature maps respectively, W1 and W2 represent the convolution kernel weights of the first and second 3×3 convolution respectively, BN represents batch normalization, and ReLU represents the pixel-by-pixel execution of the ReLU activation function. The expression is
[0086] Optionally, in step 2.2, the parallel multi-scale feature extraction submodule has a jump connection from the input feature map to the output feature map on the branch with the largest image scale.
[0087] Optionally, the step 2.3 specifically includes:
[0088] Step 2.3.1: Adjust the input feature maps of each scale to the target scale, and the adjusted feature map is recorded as X' i ,i=1...K, K is the number of input feature maps of different scales;
[0089] Step 2.3.2, perform maximum pooling and average pooling operations in the channel dimension on the feature maps of each scale adjusted in step 2.3.1 to obtain a feature map of 2×H×W dimensions, where H and W are the height and width of the feature map, respectively. This is followed by a dilated convolution of a single convolution kernel to expand the receptive field and further fuse information to obtain a 1×H×W feature map representing single-scale spatial information;
[0090] Step 2.3.3: Concatenate the K single-channel feature maps extracted in step 2.3.2 in the channel dimension, and then perform a 1×1 convolution to obtain a multi-scale attention feature map W. i ,i=1...K;
[0091] Step 2.3.4: Adjust the feature map X' after step 2.3.1 i , i=1...K through the multi-scale attention feature map W proposed in step 2.3.3 i , i=1...K perform weighted summation to obtain the output feature map Y, the formula is as follows
[0092]
[0093] Where the superscript j represents the jth channel of the feature map, ⊙ represents the Hadamard product, and C represents the number of channels of the output feature map.
[0094] Optionally, the step 2.4 specifically includes:
[0095] Step 2.4.1, for the Sth stacking, the feature extraction submodule of this stage contains S+1 branches, followed by S+1 feature fusion submodules. Each feature fusion submodule receives the output of all S+1 branches of the feature extraction submodule as input, and obtains outputs of S+1 scales that are the same as the input. For the last stacking, it contains 1 feature fusion submodule, the input is the output of all branches of the feature extraction submodule, and the output scale is the largest input feature map scale;
[0096] In step 2.4.2, each time stacking is performed, a branch is added to the feature extraction submodule of the next stage. The input of this branch is obtained by convolution and downsampling the output of the minimum-scale branch of the feature extraction submodule of the current stage.
[0097] Optionally, the step 2.5 specifically includes:
[0098] Step 2.5.1, preliminary feature extraction is performed through the double convolution submodule;
[0099] Step 2.5.2, a 3×3 convolution with a stride of 2 is performed to halve the feature map size and double the number of channels, thus reducing the overall network parameters.
[0100] Step 2.5.3: Apply a 3×3 convolution with a step size of 1 to the output feature map of step 2.5.2. The feature map size is not changed, and the number of feature map channels is adjusted to the reference channel number C. base , and, connect the output feature map of step 2.5.2 to a 3×3 convolution with a stride of 2 to halve the feature map size, adjust the number of channels to twice the number of reference channels, and obtain the input feature maps of the two branches of the segmentation network trunk;
[0101] Step 2.5.4, input the two feature maps of different scales adjusted in step 2.5.3 into the segmentation network backbone to obtain the segmentation network backbone output.
[0102] In the specific implementation, a deep neural network for flotation foam image segmentation is constructed, such as Figure 3 As shown in the figure, the flotation foam image segmentation deep neural network model includes three components connected in sequence: an input adjustment module, a segmentation network backbone composed of alternately stacked parallel multi-scale feature extraction sub-modules and a multi-scale feature fusion sub-module based on an attention mechanism, and an output adjustment module; the input adjustment module receives a foam image of 512×512 pixels, performs preliminary feature extraction through a double convolution sub-module with residual connection, and obtains feature maps of different scales input to the segmentation network backbone through a downsampling operation; the parallel multi-scale feature extraction sub-module includes multiple branches composed of stacked double convolution sub-modules, different branches receive feature maps of different sizes as input, and extract features synchronously at multiple image scales, and introduces a jump connection from input to output on the branch with the largest image scale;
[0103] like Figure 4 As shown in the figure, the multi-scale feature fusion submodule based on the attention mechanism takes the output of each branch of the parallel multi-scale feature extraction submodule as input, resizes the proposed feature maps of different scales, and automatically learns a weight matrix for each adjusted feature map for weighted summation, so that the spatial information of different scales can be fused and complemented; the output adjustment module upsamples the output feature map of the network backbone to the size of the network input image, further extracts features at this scale through the double convolution submodule, and finally obtains the output of the network through a 1×1 convolution transformation feature map channel number, which specifically includes:
[0104] Step 21, construct a double convolution submodule with residual connection. By introducing residual connection, the gradient descent and gradient vanishing problems that may be caused by deepening the number of segmentation network layers are avoided. The double convolution submodule is performed by the following formula:
[0105] Y=ReLU(BN(W2(ReLU(BN(W1X)))))+X)
[0106] Where X and Y represent the input and output feature maps, respectively. W1 and W2 represent the convolution kernel weights of the first and second 3×3 convolutions, respectively. BN represents batch normalization. ReLU represents the pixel-by-pixel execution of the ReLU activation function. The expression is:
[0107] Step 22, construct a parallel multi-scale feature extraction submodule, which includes multiple scale feature extraction branches. Each branch is composed of stacked dual convolution submodules. The input and output feature map sizes and number of channels of a single branch are the same, and the input feature maps of adjacent branches have a relationship of halving the size and doubling the number of channels. On the branch with the largest image scale in a single submodule, there is a jump connection from the input feature map to the output feature map.
[0108] Step 23, construct a multi-scale feature fusion submodule based on the attention mechanism, take multiple scale image feature maps as input, adjust the input to a specified scale and perform weighted summation to obtain a single output feature map of a specified scale, specifically including:
[0109] Step 231, adjust each input scale feature map to the target scale. There are three cases: (1) the input feature map size is the same as the target feature map size and no adjustment is required; (2) the input feature map size is twice the target feature map size. n , n = 1, 2, 3, ... times, and repeat the following operation n times on the input feature map: first, double the number of feature map channels through 1 × 1 convolution, and then halve the size of the feature map through pooling operation; (3) the size of the input feature map is the size of the target feature map The following operation is repeated n times on the input feature map: first, the number of feature map channels is halved by 1×1 convolution, and then bilinear interpolation is performed to double the size of the feature map. The adjusted feature map is recorded as X' i ,i=1...K, K is the number of input feature maps of different scales;
[0110] Step 232, perform channel-dimensional maximum pooling and average pooling operations on the feature maps of each scale adjusted in step 231 to obtain a feature map of 2×H×W dimensions, where H and W are the height and width of the feature map, respectively. This is followed by a dilated convolution of a single convolution kernel to expand the receptive field and further fuse information to obtain a 1×H×W feature map representing single-scale spatial information;
[0111] Step 233, concatenate the K single-channel feature maps extracted in step 232 in the channel dimension, and then perform a 1×1 convolution to obtain a multi-scale attention feature map W i ,i=1...K;
[0112] Step 234: The feature map X' adjusted in step 231 i , i=1...K through the multi-scale attention feature map W proposed in step 233 i , i=1...K perform weighted summation to obtain the output feature map Y, the formula is as follows
[0113]
[0114] Where the superscript j represents the jth channel of the feature map, ⊙ represents the Hadamard product, and C represents the number of channels of the output feature map.
[0115] Step 24, alternately stack multiple parallel multi-scale feature extraction submodules and multi-scale feature fusion submodules based on the attention mechanism. The stacking method is as follows: for the Sth stacking, the feature extraction submodule of this stage contains S+1 branches, followed by S+1 feature fusion submodules, each feature fusion submodule receives the output of all S+1 branches of the feature extraction submodule as input, and obtains outputs of S+1 scales that are the same as the input. For the last stacking, only one feature fusion submodule is included, and the input is still the output of all branches of the feature extraction submodule, and the output scale is the largest input feature map scale. Each time stacking is performed, a branch is added to the feature extraction submodule of the next stage, and the input of the branch is obtained by convolution and downsampling of the minimum scale branch output of the feature extraction submodule of the current stage.
[0116] Step 25: make preliminary adjustments to the original input image. First, perform preliminary feature extraction through the double convolution submodule, followed by a 3×3 convolution with a step size of 2 to halve the feature map size and double the number of channels, thereby reducing the overall network parameters. Different operations are then performed to generate the input feature maps of the two branches of the segmentation network trunk: one is a 3×3 convolution with a step size of 1, which does not change the feature map size and adjusts the number of feature map channels to the baseline channel number C. base , and secondly, a 3×3 convolution with a stride of 2 is performed to halve the feature map size and adjust the number of channels to twice the number of baseline channels. The two adjusted feature maps of different scales are input into the segmentation network backbone composed of stacked feature extraction and feature fusion submodules to obtain the segmentation network backbone output;
[0117] Step 26, up-sample the network backbone output described in step 25, and then further extract features through a dual convolution submodule and transform the number of feature map channels through a 1×1 convolution to obtain a segmentation network output.
[0118] Step 3, using the training set as input, training the flotation foam image segmentation deep neural network model;
[0119] Based on the above embodiment, step 3 specifically includes:
[0120] Taking the training set as input, the flotation foam image segmentation deep neural network model is stacked with feature extraction and feature fusion submodules three times. The number of benchmark channels C base is 64, and trained as follows:
[0121] Using the Dice Loss loss function, the expression is as follows:
[0122]
[0123] Among them, P represents the predicted label, R represents the true label, |P∩R| represents the sum of the corresponding elements after dot multiplication, |·| represents the sum of each element, and the Adam optimizer is used to update the training weights. The learning rate is adjusted according to the following formula:
[0124]
[0125] Among them, lr(epoch) represents the learning rate of iteration to epoch, Indicates rounding down, n max Indicates the maximum number of iterations.
[0126] In the specific implementation, the image segmentation network is trained, and the training set described in step 1 is used as input. The feature extraction and feature fusion submodules are stacked three times, and the number of reference channels C base The input image is trained with a segmentation network model of 64; the input image is preliminarily extracted with features by the input adjustment module and downsampled to 256×256 pixel size, and then convolution operations with a step size of 1, a convolution kernel number of 64, and a step size of 2, and a convolution kernel number of 128 are performed to obtain feature maps of two scales as the input of the segmentation network trunk; after the segmentation network trunk receives the input of the feature maps of the two scales, features are extracted synchronously on the two branches of the feature extraction submodule, and then the weighted summation of features of different scales is completed in the feature fusion submodule. At the same time, the output of the feature extraction submodule will be downsampled to produce another scale feature map as the input of the subsequent feature extraction submodule; the feature extraction and feature fusion processes are repeated to make the spatial information of different scales fused and complementary, and finally the segmentation network trunk output of 256×256 pixel size is obtained, which is upsampled to 512×512 pixel size by the output adjustment module and the number of channels is changed to obtain the model prediction label; then, the Dice Loss function is used to calculate the difference between the predicted label and the true label, and the expression is as follows:
[0127]
[0128] Among them, P represents the predicted label, R represents the true label, |P∩R| represents the sum of the corresponding elements after dot multiplication, |·| represents the sum of each element, and the Adam optimizer is used to update the training weights so that the Dice loss calculated by the segmentation network output corresponding to the updated weights gradually decreases; the update speed of the model weights is controlled by adjusting the learning rate. In the early stage of model training, a larger learning rate is used to quickly update the weights and accelerate the convergence speed of the model. Then the learning rate is gradually reduced to prevent the model from oscillating near the optimal weight and optimize the accuracy of the model. The learning rate adjustment formula is as follows:
[0129]
[0130] Among them, lr(epoch) represents the learning rate of iteration to epoch, Indicates rounding down, n max Indicates the maximum number of iterations.
[0131] Step 4: Obtain the flotation foam image to be segmented and crop it to a specified size, and then input it into the trained flotation foam image segmentation deep neural network model to obtain the segmentation result.
[0132] In specific implementation, after the flotation foam image segmentation deep neural network model is trained, the flotation foam image to be segmented can be obtained and input into an industrial computer. The industrial computer crops the input image to a specified size and then inputs it into the trained segmentation network to obtain the segmentation network output as the segmentation result.
[0133] The present embodiment provides a method for segmenting a potassium salt flotation foam image based on multi-scale feature extraction and fusion. By combining the multi-scale and weak-edge morphological features in the potassium salt flotation foam image, a foam segmentation deep network is constructed. In the case of a small number of label data sets due to the difficulty in making segmentation labels, the foam image segmentation is achieved more accurately. A parallel multi-scale feature extraction submodule consisting of stacked residual blocks is designed. During the feature extraction process, the feature map size is gradually halved and the number of channels is doubled, so that the network's receptive field is gradually expanded, and larger foam edges are better captured. While downsampling for feature extraction, feature extraction on the high-resolution feature map is still performed synchronously to avoid the loss of small foam edge information. By extracting features at multiple scales in parallel, the network can achieve a larger receptive field while reducing the loss of image information, taking into account the learning of foam features at different scales; for the extracted feature maps of multiple scales, considering that there are certain differences in the spatial information concerned at different scales, the feature maps with larger scales contain more detailed information, which is more beneficial for identifying small objects. Conversely, the feature maps with smaller scales can better identify large objects. A multi-scale feature fusion submodule based on the attention mechanism is designed, which allows the network to learn the spatial information concerned at each scale and assign different weights and then perform weighted summation to achieve feature fusion and complementarity, thereby improving the segmentation accuracy of potassium salt flotation foam images.
[0134] In order to further illustrate the beneficial effects of the method of the present disclosure, Figure 5 The comparison of the segmentation effects of different methods for foam images with a size of 512×512 pixels is shown, where (a) and (b) are the original foam images and the manually annotated true labels, respectively, and (c) to (f) are the foam segmentation effects of SegNet, U-Net, DeepLab-v3+ and the method proposed in the present invention, respectively. It can be seen from the figure that the method proposed in the present invention can segment foams of different sizes more accurately, and at the same time can more accurately identify the edges of foams in tightly adhered foam sheets, which is better than other methods. In order to quantitatively evaluate the excellence of different methods, the present invention uses the direct application of foam image segmentation, i.e., size feature extraction, and uses the number of bubbles extracted from the segmentation results and the relative error Er and absolute error Ea of the average bubble size between the true label to judge the quality of the segmentation algorithm.
[0135] in E a =mean gt -mean pre ,mean gt 、mean preRepresent the average bubble size (in pixels) of the true label and the output result of the segmentation algorithm respectively. The closer the number of bubbles is to the true label and the smaller the absolute value of the error of the average bubble size is, the more accurate the proposed size feature is, that is, the segmentation network has better learned the characteristics of the foam image. Table 1 shows the evaluation results of the above four segmentation methods. The experimental results show that the number of bubbles predicted by the method proposed in the present invention (140) is closest to the actual number of bubbles (158), the absolute value of the relative error of the average bubble size is 2.997%, and the absolute value of the absolute error is 36.325 pixels, which is much smaller than other methods. It can be seen that the method proposed in the present invention can segment the foam image more accurately.
[0136]
[0137] Table 1
[0138] In summary, the potassium salt flotation foam image segmentation method based on multi-scale feature extraction and fusion has the following advantages:
[0139] (1) The present invention combines the multi-scale and weak-edge morphological features in the potassium salt flotation foam image to construct a foam segmentation deep network, and achieves the segmentation of the foam image more accurately when the segmentation label preparation is difficult and the label data set is small;
[0140] (2) The present invention designs a parallel multi-scale feature extraction submodule composed of stacked residual blocks. During the feature extraction process, the feature map size is gradually halved and the number of channels is doubled, so that the network's receptive field is gradually expanded to better capture larger bubble edges. While downsampling for feature extraction, feature extraction on high-resolution feature maps is still carried out synchronously to avoid the loss of small bubble edge information. By extracting features at multiple scales in parallel, the network can achieve a larger receptive field while better reducing the loss of image information, taking into account the learning of bubble features of different scales;
[0141] (3) The present invention aims at extracting multiple scale feature maps. Considering that there are certain differences in the spatial information concerned at different scales, the feature maps with larger scales contain more detailed information, which is more conducive to identifying small objects. Conversely, the feature maps with smaller scales can better identify large objects. A multi-scale feature fusion submodule based on the attention mechanism is designed to allow the network to learn the spatial information concerned at each scale and assign different weights and then perform weighted summation to achieve feature fusion and complementarity.
[0142] The above-mentioned embodiment of the present invention describes a potassium salt flotation foam image segmentation method based on multi-scale feature extraction and fusion, which constructs a segmentation network according to the foam morphological characteristics of the potassium salt flotation foam image, accurately segments the foam image, and lays a good foundation for the subsequent extraction of size characteristic parameters. Furthermore, it can provide important status information for applications such as working condition identification, grade prediction, and operation optimization related to potassium salt flotation.
[0143] It should be understood that various parts of the present disclosure may be implemented in hardware, software, firmware or a combination thereof.
[0144] The above is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.
Claims
1. A method for segmenting potassium salt flotation foam images based on multi-scale feature extraction and fusion, characterized in that: include: Step 1, collecting foam images of the potash roughing process, marking multiple pixel points at the edges of each foam in the foam image to obtain the foam area, and then obtaining a semantic segmentation label image, constructing an original data set from the foam image and the semantic segmentation label image, and expanding the samples of the original data set to divide it into a training set and a test set; Step 2, constructing a flotation foam image segmentation deep neural network model, wherein the flotation foam image segmentation deep neural network model includes a segmentation network backbone consisting of alternately stacked parallel multi-scale feature extraction submodules and a multi-scale feature fusion submodule based on an attention mechanism, an input adjustment module, and an output adjustment module; The step 2 specifically includes: Step 2.1, construct a dual convolution submodule with residual connection; Step 2.2, construct a parallel multi-scale feature extraction submodule, including multiple scale feature extraction branches, each scale feature extraction branch is composed of stacked double convolution submodules, the input and output feature map size and number of channels of a single scale feature extraction branch are the same, and the input feature maps of adjacent scale feature extraction branches are in a relationship of halving the size and doubling the number of channels; Step 2.3, construct a multi-scale feature fusion submodule based on the attention mechanism, take multiple scale image feature maps as input, adjust the input to the specified scale and perform weighted summation to obtain a single output feature map of the specified scale; Step 2.4, alternately stacking multiple parallel multi-scale feature extraction sub-modules and multi-scale feature fusion sub-modules based on attention mechanism; Step 2.5, make preliminary adjustments to the original input image, input it into the segmentation network backbone, and obtain the segmentation network backbone output; Step 2.6, after upsampling the segmentation network backbone output of step 2.5, further extract features through the double convolution submodule and transform the number of feature map channels through 1×1 convolution to obtain the flotation foam image segmentation deep neural network model output; The double convolution submodule of step 2.1 is performed by the following formula: Y=ReLU(BN(W2(ReLU(BN(W1X)))))+X) Among them, X and Y represent the input and output feature maps respectively, W1 and W2 represent the convolution kernel weights of the first and second 3×3 convolution respectively, BN represents batch normalization, and ReLU represents the pixel-by-pixel execution of the ReLU activation function. The expression is In step 2.2, the parallel multi-scale feature extraction submodule has a jump connection from the input feature map to the output feature map on the branch with the largest image scale; The step 2.3 specifically includes: Step 2.3.1: Adjust the input feature maps of each scale to the target scale, and the adjusted feature map is recorded as X' i ,i=1...K, K is the number of input feature maps of different scales; Step 2.3.2, perform maximum pooling and average pooling operations in the channel dimension on the feature maps of each scale adjusted in step 2.3.1 to obtain a feature map of 2×H×W dimensions, where H and W are the height and width of the feature map, respectively. This is followed by a dilated convolution of a single convolution kernel to expand the receptive field and further fuse information to obtain a 1×H×W feature map representing single-scale spatial information; Step 2.3.3: Concatenate the K single-channel feature maps extracted in step 2.3.2 in the channel dimension, and then perform a 1×1 convolution to obtain a multi-scale attention feature map W. i ,i=1...K; Step 2.3.4: Adjust the feature map X' after step 2.3.1 i , i=1...K through the multi-scale attention feature map W proposed in step 2.3.3 i , i=1...K perform weighted summation to obtain the output feature map Y, the formula is as follows Where superscript j represents the jth channel of the feature map, ⊙ represents the Hadamard product, and C represents the number of channels of the output feature map; The step 2.4 specifically includes: Step 2.4.1, for the Sth stacking, the feature extraction submodule of this stage contains S+1 branches, followed by S+1 feature fusion submodules. Each feature fusion submodule receives the output of all S+1 branches of the feature extraction submodule as input, and obtains outputs of S+1 scales that are the same as the input. For the last stacking, it contains 1 feature fusion submodule, the input is the output of all branches of the feature extraction submodule, and the output scale is the largest input feature map scale; Step 2.4.2: Each time stacking is performed, a branch is added to the feature extraction submodule of the next stage. The input of this branch is obtained by convolution and downsampling the output of the minimum-scale branch of the feature extraction submodule of the current stage. The step 2.5 specifically includes: Step 2.5.1, preliminary feature extraction is performed through the double convolution submodule; Step 2.5.2, a 3×3 convolution with a stride of 2 is performed to halve the feature map size and double the number of channels, thus reducing the overall network parameters. Step 2.5.3: Apply a 3×3 convolution with a step size of 1 to the output feature map of step 2.5.
2. The feature map size is not changed, and the number of feature map channels is adjusted to the reference channel number C. base , and, connect the output feature map of step 2.5.2 to a 3×3 convolution with a stride of 2 to halve the feature map size, adjust the number of channels to twice the number of reference channels, and obtain the input feature maps of the two branches of the segmentation network trunk; Step 2.5.4, input the two feature maps of different scales adjusted in step 2.5.3 into the segmentation network backbone to obtain the segmentation network backbone output; Step 3, using the training set as input, training the flotation foam image segmentation deep neural network model; Step 4: Obtain the flotation foam image to be segmented and crop it to a specified size, and then input it into the trained flotation foam image segmentation deep neural network model to obtain the segmentation result.
2. The method according to claim 1, characterized in that , the step 1 specifically includes: Step 1.1, collect the froth image of the potash rougher flotation cell and crop it to 730×730 pixels; Step 1.2, using image annotation software to make fine segmentation labels for the image obtained in step 1.1, by annotating several pixel points on the edge of each bubble in the image and connecting them into a closed figure, obtaining the area where the bubble is located, and then obtaining a semantic segmentation label image, and constructing an original data set from the bubble image and the semantic segmentation label image; Step 1.3, expand the original data set by means of translation, rotation, noise addition, and affine transformation, crop the expanded data set, and obtain an image with a size of 512×512 pixels in the central area, which is divided into a training set and a test set.
3. The method according to claim 2, characterized in that , the step 3 specifically includes: Taking the training set as input, the flotation foam image segmentation deep neural network model is stacked with feature extraction and feature fusion submodules three times. The number of benchmark channels C base is 64, and trained as follows: Using the Dice Loss loss function, the expression is as follows: Among them, P represents the predicted label, R represents the true label, |P∩R| represents the sum of the corresponding elements after dot multiplication, |·| represents the sum of each element, and the Adam optimizer is used to update the training weights. The learning rate is adjusted according to the following formula: Among them, lr(epoch) represents the learning rate of iteration to epoch, Indicates rounding down, n max Indicates the maximum number of iterations.
