Flotation froth color analysis method based on multi-spectral-state double-branch depth fusion

Abstract

The invention provides a flotation froth color analysis method based on multispectral dual-branch depth fusion, and relates to the technical field of flotation froth color analysis, comprising: collecting and preprocessing a multispectral image of froth; the method comprises the following steps: inputting a preprocessed multi-spectral image # imgabs0 # into a multi-spectral double-branch deep fusion network, firstly performing multi-spectral shallow feature extraction on # imgabs1 #, and obtaining a multi-spectral shallow feature matrix F1 reflecting foam space information and spectral information; the F1 is input in parallel into two branches aiming at foam spectral features and foam spatial features, and a spectral feature matrix FM and a foam spatial feature matrix Fs are obtained; fM and Fs are input into a multispectral feature fusion module for intelligent weighted fusion, and a fusion feature matrix F of comprehensive foam spectral features and spatial features is obtained; and inputting the F into a foam color characterization module, and outputting to obtain foam color state prediction. The method can be used for predicting and analyzing the foam color.

Description

Technical Field

[0001] The present invention relates to the technical field of flotation foam color analysis, and in particular to a flotation foam color analysis method based on multi-spectral double-branch deep fusion. Background Art

[0002] In the mining industry, froth flotation is an important ore separation technology. It forms foam by mixing slurry containing metal ore with a series of reagents, so that the metal minerals are adsorbed on the surface of the foam, thereby achieving effective separation of metal minerals and other impurities. In the froth flotation process, by analyzing the color of the flotation foam, it is helpful to understand the foam state in the flotation pool at the current moment and align it with the reagent usage at this moment. Then, the amount of reagent added can be adjusted according to the color situation, the use of reagents can be optimized, and the recovery rate of metal minerals can be improved. Therefore, in the process of metal mineral froth flotation, the color analysis of the foam is very important.

[0003] At present, the color analysis of flotation foam is mainly carried out by combining RGB images with image processing technology. However, this method has some significant limitations. First, there are certain differences between individuals of the CMOS digital imaging unit of the RGB camera, and the same CMOS digital imaging unit will also age with the increase of the number of uses and the use time, resulting in the inaccuracy of the sensing part of the RGB camera. When the factors such as light intensity, color temperature, illumination, and ISO of the camera change, the same foam will show different colors when photographed, resulting in a difference between the collected color and the actual color, which is called the collected color difference; secondly, when the color of the RGB image is manually annotated, the display cannot fully reflect the color information of the flotation foam captured by the CMOS sensor, resulting in a difference between the displayed color and the actual color, which is called the displayed color difference. Due to the influence of the two color differences, the color accuracy of the RGB image is lost, resulting in its inability to accurately reflect the actual color characteristics of the flotation foam. Therefore, how to overcome the color difference problem, improve color accuracy and design a more accurate color recognition method has become a key task in the color analysis of metal ore flotation foam. Summary of the invention

[0004] In order to solve the technical problems existing in the above-mentioned prior art, the present invention provides a flotation foam color analysis method based on multi-spectral double-branch deep fusion, and the technical solution is as follows:

[0005] On the one hand, a flotation foam color analysis method based on multi-spectral dual-branch deep fusion is provided, the method comprising:

[0006] S1. Using a multispectral imaging device to collect a multispectral image of the foam to be color analyzed in the flotation working area;

[0007] S2, preprocessing the multi-spectral image;

[0008] S3, the preprocessed multi-spectral image Input the trained multi-spectral dual-branch deep fusion network, and output the color state prediction of the foam;

[0009] The multi-spectral dual-branch deep fusion network first processes the pre-processed multi-spectral image Perform multi-spectral shallow feature extraction to obtain a multi-spectral shallow feature matrix F1 that reflects the foam spatial information and spectral information;

[0010] The multi-spectral shallow feature matrix F1 is input in parallel into two branches respectively for the foam spectral feature and the spatial feature:

[0011] In the first branch, the multi-spectral shallow feature matrix F1 is input into the channel contraction segmentation module to obtain a spectrum segmentation feature matrix F3 with rich spectral information, and the spectrum segmentation feature matrix F3 is input into the foam spectrum feature extraction module to deeply extract the spectral features of the flotation foam, and output the foam spectrum feature matrix F3 which can fully reflect the spectral characteristics of the flotation foam. M ;

[0012] In the second branch, the multi-spectral shallow feature matrix F1 is passed through the spatial token generator to obtain a spatial feature token set, and the spatial feature token set is input into the foam spatial feature extraction module to deeply extract the spatial features of the flotation foam, and output the foam spatial feature matrix F that reflects the fine spatial information in the flotation foam multi-spectral image. s ;

[0013] The foam spectrum feature matrix F M and the foam space characteristic matrix F s Input the multi-spectral feature fusion module for intelligent weighted fusion to obtain the fusion feature matrix F of the comprehensive foam spectral features and spatial features;

[0014] The fused feature matrix F is input into the foam color characterization module, and the foam color state prediction is output.

[0015] Optionally, the step S2 of preprocessing the multi-spectral image specifically includes:

[0016] The edge filling technology is used to enhance the image boundary information, so as to effectively preserve the integrity of the foam in the spatial dimension and the richness in the spectral dimension;

[0017] In view of the high dimensionality and noise independence characteristics of multi-spectral image information, principal component analysis is used for dimensionality reduction and denoising.

[0018] Optionally, the multi-spectral dual-branch deep fusion network processes the pre-processed multi-spectral image Perform multi-spectral shallow feature extraction to obtain a multi-spectral shallow feature matrix F1 reflecting the foam spatial information and spectral information, specifically including:

[0019] The preprocessed multispectral image The input is sent to a module including a spectral three-dimensional convolution layer, a batch normalization layer, and a ReLU nonlinear activation function layer to extract multi-spectral shallow features, and obtain the multi-spectral shallow feature matrix F1 reflecting the foam space information and spectral information. This process is described by the following formula:

[0020]

[0021] in, represents the nonlinear activation function ReLU, represents batch normalization, represents the convolution operator, and They respectively represent the weight and bias of the three-dimensional convolution, where the three dimensions of the three-dimensional convolution include the width, height, and spectral dimensions of the image.

[0022] Optionally, the channel contraction segmentation module is composed of a channel contraction module and a segmentation module. The channel contraction module includes a two-dimensional convolution layer, a batch normalization layer and a ReLU nonlinear activation function layer, the purpose of which is to compress redundant channel information while strengthening the spectral information expression of the foam, and generate a spectral channel contraction feature matrix F2. After that, the spectral channel contraction feature matrix F2 is input into the segmentation module for token division. The token area after division is flattened into a one-dimensional vector by a flattening function to obtain the spectral segmentation feature matrix F3 with rich spectral information. This process is described by the following formula:

[0023]

[0024]

[0025] in, represents the p×p token region from the position (j,i) to the position (j+p,i+p) of the input foam multispectral image, and Flatten(·) represents the flattening function, which flattens these token regions into a one-dimensional vector.

[0026] Optionally, the foam spectrum feature extraction module includes a layer normalization layer, a spectrum multi-head self-attention layer, a layer normalization layer, and a foam spectrum feature aggregation layer, and residual connections are introduced before and after the spectrum multi-head self-attention layer and the foam spectrum feature aggregation layer respectively;

[0027] In the process of processing foam spectrum information, the spectrum segmentation feature matrix F3 is first combined with the manually added spectrum classification token to obtain the combined features, and the combined feature representation is normalized through the layer standardization layer to strengthen the global expression of the foam spectrum features;

[0028] These normalized features are then input into the spectral multi-head self-attention layer, which is used to capture the global dependencies in the foam spectrum data and enhance the model's understanding of the complex patterns and contextual information of the foam spectrum. The processed features are residually connected with the combined features to retain the original information and promote information flow;

[0029] The features after residual connection are again processed by layer standardization and enter the foam spectrum feature aggregation layer composed of feedforward neural network. The foam spectrum feature aggregation layer performs nonlinear transformation and further feature enhancement to extract the deep spectrum feature representation of the foam. Finally, the features after multi-layer processing and enhancement are integrated again through residual connection to form the foam spectrum feature matrix F that can fully reflect the spectral characteristics of flotation foam. M .

[0030] Optionally, the foam space feature extraction module includes a layer normalization layer, a spatial multi-head self-attention layer, a layer normalization layer, and a foam space feature conversion extraction layer, and residual connections are introduced before and after the spatial multi-head self-attention layer and the foam space feature conversion extraction layer respectively;

[0031] In the process of processing bubble space information, first the spatial feature token and the manually added spatial classification token Interactive learning aggregates global features to obtain combined features, and then enters the spatial multi-head self-attention layer to capture the global dependency between input features after the feature representation is standardized by the layer standardization layer. The features processed by the spatial multi-head self-attention layer are residually connected with the combined features, and then pass through the layer standardization layer again to obtain the intermediate feature F c Then, it enters the foam space feature conversion and extraction layer to convert and extract the foam space features, and directly digs deep into the rich foam space information from the feature matrix. Finally, the features that have been processed and enhanced by multiple layers are integrated again through residual connections to obtain the foam space feature matrix Fs that reflects the fine spatial information in the flotation foam multi-spectral image.

[0032] Optionally, the foam space feature conversion and extraction layer extracts foam space features by converting a tag token into an image, and then performs feature enhancement through a process of image feature extraction and image-to-token conversion to output a feature representation rich in foam space information, including a multi-spectral tag-to-image conversion module, a foam space feature extraction stage, a multi-spectral image-to-tag restoration module, a fully connected layer, a layer normalization layer, and a GELU activation function;

[0033] The multi-spectral label is transferred to the image conversion module, and the intermediate feature F c Separate into intermediate feature tokens and space classification token , and then the intermediate feature token Reshape into a 2D feature matrix , this process is described by the following formula:

[0034]

[0035] in, Represents the intermediate feature token after separation, Represents a spatial classification token. Separate(·) is a separation operation used to separate spatial classification tokens. From the intermediate feature F c Reshape(·) is a reshaping function that transforms the separated intermediate feature tokens Reshape into a two-dimensional input feature matrix ;

[0036] In the foam space feature extraction stage, the input feature matrix The spatial features are captured through the two-dimensional convolution layer, and the GELU activation function and batch normalization layer are used to enhance the feature representation. Then, the two-dimensional average pooling layer is entered. This layer is used to reduce the spatial dimension of the features and reduce the risk of overfitting. At the same time, residual connections are introduced to maintain information flow and prevent gradient disappearance, allowing the network to learn deeper feature representations. Part or all of the convolution, activation, normalization, and pooling are repeated multiple times in the network, so that the model can gradually extract the spatial information of the flotation foam. After the foam space feature extraction stage, a multi-spectral enhanced feature matrix is ​​generated. ;

[0037] The multi-spectral image to label restoration module converts the multi-spectral enhanced feature matrix Flatten it again and convert it back to feature token form and compare it with the spatial classification token Splice to obtain the spliced ​​multi-spectral enhanced feature matrix Then, a fully connected layer and a GELU nonlinear activation function are used to further refine the complex relationship between features and enhance feature representation to obtain the foam space feature matrix F s , this process is described by the following formula:

[0038]

[0039]

[0040] in, represents the multi-spectral enhanced feature matrix, Flatten(·) is used to flatten the two-dimensional feature matrix into a one-dimensional token, represents the spatial classification token, and Concatenate(·) represents the cascade operation, which combines the flattened feature token with the spatial classification token. merge, represents the multi-spectral enhanced feature matrix after splicing, represents the nonlinear activation function GELU, Representation layer normalization, represents a fully connected layer.

[0041] Optionally, the multi-spectral feature fusion module first downsamples the two input features by adaptive mean pooling to ensure that their dimensions match, then performs an expansion operation on the features with adjusted dimensions, and merges the two into a stacked feature F by a tensor stacking function. z , integrating foam information at different scales;

[0042] Then, a softmax activation function is applied along the stacking dimension to generate a spectral feature weight matrix and a spatial feature weight matrix. These weights represent the importance of the corresponding foam spectral features and spatial features in the final feature representation.

[0043] Finally, the original features are adaptively pooled and fused according to the obtained weight matrix to obtain the fusion feature matrix F of the comprehensive foam spectral features and spatial features. This process is described by the following formula:

[0044]

[0045]

[0046] in, It represents the adaptive mean pooling function, n1 and n2 are F M and F S The goal of pooling is to adjust the length of these two sequences to F M and F S The minimum dimension in represents the dimension expansion function, Stack(·) is the tensor stacking operation, Express Corresponding The softmax activation function is applied to the part of Express Corresponding The softmax activation function is applied to the part, F represents the fusion feature matrix, Represents feature weighting.

[0047] Optionally, the foam color characterization module first maps to the category label of each pixel through a fully connected layer to obtain the classification probability of each pixel, and outputs the predicted color category of each foam;

[0048] Then, the predicted color category is input into the statistical analysis layer, and the number and proportion of bubbles in each color category are counted to obtain a statistical feature vector;

[0049] Finally, the obtained statistical feature vector is passed to a multi-layer perceptron, and the color state of the flotation foam is used as a prediction target to obtain a color state prediction of the flotation tank foam.

[0050] On the other hand, a flotation foam color analysis system based on multi-spectral dual-branch deep fusion is provided, the system comprising:

[0051] A collection module, used for collecting multi-spectral images of foam to be color analyzed in a flotation working area using a multi-spectral imaging device;

[0052] A preprocessing module, used for preprocessing the multi-spectral image;

[0053] Prediction module is used to convert the preprocessed multi-spectral image Input the trained multi-spectral dual-branch deep fusion network, and output the color state prediction of the foam;

[0054] The multi-spectral dual-branch deep fusion network first processes the pre-processed multi-spectral image Perform multi-spectral shallow feature extraction to obtain a multi-spectral shallow feature matrix F1 that reflects the foam spatial information and spectral information;

[0055] The multi-spectral shallow feature matrix F1 is input in parallel into two branches respectively for the foam spectral feature and the spatial feature:

[0056] In the first branch, the multi-spectral shallow feature matrix F1 is input into the channel contraction segmentation module to obtain a spectrum segmentation feature matrix F3 with rich spectral information, and the spectrum segmentation feature matrix F3 is input into the foam spectrum feature extraction module to deeply extract the spectral features of the flotation foam, and output the foam spectrum feature matrix F3 which can fully reflect the spectral characteristics of the flotation foam. M ;

[0057] In the second branch, the multi-spectral shallow feature matrix F1 is passed through the spatial token generator to obtain a spatial feature token set, and the spatial feature token set is input into the foam spatial feature extraction module to deeply extract the spatial features of the flotation foam, and output the foam spatial feature matrix F that reflects the fine spatial information in the flotation foam multi-spectral image. s ;

[0058] The foam spectrum feature matrix F M and the foam space characteristic matrix F s Input the multi-spectral feature fusion module for intelligent weighted fusion to obtain the fusion feature matrix F of the comprehensive foam spectral features and spatial features;

[0059] The fused feature matrix F is input into the foam color characterization module, and the foam color state prediction is output.

[0060] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the above-mentioned flotation foam color analysis method based on multi-spectral dual-branch deep fusion.

[0061] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement the above-mentioned flotation foam color analysis method based on multi-spectral dual-branch deep fusion.

[0062] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0063] 1) The present invention combines multispectral information acquisition technology to analyze foam characteristics. Compared with RGB images, multispectral images (including hyperspectral images and multispectral images) can directly measure the spectral reflectance and absorption characteristics of flotation foam in multiple continuous narrow bands, revealing its spectral characteristics more deeply and accurately, and solving the problem of inaccurate foam color recognition due to color difference in RGB images.

[0064] 2) The present invention reduces the dimensionality of multi-spectral information by principal component analysis and then processes the reduced multi-spectral information by a spectral three-dimensional convolution layer to capture the multi-dimensional spectral channel characteristics of the multi-spectral information, thereby solving the problem that the RGB image of flotation foam only provides image information of the red, green and blue channels but cannot provide more dimensional spectral information.

[0065] 3) The multi-spectral dual-branch deep fusion network designed in the present invention can not only capture the spectral characteristics reflecting the global state of the entire flotation foam, but also accurately grasp the local spatial characteristics of the foam, thereby comprehensively extracting the foam characteristics at the same time, solving the problem that RGB image processing technology lacks in-depth analysis of spectral information during feature extraction.

[0066] 4) The present invention significantly improves the accuracy and adaptability of flotation froth color analysis by integrating the statistical characteristic information of froth color. By introducing the statistical characteristic vector of froth color, the model can more comprehensively capture the diversity and complexity of froth color, thereby maintaining high prediction accuracy under different flotation conditions and enhancing the model's ability to recognize the froth color state. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0068] Figure 1 It is a flow chart of a flotation foam color analysis method based on multi-spectral double-branch deep fusion provided by an embodiment of the present invention;

[0069] Figure 2 It is a schematic diagram of the multi-spectral dual-branch deep fusion network structure provided by an embodiment of the present invention;

[0070] Figure 3 It is a schematic diagram of the structure of a spectral channel shrinking and splitting module provided in an embodiment of the present invention;

[0071] Figure 4 It is a schematic diagram of the structure of the foam space characteristic conversion and extraction layer provided by an embodiment of the present invention;

[0072] Figure 5 It is a schematic diagram of the structure of a multi-spectral feature fusion module provided in an embodiment of the present invention;

[0073] Figure 6 It is a schematic diagram of the training process of a multi-spectral dual-branch deep fusion network provided by an embodiment of the present invention;

[0074] Figure 7It is a block diagram of a flotation foam color analysis system based on multi-spectral dual-branch deep fusion provided by an embodiment of the present invention;

[0075] Figure 8 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0076] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0077] The embodiment of the present invention provides a flotation foam color analysis method based on multi-spectral dual-branch deep fusion, which can be implemented by an electronic device, which can be a terminal or a server. Figure 1 The flowchart of the flotation foam color analysis method based on multi-spectral dual-branch deep fusion is shown. The processing flow of the method may include the following steps:

[0078] S1. Using a multispectral imaging device to collect a multispectral image of the foam to be color analyzed in the flotation working area;

[0079] S2, preprocessing the multi-spectral image;

[0080] Optionally, the step S2 of preprocessing the multi-spectral image specifically includes:

[0081] The edge filling technology is used to enhance the image boundary information, so as to effectively preserve the integrity of the foam in the spatial dimension and the richness in the spectral dimension;

[0082] In view of the high dimensionality and noise independence characteristics of multi-spectral image information, principal component analysis is used for dimensionality reduction and denoising.

[0083] S3, the preprocessed multi-spectral image Input the trained multi-spectral dual-branch deep fusion network, and output the color state prediction of the foam;

[0084] like Figure 2 As shown, the multi-spectral dual-branch deep fusion network first processes the preprocessed multi-spectral image Perform multi-spectral shallow feature extraction to obtain a multi-spectral shallow feature matrix F1 that reflects the foam spatial information and spectral information;

[0085] The multi-spectral shallow feature matrix F1 is input in parallel into two branches respectively for the foam spectral feature and the spatial feature:

[0086] In the first branch, the multi-spectral shallow feature matrix F1 is input into the channel contraction segmentation module to obtain a spectrum segmentation feature matrix F3 with rich spectral information, and the spectrum segmentation feature matrix F3 is input into the foam spectrum feature extraction module to deeply extract the spectral features of the flotation foam, and output the foam spectrum feature matrix F3 which can fully reflect the spectral characteristics of the flotation foam. M ;

[0087] In the second branch, the multi-spectral shallow feature matrix F1 is passed through the spatial token generator to obtain a spatial feature token set, and the spatial feature token set is input into the foam spatial feature extraction module to deeply extract the spatial features of the flotation foam, and output the foam spatial feature matrix F that reflects the fine spatial information in the flotation foam multi-spectral image. s ;

[0088] The foam spectrum feature matrix F M and the foam space characteristic matrix F s Input the multi-spectral feature fusion module for intelligent weighted fusion to obtain the fusion feature matrix F of the comprehensive foam spectral features and spatial features;

[0089] The fused feature matrix F is input into the foam color characterization module, and the color state of the foam (the color state refers to the overall color depth of the flotation tank foam reflected by the different distribution and proportion of different color foams at the current moment) is predicted.

[0090] Alternatively, if Figure 2 As shown, the multi-spectral dual-branch deep fusion network processes the pre-processed multi-spectral image Perform multi-spectral shallow feature extraction to obtain a multi-spectral shallow feature matrix F1 reflecting the foam spatial information and spectral information, specifically including:

[0091] The preprocessed multispectral image The input is sent to a module including a spectral three-dimensional convolution layer, a batch normalization layer, and a ReLU nonlinear activation function layer to extract multi-spectral shallow features, and obtain the multi-spectral shallow feature matrix F1 reflecting the foam space information and spectral information. This process is described by the following formula:

[0092]

[0093] in, represents the nonlinear activation function ReLU, represents batch normalization, represents the convolution operator, and They respectively represent the weight and bias of the three-dimensional convolution, where the three dimensions of the three-dimensional convolution include the width, height, and spectral dimensions of the image.

[0094] Alternatively, if Figure 3 As shown, the channel contraction segmentation module is composed of a channel contraction module and a segmentation module. The channel contraction module includes a two-dimensional convolution layer, a batch normalization layer and a ReLU nonlinear activation function layer. The purpose is to compress redundant channel information while strengthening the spectral information expression of the foam, and generate a spectral channel contraction feature matrix F2. After that, the spectral channel contraction feature matrix F2 is input into the segmentation module for token division. The token area after division is flattened into a one-dimensional vector by a flattening function to obtain the spectral segmentation feature matrix F3 with rich spectral information. This process is described by the following formula:

[0095]

[0096]

[0097] in, represents the p×p token region from the position (j,i) to the position (j+p,i+p) of the input foam multispectral image, and Flatten(·) represents the flattening function, which flattens these token regions into a one-dimensional vector.

[0098] Alternatively, if Figure 2 As shown, the foam spectrum feature extraction module includes a layer normalization layer, a spectrum multi-head self-attention layer, a layer normalization layer, and a foam spectrum feature aggregation layer, and residual connections are introduced before and after the spectrum multi-head self-attention layer and the foam spectrum feature aggregation layer respectively;

[0099] In the process of processing foam spectrum information, the spectrum segmentation feature matrix F3 is first combined with the manually added spectrum classification token to obtain the combined features, and the combined feature representation is normalized through the layer standardization layer to strengthen the global expression of the foam spectrum features;

[0100] These normalized features are then input into the spectral multi-head self-attention layer, which is used to capture the global dependencies in the foam spectrum data and enhance the model's understanding of the complex patterns and contextual information of the foam spectrum. The processed features are residually connected with the combined features to retain the original information and promote information flow;

[0101] The features after residual connection are again processed by layer standardization and enter the foam spectrum feature aggregation layer composed of feedforward neural network. The foam spectrum feature aggregation layer performs nonlinear transformation and further feature enhancement to extract the deep spectrum feature representation of the foam. Finally, the features after multi-layer processing and enhancement are integrated again through residual connection to form the foam spectrum feature matrix F that can fully reflect the spectral characteristics of flotation foam. M .

[0102] Alternatively, if Figure 2 As shown, the foam space feature extraction module includes a layer normalization layer, a spatial multi-head self-attention layer, a layer normalization layer, and a foam space feature conversion extraction layer, and residual connections are introduced before and after the spatial multi-head self-attention layer and the foam space feature conversion extraction layer respectively;

[0103] In the process of processing bubble space information, first the spatial feature token and the manually added spatial classification token Interactive learning aggregates global features to obtain combined features, and then enters the spatial multi-head self-attention layer to capture the global dependency between input features after the feature representation is standardized by the layer standardization layer. The features processed by the spatial multi-head self-attention layer are residually connected with the combined features, and then pass through the layer standardization layer again to obtain the intermediate feature F c Then, it enters the foam space feature conversion and extraction layer to convert and extract the foam space features, and directly digs deep into the rich foam space information from the feature matrix. Finally, the features that have been processed and enhanced by multiple layers are integrated again through residual connections to obtain the foam space feature matrix Fs that reflects the fine spatial information in the flotation foam multi-spectral image.

[0104] Alternatively, if Figure 4 As shown, the foam space feature conversion and extraction layer extracts foam space features by converting the marked token into an image, and then performs feature enhancement through the process of image feature extraction and image-to-token conversion, and outputs a feature representation rich in foam space information, including a multi-spectral mark-to-image conversion module, a foam space feature extraction stage, a multi-spectral image-to-marker restoration module, a fully connected layer, a layer normalization layer and a GELU activation function;

[0105] The multi-spectral label is transferred to the image conversion module, and the intermediate feature F c Separate into intermediate feature tokens and space classification token , and then the intermediate feature token Reshape into a 2D feature matrix , this process is described by the following formula:

[0106]

[0107] in, Represents the intermediate feature token after separation, Represents a spatial classification token. Separate(·) is a separation operation used to separate spatial classification tokens. From the intermediate feature F c Reshape(·) is a reshaping function that transforms the separated intermediate feature tokens Reshape into a two-dimensional input feature matrix ;

[0108] In the foam space feature extraction stage, the input feature matrix The spatial features are captured through the two-dimensional convolution layer, and the feature representation is enhanced by using the GELU activation function and the batch normalization layer. Then, the two-dimensional average pooling layer is entered. This layer is used to reduce the spatial dimension of the features and reduce the risk of overfitting. At the same time, the residual connection is introduced to maintain the flow of information and prevent the gradient from disappearing, allowing the network to learn deeper feature representations (since the addition of residual connections will increase the number of network parameters and computational complexity, residual connections are only added to some networks). Part or all of the convolution, activation, normalization, and pooling are repeated multiple times in the network, so that the model can gradually extract the spatial information of the flotation foam. After the foam space feature extraction stage, a multi-spectral enhanced feature matrix is ​​generated. ;

[0109] The multi-spectral image to label restoration module converts the multi-spectral enhanced feature matrix Flatten it again and convert it back to feature token form and compare it with the spatial classification token Splice to obtain the spliced ​​multi-spectral enhanced feature matrix Then, a fully connected layer and a GELU nonlinear activation function are used to further refine the complex relationship between features and enhance feature representation to obtain the foam space feature matrix F s , this process is described by the following formula:

[0110]

[0111]

[0112] in, represents the multi-spectral enhanced feature matrix, Flatten(·) is used to flatten the two-dimensional feature matrix into a one-dimensional token, represents the spatial classification token, and Concatenate(·) represents the cascade operation, which combines the flattened feature token with the spatial classification token. merge, represents the multi-spectral enhanced feature matrix after splicing, represents the nonlinear activation function GELU, Representation layer normalization, represents a fully connected layer.

[0113] Alternatively, if Figure 5 As shown in the figure, the multi-spectral feature fusion module first downsamples the two input features through adaptive mean pooling to ensure that their dimensions match, then performs an expansion operation on the features with adjusted dimensions, and merges the two into a stacked feature F through a tensor stacking function. z , integrating foam information at different scales;

[0114] Then, a softmax activation function is applied along the stacking dimension to generate a spectral feature weight matrix and a spatial feature weight matrix. These weights represent the importance of the corresponding foam spectral features and spatial features in the final feature representation.

[0115] Finally, the original features are adaptively pooled and fused according to the obtained weight matrix to obtain the fusion feature matrix F of the comprehensive foam spectral features and spatial features. This process is described by the following formula:

[0116]

[0117]

[0118] in, It represents the adaptive mean pooling function, n1 and n2 are F M and F S The goal of pooling is to adjust the length of these two sequences to F M and F S The minimum dimension in represents the dimension expansion function, Stack(·) is the tensor stacking operation, Express Corresponding The softmax activation function is applied to the part of Express Corresponding The softmax activation function is applied to the part, F represents the fusion feature matrix, Represents feature weighting.

[0119] Alternatively, if Figure 2 As shown, the foam color characterization module first maps to the category label of each pixel through a fully connected layer to obtain the classification probability of each pixel and outputs the predicted color category of each foam;

[0120] Then, the predicted color category is input into the statistical analysis layer, and the number and proportion of bubbles in each color category are counted to obtain a statistical feature vector;

[0121] Finally, the obtained statistical feature vector is passed to a multi-layer perceptron, and the color state of the flotation foam is used as a prediction target to obtain a color state prediction of the flotation tank foam.

[0122] The training process of the multi-spectral dual-branch deep fusion network of the embodiment of the present invention is as follows: Figure 6 As shown, specifically including:

[0123] 1) Data collection;

[0124] Use a multispectral imaging device to collect multiple multispectral images of the froth in the flotation working area.

[0125] 2) Data preprocessing;

[0126] The edge filling technology is used to enhance the image boundary information, so as to effectively preserve the integrity of the foam in the spatial dimension and the richness in the spectral dimension;

[0127] In view of the high dimensionality and noise independence characteristics of multi-spectral image information, principal component analysis is used for dimensionality reduction and denoising.

[0128] 3) Data annotation;

[0129] While collecting data, the overall color state of the foam in the flotation tank at the current moment is marked by directly observing the color of the foam in the flotation tank. At the same time, image processing software is used to automatically draw the foam contour area on the multi-spectral image and assign corresponding color category labels to train the multi-spectral dual-branch deep fusion network.

[0130] 4) Dataset division;

[0131] The manually annotated data sets are randomly divided into training sets and test sets in a ratio of 8:2 to facilitate subsequent model training and testing;

[0132] 5) Model training and testing;

[0133] The multi-spectral dual-branch deep fusion network is trained using the data in the training set, and the model predictions are evaluated using the data in the test set. The accuracy of the model in analyzing the flotation foam color is verified. Based on the model's performance on the test set, its ability to analyze foam color is deeply understood, and the multi-spectral information technology is effectively combined with the specific characteristics of the foam to achieve accurate identification of the foam color.

[0134] like Figure 7As shown, an embodiment of the present invention further provides a flotation foam color analysis system based on multi-spectral dual-branch deep fusion, the system comprising:

[0135] The acquisition module 710 is used to acquire the multi-spectral image of the foam to be color analyzed in the flotation working area by using the multi-spectral imaging device;

[0136] A preprocessing module 720, used for preprocessing the multi-spectral image;

[0137] Prediction module 730, used to convert the pre-processed multi-spectral image Input the trained multi-spectral dual-branch deep fusion network, and output the color state prediction of the foam;

[0138] The multi-spectral dual-branch deep fusion network first processes the pre-processed multi-spectral image Perform multi-spectral shallow feature extraction to obtain a multi-spectral shallow feature matrix F1 that reflects the foam spatial information and spectral information;

[0139] The multi-spectral shallow feature matrix F1 is input in parallel into two branches respectively for the foam spectral feature and the spatial feature:

[0140] In the first branch, the multi-spectral shallow feature matrix F1 is input into the channel contraction segmentation module to obtain a spectrum segmentation feature matrix F3 with rich spectral information, and the spectrum segmentation feature matrix F3 is input into the foam spectrum feature extraction module to deeply extract the spectral features of the flotation foam, and output the foam spectrum feature matrix F3 which can fully reflect the spectral characteristics of the flotation foam. M ;

[0141] In the second branch, the multi-spectral shallow feature matrix F1 is passed through the spatial token generator to obtain a spatial feature token set, and the spatial feature token set is input into the foam spatial feature extraction module to deeply extract the spatial features of the flotation foam, and output the foam spatial feature matrix F that reflects the fine spatial information in the flotation foam multi-spectral image. s ;

[0142] The spectral feature matrix F M and the foam space characteristic matrix F s Input the multi-spectral feature fusion module for intelligent weighted fusion to obtain the fusion feature matrix F of the comprehensive foam spectral features and spatial features;

[0143] The fused feature matrix F is input into the foam color characterization module, and the foam color state prediction is output.

[0144] An embodiment of the present invention provides a flotation foam color analysis system based on multi-spectral dual-branch deep fusion, and its functional structure corresponds to a flotation foam color analysis method based on multi-spectral dual-branch deep fusion provided by an embodiment of the present invention, which will not be repeated here.

[0145] Figure 8 It is a structural schematic diagram of an electronic device 800 provided in an embodiment of the present invention. The electronic device 800 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 801 and one or more memories 802, wherein at least one instruction is stored in the memory 802, and the at least one instruction is loaded and executed by the processor 801 to implement the steps of the above-mentioned flotation foam color analysis method based on multi-spectral dual-branch deep fusion.

[0146] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, which can be executed by a processor in a terminal to complete the above-mentioned flotation foam color analysis method based on multi-spectral dual-branch deep fusion. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0147] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0148] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A flotation foam color analysis method based on multi-spectral dual-branch deep fusion, characterized in that: The method comprises: S1. Using a multispectral imaging device to collect a multispectral image of the foam to be color analyzed in the flotation working area; S2, preprocessing the multi-spectral image; S3, the preprocessed multi-spectral image Input the trained multi-spectral dual-branch deep fusion network, and output the color state prediction of the foam; The multi-spectral dual-branch deep fusion network first processes the pre-processed multi-spectral image Perform multi-spectral shallow feature extraction to obtain a multi-spectral shallow feature matrix F1 that reflects the foam spatial information and spectral information; The multi-spectral shallow feature matrix F1 is input in parallel into two branches respectively for the foam spectral feature and the spatial feature: In the first branch, the multi-spectral shallow feature matrix F1 is input into the channel contraction segmentation module to obtain a spectrum segmentation feature matrix F3 with rich spectral information, and the spectrum segmentation feature matrix F3 is input into the foam spectrum feature extraction module to deeply extract the spectral features of the flotation foam, and output the foam spectrum feature matrix F3 which can fully reflect the spectral characteristics of the flotation foam. M ; In the second branch, the multi-spectral shallow feature matrix F1 is passed through the spatial token generator to obtain a spatial feature token set, and the spatial feature token set is input into the foam spatial feature extraction module to deeply extract the spatial features of the flotation foam, and output the foam spatial feature matrix F that reflects the fine spatial information in the flotation foam multi-spectral image. s ; The foam spectrum feature matrix F M and the foam space characteristic matrix F s Input the multi-spectral feature fusion module for intelligent weighted fusion to obtain the fusion feature matrix F of the comprehensive foam spectral features and spatial features; The fused feature matrix F is input into the foam color characterization module, and the foam color state prediction is output.

2. The method according to claim 1, characterized in that The step S2 of preprocessing the multi-spectral image specifically includes: The edge filling technology is used to enhance the image boundary information, so as to effectively preserve the integrity of the foam in the spatial dimension and the richness in the spectral dimension; In view of the high dimensionality and noise independence characteristics of multi-spectral image information, principal component analysis is used for dimensionality reduction and denoising.

3. The method according to claim 1, characterized in that: The multi-spectral dual-branch deep fusion network processes the pre-processed multi-spectral image Perform multi-spectral shallow feature extraction to obtain a multi-spectral shallow feature matrix F1 reflecting the foam spatial information and spectral information, specifically including: The preprocessed multispectral image The input is sent to a module including a spectral three-dimensional convolution layer, a batch normalization layer, and a ReLU nonlinear activation function layer to extract multi-spectral shallow features, and obtain the multi-spectral shallow feature matrix F1 reflecting the foam space information and spectral information. This process is described by the following formula: ； in, represents the nonlinear activation function ReLU, represents batch normalization, represents the convolution operator, and They respectively represent the weight and bias of the three-dimensional convolution, where the three dimensions of the three-dimensional convolution include the width, height, and spectral dimensions of the image.

4. The method according to claim 1, characterized in that The channel contraction segmentation module is composed of a channel contraction module and a segmentation module. The channel contraction module includes a two-dimensional convolution layer, a batch normalization layer and a ReLU nonlinear activation function layer. The purpose is to compress redundant channel information while strengthening the spectral information expression of the foam to generate a spectral channel contraction feature matrix F2. After that, the spectral channel contraction feature matrix F2 is input into the segmentation module for token division. The token area after division is flattened into a one-dimensional vector by a flattening function to obtain the spectral segmentation feature matrix F3 with rich spectral information. This process is described by the following formula: ； ； in, represents the p×p token region from the position (j,i) to the position (j+p,i+p) of the input foam multispectral image, and Flatten(·) represents the flattening function, which flattens these token regions into a one-dimensional vector.

5. The method according to claim 1, characterized in that The foam spectrum feature extraction module includes a layer normalization layer, a spectrum multi-head self-attention layer, a layer normalization layer, and a foam spectrum feature aggregation layer, and residual connections are introduced before and after the spectrum multi-head self-attention layer and the foam spectrum feature aggregation layer respectively; In the process of processing foam spectrum information, the spectrum segmentation feature matrix F3 is first combined with the manually added spectrum classification token to obtain the combined features, and the combined feature representation is normalized through the layer standardization layer to strengthen the global expression of the foam spectrum features; These normalized features are then input into the spectral multi-head self-attention layer, which is used to capture the global dependencies in the foam spectrum data and enhance the model's understanding of the complex patterns and contextual information of the foam spectrum. The processed features are residually connected with the combined features to retain the original information and promote information flow; The features after residual connection are again processed by layer standardization and enter the foam spectrum feature aggregation layer composed of feedforward neural network. The foam spectrum feature aggregation layer performs nonlinear transformation and further feature enhancement to extract the deep spectrum feature representation of the foam. Finally, the features after multi-layer processing and enhancement are integrated again through residual connection to form the foam spectrum feature matrix F that can fully reflect the spectral characteristics of flotation foam. M .

6. The method according to claim 1, characterized in that The foam space feature extraction module includes a layer normalization layer, a spatial multi-head self-attention layer, a layer normalization layer, and a foam space feature conversion extraction layer, and residual connections are introduced before and after the spatial multi-head self-attention layer and the foam space feature conversion extraction layer respectively; In the process of processing bubble space information, first the spatial feature token and the manually added spatial classification token Interactive learning aggregates global features to obtain combined features, and then enters the spatial multi-head self-attention layer to capture the global dependency between input features after the feature representation is standardized by the layer standardization layer. The features processed by the spatial multi-head self-attention layer are residually connected with the combined features, and then pass through the layer standardization layer again to obtain the intermediate feature F c Then, it enters the foam space feature conversion and extraction layer to convert and extract the foam space features, and directly digs deep into the rich foam space information from the feature matrix. Finally, the features that have been processed and enhanced by multiple layers are integrated again through residual connections to obtain the foam space feature matrix Fs that reflects the fine spatial information in the flotation foam multi-spectral image.

7. The method according to claim 6, characterized in that The foam space feature conversion and extraction layer extracts foam space features by converting the marked token into an image, and then performs feature enhancement through the process of image feature extraction and image-to-token conversion to output a feature representation rich in foam space information, including a multi-spectral mark-to-image conversion module, a foam space feature extraction stage, a multi-spectral image-to-marker restoration module, a fully connected layer, a layer normalization layer and a GELU activation function; The multi-spectral label is transferred to the image conversion module, and the intermediate feature F c Separate into intermediate feature tokens and space classification token , and then the intermediate feature token Reshape into a 2D feature matrix , this process is described by the following formula: ； in, Represents the intermediate feature token after separation, Represents a spatial classification token. Separate(·) is a separation operation used to separate spatial classification tokens. From the intermediate feature F c Reshape(·) is a reshaping function that transforms the separated intermediate feature tokens Reshape into a two-dimensional input feature matrix ; In the foam space feature extraction stage, the input feature matrix The spatial features are captured through the two-dimensional convolution layer, and the GELU activation function and batch normalization layer are used to enhance the feature representation. Then, the two-dimensional average pooling layer is entered. This layer is used to reduce the spatial dimension of the features and reduce the risk of overfitting. At the same time, residual connections are introduced to maintain information flow and prevent gradient disappearance, allowing the network to learn deeper feature representations. Part or all of the convolution, activation, normalization, and pooling are repeated multiple times in the network, so that the model can gradually extract the spatial information of the flotation foam. After the foam space feature extraction stage, a multi-spectral enhanced feature matrix is ​​generated. ; The multi-spectral image to label restoration module converts the multi-spectral enhanced feature matrix Flatten it again and convert it back to feature token form and compare it with the spatial classification token Splice to obtain the spliced ​​multi-spectral enhanced feature matrix Then, a fully connected layer and a GELU nonlinear activation function are used to further refine the complex relationship between features and enhance feature representation to obtain the foam space feature matrix F s , this process is described by the following formula: ； ； in, represents the multi-spectral enhanced feature matrix, Flatten(·) is used to flatten the two-dimensional feature matrix into a one-dimensional token, represents the spatial classification token, and Concatenate(·) represents the cascade operation, which combines the flattened feature token with the spatial classification token. merge, represents the multi-spectral enhanced feature matrix after splicing, represents the nonlinear activation function GELU, Representation layer normalization, represents a fully connected layer.

8. The method according to claim 1, characterized in that The multi-spectral feature fusion module first downsamples the two input features through adaptive mean pooling to ensure that their dimensions match, then performs an expansion operation on the features with adjusted dimensions and merges the two into a stacked feature F through a tensor stacking function. z , integrating foam information at different scales; Then, a softmax activation function is applied along the stacking dimension to generate a spectral feature weight matrix and a spatial feature weight matrix. These weights represent the importance of the corresponding foam spectral features and spatial features in the final feature representation. Finally, the original features are adaptively pooled and fused according to the obtained weight matrix to obtain the fusion feature matrix F of the comprehensive foam spectral features and spatial features. This process is described by the following formula: ； ； in, It represents the adaptive mean pooling function, n1 and n2 are F M and F S The goal of pooling is to adjust the length of these two sequences to F M and F S The minimum dimension in represents the dimension expansion function, Stack(·) is the tensor stacking operation, Express Corresponding The softmax activation function is applied to the part of Express Corresponding The softmax activation function is applied to the part, F represents the fusion feature matrix, Represents feature weighting.

9. The method according to claim 1, characterized in that: The foam color characterization module first maps to the category label of each pixel through a fully connected layer to obtain the classification probability of each pixel and outputs the predicted color category of each foam; Then, the predicted color category is input into the statistical analysis layer, and the number and proportion of bubbles in each color category are counted to obtain a statistical feature vector; Finally, the obtained statistical feature vector is passed to a multi-layer perceptron, and the color state of the flotation foam is used as a prediction target to obtain a color state prediction of the flotation tank foam.

10. A flotation foam color analysis system based on multi-spectral dual-branch deep fusion, characterized in that: The system comprises: A collection module, used for collecting multi-spectral images of foam to be color analyzed in a flotation working area using a multi-spectral imaging device; A preprocessing module, used for preprocessing the multi-spectral image; Prediction module is used to convert the preprocessed multi-spectral image Input the trained multi-spectral dual-branch deep fusion network, and output the color state prediction of the foam; The multi-spectral dual-branch deep fusion network first processes the pre-processed multi-spectral image Perform multi-spectral shallow feature extraction to obtain a multi-spectral shallow feature matrix F1 that reflects the foam spatial information and spectral information; The multi-spectral shallow feature matrix F1 is input in parallel into two branches respectively for the foam spectral feature and the spatial feature: In the first branch, the multi-spectral shallow feature matrix F1 is input into the channel contraction segmentation module to obtain a spectrum segmentation feature matrix F3 with rich spectral information, and the spectrum segmentation feature matrix F3 is input into the foam spectrum feature extraction module to deeply extract the spectral features of the flotation foam, and output the foam spectrum feature matrix F3 which can fully reflect the spectral characteristics of the flotation foam. M ; In the second branch, the multi-spectral shallow feature matrix F1 is passed through the spatial token generator to obtain a spatial feature token set, and the spatial feature token set is input into the foam spatial feature extraction module to deeply extract the spatial features of the flotation foam, and output the foam spatial feature matrix F that reflects the fine spatial information in the flotation foam multi-spectral image. s ; The foam spectrum feature matrix F M and the foam space characteristic matrix F s Input the multi-spectral feature fusion module for intelligent weighted fusion to obtain the fusion feature matrix F of the comprehensive foam spectral features and spatial features; The fused feature matrix F is input into the foam color characterization module, and the foam color state prediction is output.

Images