Coal gangue identification method based on visible-near infrared spectrum and image multi-modal information fusion

Abstract

The invention belongs to the technical field of coal gangue recognition and separation, and particularly relates to a coal gangue recognition method based on visible-near infrared spectrum and image multi-modal information fusion. Comprising the following steps: S1, collecting spectral information and image information of a to-be-identified sample; s2, preprocessing the spectral information and the image information respectively; s3, extracting spectral features from the spectral data set by using a spectral feature extraction neural network model; extracting image features from the image data set by using an image feature extraction neural network model; s4, inputting the extracted spectral features and image features into a double-flow fusion network; s5, respectively inputting the extracted spectral features, the image features and the comprehensive features into a spectral branch classifier, an image branch classifier and a fusion branch classifier; s6, calculating importance weights of the spectrum branch, the image branch and the map fusion branch; and S7, multiplying and accumulating the importance weight and the corresponding confidence coefficient to obtain a coal gangue score matrix, and realizing coal gangue identification by using the score matrix.

Description

Technical Field

[0001] The invention belongs to the technical field of coal gangue identification and sorting, and specifically is a coal gangue identification method based on visible-near infrared spectrum and image multimodal information fusion. Background Art

[0002] my country is a major energy producer and consumer. High-quality energy development is the basic guarantee for the long-term stable growth of the national economy. In the process of modern mechanical mining, due to mining environment restrictions, raw coal usually contains a large amount of gangue. The increase in gangue content will greatly reduce the quality of raw coal, resulting in low coal utilization efficiency and more pollutant emissions. It is internationally recognized that coal preparation is the preferred solution for efficient and clean utilization of coal. Coal gangue sorting technology will become an important technical support for the development of the coal industry, and will be fully popularized by 2035, realizing intelligent, safe and precise mining of coal, and promoting the innovative development of my country's main energy technology.

[0003] Traditional gangue sorting mainly relies on the sorting workers' own experience. Although this method is simple to operate, it is easily affected by individual factors of workers, has high work intensity and low gangue sorting efficiency. Limited by the low accuracy of raw coal sorting by traditional dry sorting method, my country currently mainly uses wet sorting method for raw coal sorting, but it has many problems such as high energy consumption of gangue crushing, waste of water resources, and low lump coal rate. In recent years, methods based on spectral analysis and image recognition have improved their usability due to their advantages of rapid and non-destructive detection, but spectral data and image data are easily affected by optical system parameters, and it is difficult to handle actual abnormal conditions of coal slime coverage and gangue stacking.

[0004] In order to solve the above problems, we proposed a coal gangue intelligent identification method based on visible-near infrared spectrum and image multimodal information fusion. The information of two different information sources, spectrum and image, was processed, analyzed, integrated and inferred, and the spectral characteristics and image characteristics of coal gangue were preprocessed, feature extracted, feature fused, pattern matched, and decision fused to finally realize coal gangue identification under harsh working conditions.

[0005] In today's era of information explosion, we are faced with the challenges of massive data and diversified information. A single source of information often cannot meet our needs for comprehensive understanding and decision-making of problems. The neural network fusion algorithm integrates spectral and image information from multiple angles and levels, revealing the connections hidden between various types of information, thereby generating more comprehensive and comprehensive knowledge and providing more comprehensive, accurate and reliable results. Summary of the invention

[0006] In order to solve the above problems, the present invention provides a scraper conveyor test platform and test method based on similarity theory.

[0007] The present invention adopts the following technical scheme: a coal gangue identification method based on visible-near infrared spectrum and image multimodal information fusion, comprising: S1: Collect the spectral information and image information of the sample to be identified; S2: Preprocess the spectral information and image information respectively, and construct a spectrum data set with one spectrum and one image. The spectrum data set includes the spectral data set and the image data set to obtain the model prior knowledge; S3: Extract spectral features from the spectral dataset using the spectral feature extraction neural network model; extract image features from the image dataset using the image feature extraction neural network model; S4: The extracted spectral features and image features are input into the two-stream fusion network to output a comprehensive feature; S5: respectively inputting the extracted spectral features, image features and comprehensive features into the spectral branch classifier, the image branch classifier and the fusion branch classifier to obtain the category confidence under the three classifiers; S6: Calculate the importance weights of the spectrum branch, image branch and atlas fusion branch; S7: The importance weights are multiplied and added with the corresponding confidence levels to obtain the gangue scoring matrix, and the scoring matrix is ​​used to realize gangue identification.

[0008] In some embodiments, in step S1, a fiber optic probe is used to collect spectral information of the sample surface. Before collecting the spectral information of the sample, the halogen lamp and the spectrometer are turned on for preheating. The spectral information of the polytetrafluoroethylene white board and the probe when covered is first collected for data processing. The spectral information of the front, middle and tail of each sample is collected, and the average spectrum of the three positions is taken as the spectral information of the sample; An industrial lens is used to collect image information of the sample, and the image information corresponds one-to-one with the spectral information.

[0009] In some embodiments, in step S2, Spectral information preprocessing includes: It is processed in sequence by the existing black and white correction, initial and final band elimination, SG smoothing and standard normal vector transformation techniques; Image information preprocessing includes: Uniformly resize images, convert resized images to tensor format, and normalize images.

[0010] In some embodiments, in step S2, the preprocessed spectral features and image features are matched one by one and encapsulated in a TensorDataset together with the labels.

[0011] In some embodiments, in step S3, The spectral feature extraction neural network model includes: A recurrent neural network branch, wherein the recurrent neural network branch includes: an LSTM layer, a batch normalization layer, and an expansion layer, and the spectral information is sequentially expanded through the LSTM layer and the batch normalization layer to obtain spectral sequence features; A convolutional neural network branch, wherein the convolutional neural network branch comprises: a one-dimensional convolution layer, a maximum pooling layer and a Dropout layer, and the spectral information is passed through the one-dimensional convolution layer, the maximum pooling layer and the Dropout layer to obtain spectral convolution features; The spectral sequence features and spectral convolution features are fused through the Concat operation and input into the second convolution layer and Relu layer to obtain the spectral features.

[0012] In some embodiments, in step S3, The image feature extraction neural network model consists of a convolution block, 17 inverted residual blocks and a cascade of convolution blocks, each convolution block includes a two-dimensional convolution layer, a batch normalization layer and a Relu6 layer, and each inverted residual block includes a lightweight dilated convolution layer, a depth-wise separable convolution layer and a point-by-point convolution layer.

[0013] In some embodiments, in step S4, the dual-stream fusion network includes: A first convolution branch, wherein the first convolution branch comprises: a convolution layer, a batch normalization layer and a convolution layer, the spectral feature is input into the first convolution branch, and the spectral feature 1 is output; A second convolution branch, wherein the second convolution branch comprises: a convolution layer, a batch normalization layer and a convolution layer, the image feature is input into the second convolution branch, and the image feature 1 is output; A third convolution branch, the third convolution branch comprising: a convolution layer and a batch normalization layer; the spectral feature is input into the third convolution branch, and the spectral feature 2 is output; A fourth convolution branch, wherein the fourth convolution branch comprises: a convolution layer, a batch normalization layer and an average pooling layer, the image feature is input into the fourth convolution branch, and the image feature 2 is output; Self-attention mechanism, spectral feature 2 and image feature 2 are subjected to the self-attention mechanism to obtain the weight distribution of the original features in the fusion feature space; The spectral feature 2 is multiplied and fused with the image feature 2 to obtain the fused feature 1. The spectral feature 2 and the image feature 2 are respectively multiplied and fused with the fused feature 1, and then a splicing operation is performed. Normalization is performed in the global feature, and the normalized feature weights are respectively multiplied with the spectral feature 1 and the image feature 1 to obtain the spectral weight feature and the image weight feature under the guidance of the fused feature space. The two-modal weight features are added and fused to obtain the comprehensive feature, and then a convolution process is performed to obtain a deeper comprehensive feature.

[0014] In some embodiments, in step S5: The spectral branch classifier includes: fully connected layer and Sigmoid layer; The image branch classifier includes: adaptive average pooling layer and 1*1 convolution layer; The fusion branch classifier includes: two fully connected layers, and the activation function is Relu.

[0015] In some embodiments, step S6 includes: Based on the atlas dataset, a neural network model consisting of the above-mentioned spectral feature extraction neural network model and image feature extraction neural network model, a two-stream fusion network, a spectral branch classifier, an image branch classifier, and a fusion branch classifier is trained to obtain the accuracy, F1 value, and root mean square error of the spectral branch classifier, the image branch classifier, and the fusion branch classifier, which are used as prior knowledge; According to the requirements and scenarios of gangue sorting, the target layer, indicator layer and scheme layer are constructed. The target layer is gangue sorting, the indicator layer is classification accuracy, F1 value and root mean square error, and the scheme layer is spectral classifier, image classifier and fusion classifier. According to the task scenario and requirements, the importance of each indicator in the indicator layer relative to the task is judged, and the judgment matrix between the target layer and the indicator layer is formulated, where the judgment matrix element D ij It indicates the importance of the i-th row factor relative to the j-th column factor. The larger the value, the more important the factor is. The row and column factors are accuracy, F1 value and root mean square error respectively. Finally, a 3×3 judgment matrix is ​​obtained. According to the prior knowledge, namely the accuracy, F1 value and root mean square error of the spectral classifier, image classifier and fusion classifier, the maximum and minimum values ​​of each indicator are evenly divided into 9 intervals. The closer the indicator of the classifier is to the optimal value, the larger its quantization value is, which indicates the importance of the indicator of the i-th row classifier relative to the j-th column classifier. The row and column classifiers represent the spectral classifier, image classifier and fusion classifier respectively. The above operations are performed on each indicator, and finally three 3×3 judgment matrices are obtained; The factor weight matrix between the target layer and the indicator layer, and between the indicator layer and the solution layer is calculated using the eigenvalue method according to the judgment matrix, and the final classifier weight matrix is ​​formed by the factor weight matrix; Check whether the judgment matrix passes the consistency test. If it passes, the importance weight of each classifier is obtained by multiplying and adding the first column of the final weight matrix with the last three columns. If it fails, it proves that the judgment matrix is ​​set incorrectly and the judgment matrix is ​​reconstructed.

[0016] In some embodiments, step S7 includes: The three category confidence matrices output by the spectral classifier, image classifier and fusion classifier are multiplied and added with the classifier weight matrix to finally obtain the scores predicted as coal and gangue, thereby realizing coal and gangue identification.

[0017] Compared with the prior art, the present invention has the following beneficial effects: 1) LSTM-CNN is used to extract spectral feature maps of the collected spectral data of coal gangue samples. By combining two different types of neural network structures, LSTM and CNN, the wavelength dependency and local features in the data are captured simultaneously, and the characteristic information in the spectral data is fully extracted, thereby improving the model's ability to understand and analyze spectral data; 2) Extract image data features through the MobilNetV2 network, which has the advantages of lightweight, fast inference speed, and high performance; 3) The spectral features and image features are matched in terms of dimension and shape by transposing and multiplying the spectral one-dimensional data, so as to better integrate the feature information of different branches; 4) The extracted two-modal feature maps are complementary fused through the dual-stream branch fusion module. Under the guidance of different scales, feature representations of different scales are captured. Compared with the simple fusion method, this dual-stream guided fusion is more efficient; 5) By introducing the modal self-attention mechanism, this method uses the features obtained by multiplying and fusion of spectrum and image to guide the feature weights of spectrum and image, effectively learns the correlation and importance between different modal data, better guides the model to focus on important information, and reduces information redundancy; 6) The importance weights of the three branch classifiers of spectrum, image, and fusion are calculated through the hierarchical analysis method to improve the model's overall grasp and processing capabilities of the task, guide the model to pay more attention to and utilize the information of each branch more accurately, improve the model's robustness to data noise and interference, and make the model more stable and reliable.

[0018] The present invention uses multimodal information fusion technology to perform composite fusion of spectral information and image information. Feature fusion improves the model recognition performance and can effectively solve the problem of low recognition accuracy caused by harsh and complex working conditions such as large amounts of coal dust, poor lighting conditions, and noise interference in the mine. Decision fusion improves the stability and adaptability of the model and can handle the problem of misidentification caused by one-sided single-modal information under abnormal working conditions such as coal slime coverage. This method provides a technical basis for underground coal gangue sorting and on-site filling, which is not only beneficial to resource utilization and environmental protection, but also improves mine safety production and economic benefits. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is the flow chart of coal gangue multimodal information fusion recognition; Figure 2It is a schematic diagram of the structure of visible-near infrared spectrum and image information acquisition; Figure 3 is the spectrum preprocessing graph; Figure 4 It is the spectral feature extraction branch; Figure 5 It is the core inverted residual structure of the image feature extraction branch; Figure 6 It is a dual-stream guided feature fusion structure; Figure 7 It is a hierarchical model; Figure 8 It is a multi-branch weight calculation flow chart based on the analytic hierarchy process; In the figure: 1. halogen lamp; 2. g optical fiber bracket; 3. optical fiber probe; 4. optical fiber clamp; 5. industrial camera; 6. camera bracket; 7. spectrometer; 8. computer. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described clearly and completely in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] The present invention proposes a coal gangue identification method based on feature fusion and decision fusion of visible-near infrared spectrum and image two-modal information, and its flow chart is as follows: Figure 1 As shown, the following steps are included: The spectrum analysis instrument module is a set of equipment consisting of a spectrometer (USB2000+, Ocean Insight, USA), a fiber optic probe (1SMA1S-SI0.6-1500S, Nanjing Shen Lue Technology Co., Ltd, China) and a halogen lamp (JCR15V150WBAU), which is designed to collect the spectral intrinsic chemical information of coal gangue samples. In this module, the spectrometer is responsible for analyzing and processing the sample spectral signal, the fiber optic probe is used to collect the spectral data of the sample surface and transmit it to the spectrometer for further analysis, and the halogen lamp provides the required light source to excite the emission spectrum of the sample.

[0022] The machine vision instrument module is usually composed of Hikvision industrial area array camera (MV-CA050-12UC) and Hikvision 8-megapixel industrial lens (MVL-MF0828M-8MP 8mm), which is designed to collect image appearance information of coal gangue samples. The industrial area array camera is a high-performance image acquisition device that can quickly capture high-resolution images and transmit them to a computer for subsequent processing. The industrial lens is responsible for high-definition imaging of the sample to ensure image quality and clarity.

[0023] Data acquisition module: The halogen lamp emits light of different wavelengths, which are absorbed and reflected by the sample and transmitted to the spectrometer by the optical fiber probe. The optical signal is converted into an electrical signal. The spectral information is displayed and collected in real time through the spectrometer supporting software OceanView2.0.7. The entire system is placed under a light-proof black cloth to reduce interference from external stray light.

[0024] Before collecting the sample spectral information, turn on the halogen lamp and preheat the spectrometer for 20 minutes. First, collect the spectral information of the polytetrafluoroethylene white board and the covered probe for data processing. The wavelength range collected by the spectrometer is 369~1049nm, involving 2048 wavelengths, and the integration time is 8ms.

[0025] In order to prevent the sample from being inclined at too large an angle, causing the reflected spectrum to exceed the detection range of the probe, the present invention collects spectral information at three positions: the front, middle and tail of each sample, and takes the average spectrum of the three positions as the spectral information of the sample. During the collection process, the position and posture of the optical fiber probe and the halogen lamp remain unchanged, and the sample is only translated forward and backward.

[0026] The machine vision software MVS was used to manually control the camera to capture and store sample image information. The image resolution was 2448×2048, the pixel size was 3.45um, the exposure time was 10ms, and the spectral data corresponded to the image data one by one.

[0027] The spectrometer acquisition and camera shooting can be triggered by a photoelectric switch. For example, when the sample to be identified passes by on the conveyor belt, the timing starts, and the delay time is calculated by the belt speed and the distance from the spectrometer and the camera to the photoelectric switch. When the delay time is reached, the spectrum and image data are collected based on the communication between Python and the two acquisition devices. In addition, there is no requirement for the acquisition area of ​​the fiber optic probe and the camera, but it is best not to collect at the same viewing angle. The fiber optic probe can be placed on the side of the conveyor belt to collect the side data of the sample, and the camera can be placed on the top to collect the top image of the sample. The present invention does not make specific provisions.

[0028] Data processing module: Rely on matlab and Python to complete the noise reduction of spectral data and reduce the interference of instruments, environment, background and redundant information on the data. Figure 3As shown in the figure, the spectral raw data is processed by the existing black and white correction, the removal of the initial and final bands, the SG smoothing filter, and the standard normal vector transformation technology and then stored in the mat file. The image data is first uniformly resized to 224×224, and then the resized image is converted into a tensor format. Finally, the image is standardized, and the value of each channel is subtracted by 0.5 and divided by 0.5, so that the image data is between -1 and 1.

[0029] Dataset: Convert the above preprocessed spectral data, image data and labels into tensor array format, where the spectral information in the spectral dataset corresponds one-to-one with the image information. Store the dataset in a TensorDataset, which contains input1, input2 and target, and divide the training set, validation set and test set into 6:2:2. Create their own data loaders DataLoader, and set the batch size to 16.

[0030] Feature extraction module: This module mainly uses a deep learning neural network model to complete a more comprehensive and accurate extraction of spectral information and image information features. It is composed of a spectrum-image two-branch feature extraction neural network model. The spectral feature extraction branch is composed of the LSTM-CNN network we built. This network incorporates a recurrent neural network on the basis of the CNN network, which can effectively supplement the captured wavelength sequence information. The image feature extraction neural network model is composed of the MobilNetV2 network, which has the advantages of being lightweight and efficient.

[0031] Feature fusion module: Taking the bimodal features extracted by the two-branch feature extraction neural network as input, the built dual-stream fusion network is used to enhance the interactive fusion of spectral and image feature representations, realize the complementarity and enhancement of spectral modal information, form a more comprehensive feature representation, provide more comprehensive and accurate results, and enhance the model's ability to handle complex tasks. In addition, the modal self-attention mechanism is introduced in the dual-stream feature fusion network to allocate the weights of the two modal features in the fused features, achieve a more flexible feature fusion method, realize the adaptive weight allocation of features, and enable the model to more flexibly capture the correlation and importance between different features.

[0032] Three-branch decision fusion module: This module first decomposes the nature of the task into different influencing factors based on the improved hierarchical analysis method, and then formulates the judgment matrix between each layer according to the interaction and affiliation between the factors. The factor weight matrix is ​​calculated based on the judgment matrix. After passing the consistency test, the importance weights of the spectral branch, image branch and atlas fusion branch are calculated. The weights are multiplied and accumulated with the corresponding confidence to obtain the gangue score matrix. The score matrix is ​​used to identify and evaluate the gangue. By comprehensively considering the importance weights and confidence of different feature branches, the recognition accuracy and reliability of the model are improved. In this way, when the data quality collected by a certain modality is poor, the second modality can still maintain a high weight, thereby not affecting the recognition accuracy. When both modalities are highly affected, the fusion modality plays its leading role.

[0033] The data processing module pre-processes the collected spectral information to reduce the influence of equipment, environment and background, such as Figure 3 The specific components shown include the following: Black and white calibration: Use the collected spectral data of the white reference (polytetrafluoroethylene standard white plate) with a reflectivity close to 100% and the black reference (covered probe) with a reflectivity close to 0% to calibrate the spectral data, normalize the reflection intensity of the entire spectrum to a unified standard, and eliminate the brightness difference caused by uneven light source intensity and the interference of the instrument dark current. The relative reflectivity is specifically: Where R is the relative reflectivity of the sample, is the reflectance spectrum intensity of the sample, is the reflected spectrum intensity of the white reference, is the reflected spectrum intensity of the black reference.

[0034] Elimination of initial and final bands: Affected by instrument noise or environmental interference, the initial and final band data are removed, and the 500~850nm band spectrum is retained for analysis to improve the quality and accuracy of the data and make the analysis results more reliable.

[0035] SG smoothing: Using the Savitzky-Golay algorithm, based on local polynomial fitting, the spectrum is smoothed by performing a sliding window convolution operation, and a sliding window of appropriate size is selected. The window size is usually an odd number, and the noise and mutation in the spectral data are removed, the main trends and features of the data are retained, and the quality and readability of the data are improved. In the present invention, the polynomial fitting order is 1, that is, linear fitting, and the sliding window size is 21, that is, 10 data points are taken on the left and right sides of each data point for fitting. The sliding window size here is determined by the specific spectral curve, and the present invention does not make any requirements.

[0036] Standard normal vector transformation: This algorithm implements standard normal vector transformation by centering the mean and standardizing the standard deviation of the sample spectrum at each wavelength point.

[0037] Mean centering is to subtract the average value of all sample spectral data at that wavelength point to make the mean of the spectral data zero, and standard deviation standardization is to divide the data by its standard deviation to make the spectral data at each wavelength point have the same scale.

[0038] This method can eliminate the offset and amplitude difference of the overall spectrum, highlight the information related to the chemical composition of the sample, and effectively eliminate the influence of surface scattering, solid particle size and optical path change on the spectrum. The specific calculation is as follows: in, m is the number of wavelength points, k =1,2,3, …, m .

[0039] The feature extraction module extracts high-level features from spectral and image information based on each feature extraction branch, and specifically includes the following components: LSTM-CNN network: The spectral feature extraction branch LSTM-CNN is composed of a recurrent neural network and a convolutional neural network. Figure 4 As shown in the figure, the spectral information in the recurrent neural network branch passes through the LSTM layer, batch normalization layer, and expansion layer to obtain the spectral sequence features. The spectral information in the convolutional neural network branch passes through the one-dimensional convolution layer, the maximum pooling layer, and the Dropout layer to obtain the spectral convolution features. Then the two features are fused through the Concat operation and input into the second convolution layer and the Relu layer to finally obtain the spectral features.

[0040] MobilNetV2 network: The image feature extraction branch is implemented by the MobilNetV2 network. The feature extraction layer of the network consists of a convolution block, 17 inverted residual blocks and a cascade of convolution blocks. Each convolution block includes a two-dimensional convolution layer, a batch normalization layer and a Relu6 layer, and each inverted residual block consists of a lightweight dilated convolution layer, a depth-wise separable convolution layer and a point-by-point convolution layer, as shown in Figure 5 shown.

[0041] Feature matching module: The features extracted by the designed parallel dual-branch feature extraction network do not match in dimension and shape. The extracted spectral sequence features are converted into a two-dimensional matrix through transposition and multiplication operations, and then the channels are added through the convolution layer. The image features are downsampled and globally averaged pooled to achieve matching of spectral information and image information in space and channel dimensions, thereby achieving effective feature fusion.

[0042] The feature fusion module is based on the bilateral guided fusion module of the semantic segmentation network BiSeNetV2 to achieve efficient interaction between spectral and image information, and adds a self-attention mechanism within the module to better capture the correlation between features and enhance the model's ability to learn the association between different features. Figure 6 As shown, it specifically includes the following components: Bilateral guided fusion module: This module draws on the module in BiSeNetV2 that fuses the features of the detail branch and the semantic branch. The input spectral and image features are input into the two convolution branches respectively. The two features are processed through the convolution layer, batch normalization layer, and convolution layer to obtain the deeper spectral feature 1 and image feature 1. On the other hand, the spectral feature is processed through the convolution layer and batch normalization layer to obtain spectral feature 2, and the image feature is processed through the convolution layer, batch normalization layer, and average pooling layer to obtain image feature 2. The spectral feature 2 and the image feature 2 are processed through the self-attention mechanism to obtain the weight distribution of the original features in the fusion feature space. The weight distribution is multiplied and added with the spectral feature 1 and the image feature 2 to obtain the atlas fusion feature map.

[0043] Self-attention mechanism: This mechanism realizes the allocation of weights of spectral features and image features in the fusion feature space, so that the model can better learn the correlation between features. Spectral feature 2 is multiplied and fused with image feature 2 to obtain fusion feature 1, and spectral feature 2 and image feature 2 are allocated and multiplied with fusion feature 1, and then concatenated. Normalization is performed in the global feature, and the normalized feature weights are multiplied with spectral feature 1 and image feature 1 respectively. Finally, the modal weight features under the guidance of the fusion feature space are obtained, realizing the self-attention mechanism when fusing the two modal features.

[0044] The three-branch decision fusion module realizes three-branch gangue identification by inputting the spectral feature extraction branch, image feature extraction branch and feature fusion branch into the classifier. The weights of each classifier are calculated based on the hierarchical analysis method and prior knowledge to realize the decision fusion identification of gangue. It specifically includes the following components: Classifier: The spectral branch classifier consists of a fully connected layer and a Sigmoid layer. The image branch classifier is implemented by an adaptive average pooling layer and a 1*1 convolution layer. The fusion branch classifier is composed of two fully connected layers, and the activation function is Relu.

[0045] Prior knowledge: According to the task requirements, subjective judgment combined with expert opinions and data analysis results, a judgment matrix between the target layer and the indicator layer is constructed. According to the performance indicators of each model in the spectral data set, image data set, and atlas data set, the importance of the factors of the same level relative to the factors of the previous level is quantitatively expressed, and a judgment matrix between the indicator layer and the solution layer is constructed. The evaluation indicators selected by this method are accuracy, F1 value, and root mean square error. The accuracy rate measures the proportion of the model's correct classification in all predicted samples. The F1 value is a comprehensive consideration of the model's precision and recall rate. In addition, the root mean square error between the probability of the class to which the predicted sample belongs and the true class is used as an indicator of model reliability. The number and type of evaluation indicators are not required and are set according to the task requirements.

[0046] AHP: AHP decomposes a complex decision-making problem into a series of relatively simple hierarchical structures, and then compares and evaluates the hierarchical structures to obtain the relative weights of the bottom-level solutions to the top-level goals. The steps include constructing a hierarchical structure, constructing a judgment matrix, weight calculation, consistency test and comprehensive decision-making, such as Figure 8 shown.

[0047] Analyze the model requirements for underground coal gangue identification and set the hierarchical model of this task as follows Figure 7 As shown in the figure, the goal is to identify underground coal gangue, and the evaluation indicators include accuracy, F1 value and root mean square error. The solutions are spectral classifier, image classifier and fusion classifier.

[0048] Then, the importance of the two factors of the indicator layer relative to the task of coal gangue separation is compared, and the judgment matrix between the target layer and the indicator layer is constructed as follows: , where the judgment matrix element D ij It indicates the importance of the factor in the i-th row relative to the factor in the j-th column. The larger the value, the more important the factor is. The row and column factors are accuracy, F1 value and root mean square error respectively. This means that in this task, we believe that accuracy is the most important indicator, followed by F1 value, and root mean square error is the least important.

[0049] In order to avoid subjective assumptions, we have formulated quantitative provisions for the judgment matrix between the indicator layer and the solution layer, as shown in the following table: Quantitative value regulations of judgment matrix at indicator level and solution level Where A is a set of evaluation indicators under the three classifiers. and is the minimum and maximum value in the set, and m is .

[0050] According to the above judgment matrix quantification regulations, the judgment matrices between the solution layers under the three indicators are constructed in turn, and then the factor weights are calculated by the four judgment matrices. This method uses the eigenvalue method for calculation. The normalized eigenvector corresponding to the maximum eigenvalue of the judgment matrix is ​​the factor weight matrix. The column factor weight matrix of the target layer and the indicator layer constitutes the first column of the final weight matrix, and the three row factor weight matrices of the indicator layer and the solution layer constitute the last three columns of the final weight matrix, where the first column represents the weight of the indicator for the task, and the last three columns represent the weights of each classifier for the indicator.

[0051] The rationality of the judgment matrix setting is verified by consistency test. When the consistency ratio CR is less than 0.1, the consistency test is passed, indicating that the judgment matrix setting is reasonable. Otherwise, the judgment matrix needs to be corrected. The calculation method is shown in the following formula in is the maximum eigenvalue of the judgment matrix, n is the order of the judgment matrix, and the consistency index is obtained by looking up the following table: After the consistency check, the first column of the final weight matrix is ​​multiplied and added with the corresponding data in the last three columns to obtain the weights of each classifier.

[0052] When applied, the category confidence matrices under the three classifiers obtained by the three-branch neural network are multiplied and added with the corresponding classifier weights to obtain the final prediction score of coal or gangue, thereby realizing coal and gangue identification.

[0053] The coal gangue identification method provided by the present invention realizes coal gangue identification by performing feature fusion and decision fusion on the reflection spectrum and image collected by a computer using a spectrometer and a camera. Feature fusion improves the accuracy of coal gangue identification, and decision fusion improves the stability of coal gangue identification. In addition, high-quality interaction of the two modes of the spectrum is achieved by building a fusion module, and the classifier weights are objectively and scientifically calculated by improving the hierarchical analysis method, effectively combining multiple classifiers, and enhancing robustness, which has a good recognition effect on coal gangue sorting and abnormal working conditions in harsh environments.

[0054] The above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, improvements, additions or substitutions made by technicians in this technical field within the essential scope of the present invention also fall within the protection scope of the present invention.

Claims

1. A coal gangue identification method based on visible-near infrared spectrum and image multimodal information fusion, characterized in that: include: S1: Collect the spectral information and image information of the sample to be identified; S2: Preprocess the spectral information and image information respectively, and construct a spectrum data set with one spectrum and one image. The spectrum data set includes the spectral data set and the image data set to obtain the model prior knowledge; S3: Extract spectral features from the spectral dataset using the spectral feature extraction neural network model; extract image features from the image dataset using the image feature extraction neural network model; S4: The extracted spectral features and image features are input into the two-stream fusion network to output a comprehensive feature; S5: respectively inputting the extracted spectral features, image features and comprehensive features into the spectral branch classifier, the image branch classifier and the fusion branch classifier to obtain the category confidence under the three classifiers; S6: Calculate the importance weights of the spectrum branch, image branch and atlas fusion branch; S7: The importance weights are multiplied and added with the corresponding confidence levels to obtain the gangue scoring matrix, and the scoring matrix is ​​used to realize gangue identification.

2. The coal gangue identification method based on visible-near infrared spectrum and image multimodal information fusion according to claim 1 is characterized in that: In the step S1, a fiber optic probe is used to collect spectral information of the sample surface. Before collecting the spectral information of the sample, the halogen lamp and the spectrometer are turned on for preheating. The spectral information of the polytetrafluoroethylene white board and the probe when covered is first collected for data processing. The spectral information of the front, middle and tail of each sample is collected, and the average spectrum of the three positions is taken as the spectral information of the sample; An industrial lens is used to collect image information of the sample, and the image information corresponds one-to-one with the spectral information.

3. The coal gangue identification method based on visible-near infrared spectrum and image multimodal information fusion according to claim 1 is characterized in that: In the step S2, Spectral information preprocessing includes: It is processed in sequence by the existing black and white correction, initial and final band elimination, SG smoothing and standard normal vector transformation techniques; Image information preprocessing includes: Uniformly resize images, convert resized images to tensor format, and normalize images.

4. The coal gangue identification method based on visible-near infrared spectrum and image multimodal information fusion according to claim 1 is characterized in that: In the step S2, the preprocessed spectral features and image features are matched one by one and encapsulated in a TensorDataset together with the labels.

5. The coal gangue identification method based on visible-near infrared spectrum and image multimodal information fusion according to claim 1 is characterized in that: In the step S3, The spectral feature extraction neural network model includes: A recurrent neural network branch, wherein the recurrent neural network branch includes: an LSTM layer, a batch normalization layer, and an expansion layer, and the spectral information is sequentially expanded through the LSTM layer and the batch normalization layer to obtain spectral sequence features; A convolutional neural network branch, wherein the convolutional neural network branch comprises: a one-dimensional convolution layer, a maximum pooling layer and a Dropout layer, and the spectral information is passed through the one-dimensional convolution layer, the maximum pooling layer and the Dropout layer to obtain spectral convolution features; The spectral sequence features and spectral convolution features are fused through the Concat operation and input into the second convolution layer and Relu layer to obtain the spectral features.

6. The coal gangue identification method based on visible-near infrared spectrum and image multimodal information fusion according to claim 1 is characterized in that: In the step S3, The image feature extraction neural network model consists of a convolution block, 17 inverted residual blocks and a cascade of convolution blocks, each convolution block includes a two-dimensional convolution layer, a batch normalization layer and a Relu6 layer, and each inverted residual block includes a lightweight dilated convolution layer, a depth-wise separable convolution layer and a point-by-point convolution layer.

7. The coal gangue identification method based on visible-near infrared spectrum and image multimodal information fusion according to claim 1 is characterized in that: In step S4, the dual-stream fusion network includes: A first convolution branch, wherein the first convolution branch comprises: a convolution layer, a batch normalization layer and a convolution layer, the spectral feature is input into the first convolution branch, and the spectral feature 1 is output; A second convolution branch, wherein the second convolution branch comprises: a convolution layer, a batch normalization layer and a convolution layer, the image feature is input into the second convolution branch, and the image feature 1 is output; A third convolution branch, the third convolution branch comprising: a convolution layer and a batch normalization layer; the spectral feature is input into the third convolution branch, and the spectral feature 2 is output; A fourth convolution branch, wherein the fourth convolution branch comprises: a convolution layer, a batch normalization layer and an average pooling layer, the image feature is input into the fourth convolution branch, and the image feature 2 is output; Self-attention mechanism, spectral feature 2 and image feature 2 are subjected to the self-attention mechanism to obtain the weight distribution of the original features in the fusion feature space; The spectral feature 2 is multiplied and fused with the image feature 2 to obtain the fused feature 1. The spectral feature 2 and the image feature 2 are respectively multiplied and fused with the fused feature 1, and then a splicing operation is performed. Normalization is performed in the global feature, and the normalized feature weights are respectively multiplied with the spectral feature 1 and the image feature 1 to obtain the spectral weight feature and the image weight feature under the guidance of the fused feature space. The two-modal weight features are added and fused to obtain the comprehensive feature, and then a convolution process is performed to obtain a deeper comprehensive feature.

8. The coal gangue identification method based on visible-near infrared spectrum and image multimodal information fusion according to claim 1 is characterized in that: In step S5: The spectral branch classifier includes: fully connected layer and Sigmoid layer; The image branch classifier includes: adaptive average pooling layer and 1*1 convolution layer; The fusion branch classifier includes: two fully connected layers, and the activation function is Relu.

9. The coal gangue identification method based on visible-near infrared spectrum and image multimodal information fusion according to claim 1 is characterized in that: The step S6 comprises: Based on the atlas dataset, a neural network model consisting of the above-mentioned spectral feature extraction neural network model and image feature extraction neural network model, a two-stream fusion network, a spectral branch classifier, an image branch classifier, and a fusion branch classifier is trained to obtain the accuracy, F1 value, and root mean square error of the spectral branch classifier, the image branch classifier, and the fusion branch classifier, which are used as prior knowledge; According to the requirements and scenarios of gangue sorting, the target layer, indicator layer and scheme layer are constructed. The target layer is gangue sorting, the indicator layer is classification accuracy, F1 value and root mean square error, and the scheme layer is spectral classifier, image classifier and fusion classifier. According to the task scenario and requirements, the importance of each indicator in the indicator layer relative to the task is judged, and the judgment matrix between the target layer and the indicator layer is formulated, where the judgment matrix element D ij It indicates the importance of the i-th row factor relative to the j-th column factor. The larger the value, the more important the factor is. The row and column factors are accuracy, F1 value and root mean square error respectively. Finally, a 3×3 judgment matrix is ​​obtained. Based on the prior knowledge, namely the accuracy, F1 value and root mean square error of the spectral classifier, image classifier and fusion classifier, the maximum and minimum values ​​of each indicator are evenly divided into 9 intervals. The closer the classifier's indicator is to the optimal value, the larger its quantization value is, indicating the importance of the indicator of the i-th row classifier relative to the j-th column classifier. The row and column classifiers represent the spectral classifier, image classifier and fusion classifier respectively. The above operations are performed on each indicator, and finally three 3×3 judgment matrices are obtained; The factor weight matrix between the target layer and the indicator layer, and between the indicator layer and the solution layer is calculated using the eigenvalue method according to the judgment matrix, and the final classifier weight matrix is ​​formed by the factor weight matrix; Check whether the judgment matrix passes the consistency test. If it passes, the importance weight of each classifier is obtained by multiplying and adding the first column of the final weight matrix with the last three columns. If it fails, it proves that the judgment matrix is ​​set incorrectly and the judgment matrix is ​​reconstructed.

10. The coal gangue identification method based on visible-near infrared spectrum and image multimodal information fusion according to claim 1, characterized in that: The step S7 comprises: The three category confidence matrices output by the spectral classifier, image classifier and fusion classifier are multiplied and added with the classifier weight matrix to finally obtain the scores predicted as coal and gangue, thereby realizing coal and gangue identification.

Images