Small sample infrared thermal imaging coal rock character recognition method based on generative adversarial network
By building an infrared thermal imaging experimental bench on a coal miner, high-quality data is generated using generative adversarial network and adaptive feature fusion technology, the problem of small sample coal rock trait recognition is solved, and efficient identification of coal rock traits and coal mining machine is realized. The intelligent operation of coal mining machines supports the "unmanned" mining.
Patent Information
- Application Number
- CN202410912665.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-09
- Publication Date
- 2024-11-08
AI Technical Summary
During the working process of the coal miner, due to the complex distribution of the traits of the coal wall and the single cutting mode of the coal miner, it is difficult to obtain sufficient data for deep learning model training, resulting in low accuracy of coal rock trait recognition under small samples and cannot be effectively supported Intelligent operation of coal mining machines.
Using a small sample infrared thermal imaging coal lithologic trait recognition method based on a generative adversarial network, an improved WGAN-GP model and 2D-CNN recognition network are constructed by installing infrared thermal cameras and data processing equipment on the coal mining machine, and using adaptive feature fusion The conditional random field technology generates high-quality infrared thermal imaging pictures, expands the data set for model training, and realizes automatic identification of coal rock traits.
Under small sample conditions, efficient identification of coal rock traits is achieved, the intelligence level of coal mining machines is improved, the stable operation and safety of coal mining machines is enhanced, and the "unmanned" or "less-manned" mining is supported.
Smart Images

Figure CN118918432A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of coal-rock intelligent identification, and specifically relates to a small sample infrared thermal imaging coal-rock property identification method based on a generative adversarial network. Background Art
[0002] As a key equipment in fully mechanized mining, the intelligence of coal mining machines is a necessary condition for realizing unmanned mining. At present, the distribution of coal and rock properties of coal walls frequently changes, which makes the operation state of coal mining machines in the cutting process easy to deteriorate, leading to reduced cutting efficiency, equipment failure, and even accidents causing casualties. Therefore, it is urgent to ensure the efficient and stable operation of coal mining machines, and the primary key technology for the stable operation of coal mining machines is the effective identification of coal and rock properties.
[0003] Considering that the temperature of the coal wall will change significantly before and after the coal wall is cut during the operation of the coal mining machine, and the temperature values of different coal and rock properties are different, the thermal distribution information during the coal wall cutting process can be obtained through infrared thermal imaging technology to realize the recognition of coal and rock properties. At present, the ideal data coal and rock property recognition method based on deep learning models such as machine learning, convolutional neural networks and stacked autoencoders has achieved remarkable results. However, due to the potential safety hazards of the fully mechanized mining face and the single cutting mode of the coal mining machine, it is usually difficult to obtain sufficient available coal and rock data to train the deep learning recognition model. Therefore, the research on coal and rock property recognition under small sample conditions is more realistic.
[0004] Generative adversarial network (GAN) provides a solution to the problem of small sample recognition from the perspective of data generation, and has good portability and popularity. However, the combination of generative adversarial network and infrared thermal imaging in the field of recognition is still in its infancy, and still cannot solve the problem of effective recognition of coal and rock properties. Therefore, it is urgent to provide a small sample infrared thermal imaging coal and rock property recognition method based on generative adversarial network to effectively solve the problem of coal and rock property recognition in small sample conditions. Summary of the invention
[0005] In response to the problems existing in the above-mentioned prior art, the present invention provides a small sample infrared thermal imaging coal and rock property identification method based on a generative adversarial network. The method has simple implementation steps, high recognition accuracy, and a high degree of automation in the recognition process. It can efficiently complete the recognition of coal and rock properties under small sample conditions, and can provide reliable technical support for the intelligent and stable operation of the coal mining machine.
[0006] In order to achieve the above object, the present invention provides a small sample infrared thermal imaging coal and rock property recognition method based on a generative adversarial network, comprising the following steps:
[0007] Step 1: Install an infrared thermal camera on the shearer rocker arm, establish a communication connection between the infrared thermal camera and the data processing equipment, and use the shearer rocker arm, infrared thermal camera and data processing equipment to build a shearer coal wall cutting test bench;
[0008] Step 2: According to the order of fully mechanized mining work, the coal wall test blocks with different properties are cut in turn, and the infrared thermal camera is used to collect infrared thermal imaging pictures of the cutting area of the coal mining machine under different coal and rock properties at set time intervals, and send them to the data processing equipment;
[0009] Step 3: The data processing device pre-processes the received infrared thermal imaging images, and then randomly divides the processed infrared thermal imaging images into a small sample training set and a normal number of test sets;
[0010] Step 4: Build an improved WGAN-GP model based on dual discriminators and adaptive feature fusion, and use a small sample training set to train the improved WGAN-GP model to generate high-quality infrared thermal imaging images;
[0011] Step 5: Select 2D-CNN as the recognition network model, use the generated high-quality infrared thermal imaging images and a small sample training set to construct an expanded data set, and then train the recognition network model through the expanded data set to obtain a trained coal and rock property recognition network model. Then, use a normal number of test sets to test the trained coal and rock property recognition model.
[0012] Step 6: During the cutting operation of the coal wall test block, the infrared thermal imaging images collected in real time are input into the trained coal and rock property recognition network model, and the coal and rock property classification results are output to achieve effective recognition of coal and rock properties under small sample conditions.
[0013] Furthermore, in order to improve the robustness of model training and the diversity of generated data, so as to generate higher quality images and further more effectively assist the coal and rock identification process under small sample conditions, in step 4, the construction process of the improved WGAN-GP model is as follows:
[0014] A1: Set up a dual discriminator network;
[0015] A11: Using WGAN-GP as the basic generative model, a dual discriminator network with different discriminant criteria is set; one discriminator D1 uses real samples as the discrimination criterion, the first input of the discriminator D1 is the infrared thermal imaging picture, and the second input is the generated sample obtained by the generator network G based on random noise z; the other discriminator D2 uses the reconstructed sample extracted by the feature encoder as the discrimination criterion, the first input of the discriminator D2 is the reconstructed sample obtained by the feature encoder based on the infrared thermal imaging picture, and the second input is the generated sample obtained by the generator network G based on random noise z; at the same time, the generator loss function of the WGAN-GP model is obtained according to formula (1), and the discriminator loss function of the WGAN-GP model is obtained according to formula (2);
[0016]
[0017] In the formula, x represents the probability distribution P r(x) A real sample of is the result of random interpolation sampling of random noise between real samples and generated samples, and λ×GP is the gradient penalty term;
[0018] A12: The generator loss function of the dual discriminator network is derived by formula (1), as shown in formula (3);
[0019]
[0020] In the formula, ω is the adaptive weight of model training, Its value ranges from 0 to 1; D1(G(z)) is the probability that the discriminator D1 distinguishes between real samples and fake samples, D2(G(z)) is the probability that the discriminator D2 distinguishes between real samples and reconstructed samples, and e is a natural base;
[0021] A2: Adaptive feature fusion mechanism;
[0022] A21: Introduce an adaptive feature fusion mechanism. First, use the feature encoder to extract feature information from the infrared thermal imaging image, and input the feature information into the U-Net reconstruction network. Then, use the U-Net reconstruction network to extract the intermediate feature F. Then, use the fusion mechanism to adaptively superimpose the random noise z and the intermediate feature F to obtain the output m of the feature fusion mechanism, as shown in formula (4). Then, input the output m of the feature fusion mechanism into the generator network G to obtain the generated sample, and then use the generated sample as the second input of the discriminator D1 and the second input of the discriminator D2. At the same time, use the conditional random field CRF to post-process the output of the U-Net reconstruction network to obtain the reconstructed sample, and then use the reconstructed sample as the first input of the discriminator D2.
[0023] m = Concat (λF(x), z) + z (4);
[0024] In the formula, z is random noise, F(x) is the intermediate feature of the feature encoder, λ is the feature fusion weight, which gradually decreases with the increase of the number of iterations, and Concat is the merging operation;
[0025] A22: Combining formula (3) and formula (4), the generator loss function after introducing the adaptive feature fusion mechanism is obtained, as shown in formula (5);
[0026]
[0027] In the formula, m represents the probability distribution P m Fusion samples of
[0028] A23: Based on formula (5), the sample reconstruction loss term is introduced to obtain a new generator loss function, as shown in formula (6);
[0029]
[0030] In the formula, |G(m)-x True 2 Represents the reconstruction loss of the sample.
[0031] Furthermore, in order to improve the accuracy of sample generation, in A21, the process of post-processing the output of the U-Net reconstruction network using the conditional random field CRF is as follows:
[0032] The conditional random field method is used to post-process the image generated by the U-Net reconstruction network, so that the boundary area between the target and the background is more accurate when generated; assuming C = {c1, c2, ···, c N} represents the N categories generated by the real image I, and the probability undirected graph of the generated results is D = {d1, d2, ···, d N}, and the essence of image generation is based on the real image I. The conditional probability of all pixels in I belonging to which category is P(D=c k |I); According to graph theory, the probability undirected graph corresponding to D satisfies the global Markov property, and the conditional random field P(D|I) is obtained according to formula (7);
[0033]
[0034] Where Z(I) is the normalization factor, Among them, ψ1(d j ) is a univariate potential function, which represents the characteristics, structure, position, color, texture and gradient information in the image; ψ2(d j ,d k) is a binary potential function, representing the influence of the mutual relationship between pixel points on image generation. Among them, each is the feature f extracted from pixels j and k, and is weighted by the parameter ω g to obtain the vector f j and f k which are feature vectors of any dimension at the positions of pixels j and k respectively; ω g is the weight of the linear combination term. Among them, Q j and Q k are the pixel values of the j-th and k-th pixels respectively, and E j and E k are the positions of the j-th and k-th pixels in the real image I respectively. θ, β, and δ are all constants.
[0035] Furthermore, in order to ensure the quality of the obtained infrared thermal imaging pictures, in step four, the process of generating high-quality infrared thermal imaging pictures using the improved WGAN-GP model is as follows:
[0036] B1: Model input and parameter setting; using a finite infrared thermal imaging picture and random noise z as the model input, setting the coefficient α of the generator, the coefficient β of the discriminator, the learning rate η D of the generator, the learning rate η G of the discriminator, the maximum number of iterations N D of the generator, and the maximum number of iterations N G of the discriminator respectively;
[0037] B2: Parameter initialization; setting the number of iterations N = 0 and the intermediate feature F = 0;
[0038] B3: Adaptive feature fusion; using a fusion mechanism to adaptively superimpose the random noise z and the intermediate feature F, and the specific process is as follows:
[0039] B31: First, use a feature encoder to extract the feature information in a finite infrared thermal imaging picture, input the feature information into a U-Net reconstruction network, then use the U-Net reconstruction network to extract the intermediate feature F, and divide the intermediate feature F into {F(x)丨F(1), F(2),…F(n)};
[0040] B32: Perform a feature fusion operation using formula (4) to obtain the output m of the feature fusion mechanism;
[0041] B33: Update the loss function L D of the discriminator and the loss function L G of the generator respectively using formulas (2) and (6);
[0042] B34: If the iteration termination condition is reached, output the result, otherwise repeat B33;
[0043] B4: Sample generation; the generator network G uses upsampling to restore the fused features to obtain generated samples, and uses conditional random fields CRF for post-processing operations;
[0044] B5: Sample discrimination: Output the generated samples to the discriminator D1 and the discriminator D2 respectively, and use the discriminator D1 and the discriminator D2 to judge whether they are true samples or false samples;
[0045] B6: If the model training reaches the maximum number of iterations N D 、N G , output the infrared thermal imaging image generated by the dual discriminator and the adaptive feature fusion WGAN-GP model, otherwise re-execute B3 until the maximum number of model iterations is met.
[0046] Furthermore, in order to adapt to the classification requirements of coal rock properties, in step five, the classifier structure of the coal rock property recognition network model includes 2 two-dimensional convolutional layers, 1 maximum pooling layer and 2 output classification layers, and the activation function is ReLu6, the optimizer is Adam, the batch-size is 2, and the epochs is 100.
[0047] In the present invention, by building a coal mining machine coal wall cutting test bench, the infrared thermal camera installed on the rocker arm can be used to collect infrared thermal imaging pictures in the coal mining machine cutting area under different coal and rock properties, providing reliable data support for the subsequent acquisition of the coal and rock property recognition network model; using data processing equipment to divide a number of infrared thermal imaging pictures into a small sample training set and a normal number of test sets of the recognition network, it is convenient to use the small sample training set to train the improved WGAN-GP model, and can further use the generated high-quality infrared imaging pictures and small sample training sets to train the recognition network to obtain the coal and rock property recognition network model; using WGAN-GP as the basic generation model, designing a dual discriminator network structure with different discrimination criteria to optimize the model training strategy, can ensure the improvement of model training robustness and generated data diversity. Since the adaptive feature fusion mechanism is constructed to improve the problem of oscillation that is easy to generate in the early stage of model training, the accuracy of the sample generation boundary can be effectively improved by introducing conditional random fields for post-processing operations. Using a small sample training set to train the constructed dual discriminator and adaptive feature fusion WGAN-GP model can generate high-quality infrared thermal imaging images, and then construct an expanded data set; then, using the expanded data set to train the recognition network model can ensure that the coal rock property recognition network model can quickly and accurately output the classification results of different coal rock properties, and thus realize the coal rock property recognition under small samples.
[0048] In summary, based on infrared thermal imaging classification technology, the present invention proposes a small sample coal and rock property identification method based on a dual discriminator and an adaptive feature fusion WGAN-GP model, which can solve the problem of low accuracy in coal and rock property identification caused by insufficient sample collection and uneven distribution due to potential safety hazards in the fully-mechanized mining face and the single cutting mode of the coal mining machine, thereby greatly improving the intelligence level of the coal mining machine, which is of great significance to the realization of "unmanned" or "few-man" mining of the fully-mechanized mining face. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 is a flow chart of the present invention;
[0050] Figure 2 It is a framework diagram of the traditional WGAN-GP model in the present invention;
[0051] Figure 3 It is a framework structure diagram of the WGAN-GP model based on dual discriminators and adaptive feature fusion in the present invention;
[0052] Figure 4 It is a framework diagram of the dual discriminator network in the present invention;
[0053] Figure 5 It is a framework diagram of the adaptive feature fusion mechanism in the present invention;
[0054] Figure 6 It is a framework diagram of the coal and rock property identification network model in the present invention. DETAILED DESCRIPTION
[0055] The present invention will be further described below in conjunction with examples, but the present invention is not limited to the following examples.
[0056] like Figures 1 to 6 As shown, the present invention provides a method for identifying coal and rock properties by infrared thermal imaging of small samples based on a generative adversarial network, comprising the following steps:
[0057] Step 1: Install an infrared thermal camera on the shearer rocker arm, establish a communication connection between the infrared thermal camera and the data processing equipment, and use the shearer rocker arm, infrared thermal camera and data processing equipment to build a shearer coal wall cutting test bench;
[0058] Since the temperature of the coal mining machine will change significantly before and after cutting the coal wall, the temperature field in the cutting area can directly reflect the coal rock properties (i.e., what hardness of the coal wall the coal mining machine is currently cutting). In addition, due to the limitations of experimental conditions and site space, the maximum hardness of the coal wall cut by the coal mining machine does not exceed the Pu's coefficient f = 7, and the four typical coal rock cutting conditions are Pu's coefficients f = 1, f = 3, f = 5, and f = 7.
[0059] Step 2: According to the order of fully mechanized mining work, the coal wall test blocks with different properties are cut in turn, and the infrared thermal camera is used to collect infrared thermal imaging pictures of the cutting area of the coal mining machine under different coal and rock properties at set time intervals, and send them to the data processing equipment;
[0060] Step 3: The data processing device pre-processes the received infrared thermal imaging images, and then randomly divides the processed infrared thermal imaging images into a small sample training set and a normal number of test sets;
[0061] Step 4: Construct an improved WGAN-GP (Wasserstein Generative Adversarial Network with Gradient Penalty) model based on dual discriminators and adaptive feature fusion, and use a small sample training set to train the improved WGAN-GP model to generate high-quality infrared thermal imaging images to assist in the identification of coal and rock properties under small sample conditions;
[0062] Step 5: Select 2D-CNN as the recognition network model, use the generated high-quality infrared thermal imaging images and a small sample training set to construct an expanded data set, and then train the recognition network model through the expanded data set to obtain a trained coal and rock property recognition network model. Then, use a normal number of test sets to test the trained coal and rock property recognition model.
[0063] Step 6: During the cutting operation of the coal wall test block, the infrared thermal imaging images collected in real time are input into the trained coal and rock property recognition network model, and the coal and rock property classification results are output to achieve effective recognition of coal and rock properties under small sample conditions.
[0064] In order to improve the robustness of model training and the diversity of generated data, so as to generate higher quality images and further more effectively assist the coal and rock identification process under small sample conditions, in step 4, the construction process of the improved WGAN-GP model is as follows:
[0065] A1: Set up a dual discriminator network;
[0066] A11: With WGAN-GP as the basic generative model, the discriminator can only distinguish which distribution a single sample comes from, causing the generative network to tend to generate high-quality samples in local intervals while ignoring global correlation features. To this end, a dual discriminator network with different discriminant criteria is set to optimize the model training strategy, and an additional loss term is added to the generator's loss function to improve the model training robustness and the diversity of generated data. One discriminator D1 uses real samples as the discrimination criterion. The first input of the discriminator D1 is an infrared thermal imaging picture, and the second input is the generated sample obtained by the generator network G based on random noise z; the other discriminator D2 uses the reconstructed sample extracted by the feature encoder as the discrimination criterion. The first input of the discriminator D2 is the reconstructed sample obtained by the feature encoder based on the infrared thermal imaging picture, and the second input is the generated sample obtained by the generator network G based on random noise z;
[0067] At the same time, in the early stage of model training, the feature encoder can only extract the significant features of the real sample, while many difficult-to-extract detail features are not fully learned. At this time, the reconstructed sample can be considered a downgraded version of the real sample. The addition of a "tolerant" discriminator will greatly reduce the difficulty of training the generator, help the generator gain an advantage in the confrontation, and make the generated samples reach the standard of "real" data, thereby improving the smoothness and convergence of the model training process. The generator loss function of the WGAN-GP model is obtained according to formula (1), and the discriminator loss function of the WGAN-GP model is obtained according to formula (2);
[0068]
[0069] In the formula, x represents the probability distribution P r(x) A real sample of is the result of random interpolation sampling of random noise between real samples and generated samples, and λ×GP is the gradient penalty term;
[0070] A12: The generator loss function of the dual discriminator network is derived by formula (1), as shown in formula (3);
[0071]
[0072] Where ω is the adaptive weight of model training, Its value ranges from 0 to 1; D1(G(z)) is the probability that the discriminator D1 distinguishes between real samples and fake samples, D2(G(z)) is the probability that the discriminator D2 distinguishes between real samples and reconstructed samples, and e is a natural base;
[0073] A2: Adaptive feature fusion mechanism;
[0074] In order to further improve the problem of oscillation that is prone to occur in the early stage of model training, the random noise z is fused with the intermediate feature F of the feature encoder through an adaptive feature fusion mechanism; specifically, in the early stage of model training, the fused features guide the generator to learn, and as the number of iterations increases, the intermediate features extracted by U-Net are gradually eliminated, and finally only the random noise is used as the input of the generator. This operation solves the problem of large fluctuations in model training on the one hand, and on the other hand, it also makes up for the problem of low diversity of generated samples caused by a single input structure. The specific process is as follows:
[0075] A21: Introduce an adaptive feature fusion mechanism. First, use the feature encoder to extract feature information from the infrared thermal imaging image, and input the feature information into the U-Net reconstruction network. Then, use the U-Net reconstruction network to extract the intermediate feature F. Then, use the fusion mechanism to adaptively superimpose the random noise z and the intermediate feature F to obtain the output m of the feature fusion mechanism, as shown in formula (4). Then, input the output m of the feature fusion mechanism into the generator network G to obtain the generated sample, and then use the generated sample as the second input of the discriminator D1 and the second input of the discriminator D2. At the same time, use the conditional random field CRF to post-process the output of the U-Net reconstruction network to obtain the reconstructed sample, and then use the reconstructed sample as the first input of the discriminator D2.
[0076] m = Concat (λF(x), z) + z (4);
[0077] In the formula, z is random noise, F(x) is the intermediate feature of the feature encoder, λ is the feature fusion weight, which gradually decreases with the increase of the number of iterations, and Concat is the merging operation;
[0078] A22: Combining formula (3) and formula (4), the generator loss function after introducing the adaptive feature fusion mechanism is obtained, as shown in formula (5);
[0079]
[0080] In the formula, m represents the probability distribution P m Fusion samples of
[0081] A23: Based on formula (5), the sample reconstruction loss term is introduced to obtain a new generator loss function, as shown in formula (6);
[0082]
[0083] In the formula, |G(m)-x True 2 Represents the reconstruction loss of the sample.
[0084] In order to improve the accuracy of sample generation, in A21, the process of post-processing the output of the U-Net reconstruction network using the conditional random field CRF is as follows:
[0085] In the training process of the above-mentioned generative adversarial network, the upsampling operation in the Generator and U-Net reconstruction network easily leads to the misclassification of the generated image boundaries and unclear boundaries. Considering that the conditional random field in the probabilistic graph model can enhance the image interpretation performance by predicting a large number of unlabeled pixels, it forms a good complement. Therefore, the conditional random field method is used to post-process the image generated by the U-Net reconstruction network, so as to make the boundary area between the target and the background more accurate during generation; assuming C = {c1,c2,···,c N} represents the N categories generated by the real image I, and the probability undirected graph of the generated results is D = {d1, d2, ···, d N}, and the essence of image generation is based on the real image I. The conditional probability of all pixels in I belonging to which category is P(D=c k |I); According to graph theory, the probability undirected graph corresponding to D satisfies the global Markov property, and the conditional random field P(D|I) is obtained according to formula (7);
[0086]
[0087] Where Z(I) is the normalization factor, Among them, ψ1(d j ) is a univariate potential function, which represents the information of the image, such as the shape, structure, position, color, texture and gradient; ψ2(d j ,d k ) is a binary potential function, which represents the influence of the relationship between pixels on image generation. Among them, each are the features f extracted from pixels j and k, and are determined by the parameter ω g Weighted, vector f j and f k are feature vectors of arbitrary dimensions at pixel j and k respectively; ω g is the weight of the linear combination term; for The above binary potential function has a value only when the labels assigned to j and k are different. If the labels assigned to the two pixels are the same, the value of the binary potential function is zero.
[0088] Among them, Q j and Q k are the pixel values of the jth and kth pixels respectively, E j and E kare the positions of the j-th and k-th pixels in the real image I respectively, and θ, β, and δ are all constants. Through equation Calculate the information such as the pixel category and color position around a certain pixel to obtain the binary potential function of the pixel, and use the minimization operation of this potential function to optimize the image generation result, so that the boundary region between the target and the background is more accurate during generation.
[0089] To ensure the quality of the obtained infrared thermal imaging pictures, in step four, the process of generating high-quality infrared thermal imaging pictures using the improved WGAN-GP model is as follows:
[0090] B1: Model input and parameter setting; Use the finite infrared thermal imaging pictures and random noise z as the model input, and set the coefficient α of the generator, the coefficient β of the discriminator, and the learning rate η of the generator D , the learning rate η of the discriminator G , the maximum number of iterations N of the generator D , the maximum number of iterations N of the discriminator G ;
[0091] B2: Parameter initialization; Let the number of iterations N = 0, and the intermediate feature F = 0;
[0092] B3: Adaptive feature fusion; Use the fusion mechanism to adaptively superimpose the random noise z and the intermediate feature F, and the specific process is as follows:
[0093] B31: First, use the feature encoder to extract the feature information in the finite infrared thermal imaging pictures, and input the feature information into the U-Net reconstruction network. Then, use the U-Net reconstruction network to extract the intermediate feature F, and divide the intermediate feature F into {F(x)丨F(1), F(2),…F(n)};
[0094] B32: Perform feature fusion operations using formula (4) to obtain the output m of the feature fusion mechanism;
[0095] B33: Use formulas (2) and (6) to update the loss function L of the discriminator D and the loss function L of the generator G ;
[0096] B34: If the iteration termination condition is reached, output the result, otherwise repeat B33;
[0097] B4: Sample generation; The generator network G uses upsampling to restore the fused features to obtain the generated samples, and performs post-processing operations using the conditional random field CRF;
[0098] B5: Sample discrimination: Output the generated samples to the discriminator D1 and the discriminator D2 respectively, and use the discriminator D1 and the discriminator D2 to judge whether they are true samples or false samples;
[0099] B6: If the model training reaches the maximum number of iterations N D 、N G , output the infrared thermal imaging image generated by the dual discriminator and the adaptive feature fusion WGAN-GP model, otherwise re-execute B3 until the maximum number of model iterations is met.
[0100] To meet the classification requirements of coal rock properties, in step five, the classifier structure of the coal rock property recognition network model includes 2 two-dimensional convolutional layers, 1 maximum pooling layer and 2 output classification layers, and the activation function is ReLu6, the optimizer is Adam, the batch-size is 2, and the epochs is 100.
[0101] In the present invention, by building a coal mining machine coal wall cutting test bench, the infrared thermal camera installed on the rocker arm can be used to collect infrared thermal imaging pictures in the coal mining machine cutting area under different coal and rock properties, providing reliable data support for the subsequent acquisition of the coal and rock property recognition network model; using data processing equipment to divide a number of infrared thermal imaging pictures into a small sample training set and a normal number of test sets of the recognition network, it is convenient to use the small sample training set to train the improved WGAN-GP model, and can further use the generated high-quality infrared imaging pictures and small sample training sets to train the recognition network to obtain the coal and rock property recognition network model; using WGAN-GP as the basic generation model, designing a dual discriminator network structure with different discrimination criteria to optimize the model training strategy, can ensure the improvement of model training robustness and generated data diversity. Since the adaptive feature fusion mechanism is constructed to improve the problem of oscillation that is easy to generate in the early stage of model training, the accuracy of the sample generation boundary can be effectively improved by introducing conditional random fields for post-processing operations. Using a small sample training set to train the constructed dual discriminator and adaptive feature fusion WGAN-GP model can generate high-quality infrared thermal imaging images, and then construct an expanded data set; then, using the expanded data set to train the recognition network model can ensure that the coal rock property recognition network model can quickly and accurately output the classification results of different coal rock properties, and thus realize the coal rock property recognition under small samples.
[0102] In summary, based on infrared thermal imaging classification technology, the present invention proposes a small sample coal and rock property identification method based on a dual discriminator and an adaptive feature fusion WGAN-GP model, which can solve the problem of low accuracy in coal and rock property identification caused by insufficient sample collection and uneven distribution due to potential safety hazards in the fully-mechanized mining face and the single cutting mode of the coal mining machine, thereby greatly improving the intelligence level of the coal mining machine, which is of great significance to the realization of "unmanned" or "few-man" mining of the fully-mechanized mining face.
Claims
1. A method for identifying coal and rock properties using infrared thermal imaging of small samples based on a generative adversarial network, characterized in that: The steps include: Step 1: Install an infrared thermal camera on the shearer rocker arm, establish a communication connection between the infrared thermal camera and the data processing equipment, and use the shearer rocker arm, infrared thermal camera and data processing equipment to build a shearer coal wall cutting test bench; Step 2: According to the order of fully mechanized mining work, the coal wall test blocks with different properties are cut in turn, and the infrared thermal camera is used to collect infrared thermal imaging pictures of the cutting area of the coal mining machine under different coal and rock properties at set time intervals, and send them to the data processing equipment; Step 3: The data processing device pre-processes the received infrared thermal imaging images, and then randomly divides the processed infrared thermal imaging images into a small sample training set and a normal number of test sets; Step 4: Build an improved WGAN-GP model based on dual discriminators and adaptive feature fusion, and use a small sample training set to train the improved WGAN-GP model to generate high-quality infrared thermal imaging images; Step 5: Select 2D-CNN as the recognition network model, use the generated high-quality infrared thermal imaging images and a small sample training set to construct an expanded data set, and then train the recognition network model through the expanded data set to obtain a trained coal and rock property recognition network model. Then, use a normal number of test sets to test the trained coal and rock property recognition model. Step 6: During the cutting operation of the coal wall test block, the infrared thermal imaging images collected in real time are input into the trained coal and rock property recognition network model, and the coal and rock property classification results are output to achieve effective recognition of coal and rock properties under small sample conditions.
2. According to the method of claim 1, a small sample infrared thermal imaging coal and rock property recognition method based on generative adversarial network is characterized in that: In step 4, the construction process of the improved WGAN-GP model is as follows: A1: Set up a dual discriminator network; A11: Using WGAN-GP as the basic generative model, a dual discriminator network with different discriminant criteria is set; one discriminator D1 uses real samples as the discrimination criterion, the first input of the discriminator D1 is the infrared thermal imaging picture, and the second input is the generated sample obtained by the generator network G based on random noise z; the other discriminator D2 uses the reconstructed sample extracted by the feature encoder as the discrimination criterion, the first input of the discriminator D2 is the reconstructed sample obtained by the feature encoder based on the infrared thermal imaging picture, and the second input is the generated sample obtained by the generator network G based on random noise z; at the same time, the generator loss function of the WGAN-GP model is obtained according to formula (1), and the discriminator loss function of the WGAN-GP model is obtained according to formula (2); In the formula, x represents the probability distribution P r(x) A real sample of is the result of random interpolation sampling of random noise between real samples and generated samples, and λ×GP is the gradient penalty term; A12: The generator loss function of the dual discriminator network is derived by formula (1), as shown in formula (3); Where ω is the adaptive weight of model training, Its value ranges from 0 to 1; D1(G(z)) is the probability that the discriminator D1 distinguishes between real samples and fake samples, D2(G(z)) is the probability that the discriminator D2 distinguishes between real samples and reconstructed samples, and e is a natural base; A2: Adaptive feature fusion mechanism; A21: Introduce an adaptive feature fusion mechanism. First, use a feature encoder to extract feature information from infrared thermal imaging pictures, and input the feature information into a U-Net reconstruction network. Then, use the U-Net reconstruction network to extract intermediate feature F, and then use the fusion mechanism to adaptively superimpose the random noise z and the intermediate feature F to obtain the output m of the feature fusion mechanism, as shown in formula (4). Then, input the output m of the feature fusion mechanism into the generator network G to obtain a generated sample, and use the generated sample as the second input of discriminator D1 and the second input of discriminator D2; at the same time, use the conditional random field CRF to post-process the output of the U-Net reconstruction network to obtain a reconstructed sample, and then use the reconstructed sample as the first input of discriminator D2; m = Concat(λF(x), z) + z (4); In the formula, z is random noise, F(x) is the intermediate feature of the feature encoder, λ is the feature fusion weight, which gradually decreases as the number of iterations increases, and Concat is the merging operation; A22: Combine formula (3) and formula (4) to obtain the generator loss function after introducing the adaptive feature fusion mechanism, as shown in formula (5); In the formula, m represents the probability distribution P m Fusion samples of A23: Introduce a sample reconstruction loss term on the basis of formula (5) to obtain a new generator loss function, as shown in formula (6); In the formula, |G(m)-x True 2 Represents the reconstruction loss of the sample.
3. According to the method of claim 2, a small sample infrared thermal imaging coal and rock property recognition method based on generative adversarial network is characterized in that: In A21, the process of using the conditional random field CRF to post-process the output of the U-Net reconstruction network is as follows: The conditional random field method is used to post-process the image generated by the U-Net reconstruction network, so that the boundary area between the target and the background is more accurate when generated; assuming C = {c1, c2, ···, c N } represents the N categories generated by the real image I, and the probability undirected graph of the generated results is D = {d1, d2, ···, d N }, and the essence of image generation is based on the real image I. The conditional probability of all pixels in I belonging to which category is P(D=c k |I); According to graph theory, the probability undirected graph corresponding to D satisfies the global Markov property, and the conditional random field P(D|I) is obtained according to formula (7); Where Z(I) is the normalization factor, Among them, ψ1(d j ) is a univariate potential function, which represents the characteristics, structure, position, color, texture and gradient information in the image; ψ2(d j ,d k ) is a binary potential function, which represents the influence of the relationship between pixels on image generation. Among them, each are the features f extracted from pixels j and k, and are determined by the parameter ω g Weighted, vector f j and f k are feature vectors of arbitrary dimensions at pixel j and k respectively; ω g is the weight of the linear combination term; Among them, Q j and Q k are the pixel values of the jth and kth pixels respectively, E j and E k are the positions of the j-th and k-th pixels in the real image I, respectively, and θ, β and δ are all constants.
4. According to the method of claim 3, a small sample infrared thermal imaging coal and rock property recognition method based on generative adversarial network is characterized in that: In step four, the process of using the improved WGAN-GP model to generate high-quality infrared thermal imaging pictures is as follows: B1: Model input and parameter setting; use limited infrared thermal imaging images and random noise z as model input, set the coefficient α of the generator, the coefficient β of the discriminator, and the learning rate η of the generator respectively D , the learning rate η of the discriminator G , the maximum number of iterations of the generator N D , the maximum number of iterations of the discriminator N G ; B2: Parameter initialization; set the number of iterations N = 0 and the intermediate feature F = 0; B3: Adaptive feature fusion; Use the fusion mechanism to adaptively superimpose the random noise z and the intermediate feature F. The specific process is as follows: B31: First, use a feature encoder to extract feature information from a limited number of infrared thermal imaging pictures, and input the feature information into a U-Net reconstruction network. Then, use the U-Net reconstruction network to extract intermediate feature F, and divide the intermediate feature F into {F(x)丨F(1), F(2),…F(n)}; B32: Use formula (4) for feature fusion operation to obtain the output m of the feature fusion mechanism; B33: Update the loss function L of the discriminator using formulas (2) and (6) respectively D And the loss function L of the generator G ; B34: If the iteration termination condition is reached, output the result, otherwise repeat B33; B4: Sample generation; The generator network G uses upsampling to restore the fusion feature to obtain a generated sample, and uses the conditional random field CRF for post-processing operation; B5: Sample discrimination; Output the generated sample to discriminator D1 and discriminator D2 respectively, and use discriminator D1 and discriminator D2 to judge whether it is a real sample or a fake sample; B6: If the model training reaches the maximum number of iterations N D 、N G , output the infrared thermal imaging image generated by the dual discriminator and the adaptive feature fusion WGAN-GP model, otherwise re-execute B3 until the maximum number of model iterations is met.
5. According to the method of claim 4, a small sample infrared thermal imaging coal and rock property recognition method based on generative adversarial network is characterized in that: In step five, the classifier structure of the coal and rock property recognition network model includes 2 two-dimensional convolutional layers, 1 max pooling layer and 2 output classification layers, the activation function is ReLu6, the optimizer is Adam, the batch-size is 2, and the epochs is 100.
Citation Information
Patent Citations
Ship infrared polarization image generation method and system based on multilayer double-discriminator generative adversarial network
CN118230092A
Machine learning based borehole data analysis
WO2024018277A2
