Coal rock cutting state identification method based on multi-sensor data full-channel response

Abstract

The invention relates to a coal rock cutting state recognition method based on multi-sensor data full-channel response, and belongs to the technical field of coal mine intellectualization. Comprising the following steps: acquiring historical original signals acquired by various types of sensors in various coal rock cutting states when the heading machine historically cuts coal rocks, performing adaptive resampling processing to obtain resampling signals of each type of sensor, and continuously processing the resampling signals into multi-channel training sample data; the multi-channel training sample data is segmented into multi-channel training data, a coal rock cutting state recognition network is trained, a trained coal rock cutting state recognition network is obtained, and the coal rock cutting state recognition network is a convolutional neural network into which an ACI module is inserted. By constructing the convolutional neural network into which the ACI module is inserted as the coal rock cutting state recognition network, the accuracy of coal rock cutting state recognition can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent coal mines, and in particular to a method for identifying coal-rock cutting states with full-channel response of multi-sensor data. Background Art

[0002] In the process of intelligent coal mining, the recognition of coal-rock cutting state is becoming increasingly important as a cutting-edge technology for adaptive cutting of roadheaders. By sensing the coal-rock cutting state in real time, the efficiency of tunnel excavation can be improved, the quality of tunnel section forming can be guaranteed, and the wear of the cutting head caused by sudden load changes can be effectively reduced. Especially in high-gas and coal dust environments, this technology can also help avoid sparks generated by high-speed rock cutting, thereby reducing the risk of explosion. However, due to the harsh conditions such as dark light, noise and water mist underground, the current excavation face mainly relies on the experience of on-site personnel to judge the state of coal-rock cutting. This not only poses a serious threat to the life safety of operators, but also hinders the process of unmanned mining. In order to solve these problems and promote the development of this field, researchers have actively carried out a lot of research work.

[0003] Thanks to the rapid development of image technology, researchers have begun to explore the direct use of machine vision to determine the current cutting medium of the tunnel boring machine. Although this method has made some progress in the field of coal and rock cutting state recognition, in actual working environments, image signals are often obstructed and interfered, and it is difficult to ensure real-time performance. In addition, optical sensing equipment is expensive and has high requirements for lighting conditions. Therefore, researchers are more inclined to use time series signals in the cutting process for recognition. A large number of studies have demonstrated the effectiveness of time series signals in coal and rock cutting state recognition, and it is generally believed that more types of sensors can improve the accuracy and robustness of recognition tasks. However, traditional signal processing methods require a lot of prior knowledge in specific fields, and this numerous and complex feature engineering is not ideal. In addition, multi-sensor data fusion methods similar to the above are often accompanied by the disadvantages of data feature loss and high computational complexity, which are not suitable for actual working environments with limited underground computing resources and limited model complexity.

[0004] With the booming development of deep learning, convolutional neural networks (CNNs) have accumulated a large number of successful cases in time series signal classification tasks, thanks to their advantage of being able to automatically learn data features. As we all know, the conventional convolutional layer in a convolutional neural network consists of a set of filters with learnable weights and biases, which respond to input data step by step from low to high layers. However, these filters are essentially a set of neurons with three-dimensional inputs that can only observe local data and are connected to spatially adjacent neurons through shared parameters. Usually, information from different channels is extracted and processed independently. If some channels capture similar features, they are still represented separately. This design ignores the interaction of response information between channels in the same layer, resulting in redundancy in channel information. In multi-sensor mode, the disadvantage of the traditional model is that as the number of layers increases, the features may be overly focused on a certain channel of a certain sensor, resulting in relatively low recognition accuracy. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a method for identifying coal-rock cutting state with full channel response of multi-sensor data. The technical solution of the present invention is as follows:

[0006] A coal-rock cutting state recognition method with full-channel response of multi-sensor data includes:

[0007] S1, obtaining historical original signals collected by various types of sensors under various coal and rock cutting conditions when the roadheader cut coal and rock in history;

[0008] S2, adaptively resampling the historical original signals collected by each type of sensor to obtain the resampled signals of each type of sensor;

[0009] S3, processing the resampled signals of each type of sensor into multi-channel training sample data;

[0010] S4, performing frame windowing on the multi-channel training sample data through a sliding window of a preset size, so as to divide the multi-channel training sample data into multi-channel training data of a specified length, and marking the coal-rock cutting state of each multi-channel training data, so as to obtain a training data set;

[0011] S5, training the coal-rock cutting state recognition network through the training data set, and obtaining a trained coal-rock cutting state recognition network when the coal-rock cutting state recognition network reaches a preset convergence condition, wherein the coal-rock cutting state recognition network is a convolutional neural network with an ACI module inserted therein, and the ACI module is inserted behind at least one convolutional layer of the convolutional neural network;

[0012] S6, obtain the current original signals collected by various types of sensors when the roadheader is currently cutting coal and rock, and process the current original signals collected by each type of sensor into multi-channel target data, input the multi-channel target data into the trained coal and rock cutting state recognition network, and output the current coal and rock cutting state of the roadheader after analysis by the ACI module and other component structures in the trained coal and rock cutting state recognition network.

[0013] Optionally, when S2 is implemented, it includes:

[0014] S21, determining the number of channels, the number of original sampling points, and the original sampling rate of the historical original signals collected by each type of sensor;

[0015] S22, selecting the minimum original sampling rate as the target sampling rate;

[0016] S23, calculating the number of samples of the resampled historical original signal of each type of sensor according to the ratio of the target sampling rate to the original sampling rate of each type of sensor and the number of original sampling points;

[0017] S24, performing low-pass filtering on the historical original signals collected by each type of sensor to obtain filtered signals of each type of sensor;

[0018] S25, interpolating and decimating the filtered signal of each type of sensor according to the number of samples of the resampled signal of each type of sensor and its original sampling rate, to obtain a resampled signal of each type of sensor at a target sampling rate.

[0019] Optionally, when S3 is implemented, it includes:

[0020] S31, finding the minimum length of the resampled signals of various types of sensors;

[0021] S32, aligning the resampled signals of various types of sensors according to the minimum length of the resampled signals and the signal timestamp, to obtain a new sensor signal of each type of sensor;

[0022] S34, splicing new sensor signals of each type of sensor according to the number of channels of each type of sensor to obtain multi-channel training sample data.

[0023] Optionally, the various types of sensors include acceleration sensors, current sensors and audio sensors, and the historical original signals include historical original acceleration signals, historical original current signals and historical original audio signals, the number of channels of the historical original acceleration signals and the historical original current signals are both three, and the number of channels of the historical original audio signals is one.

[0024] Optionally, when S3 is specifically implemented, it further includes: S33, expanding the new sensing signal of the audio sensor with one channel into an audio target sensing signal with three channels, specifically including:

[0025] S331, analyzing a new sensing signal of the audio sensor by fast Fourier transform to obtain a frequency spectrum of the new sensing signal of the audio sensor;

[0026] S332, calculating the energy of the new sensor signal of the audio sensor and its energy distribution in three frequency bands according to the amplitudes of the new sensor signal of the audio sensor in different frequency bands, wherein the three frequency bands include a low frequency, a medium frequency, and a high frequency;

[0027] S333, calculating the probability distribution of the new sensor signal of the audio sensor in the three frequency bands according to the energy of the new sensor signal of the audio sensor and the energy distribution in the three frequency bands;

[0028] S334, calculating the entropy of the new sensing signal of the audio sensor according to its probability distribution in three frequency bands;

[0029] S335, calculating weights of the three frequency bands according to the inverse of entropy and the probability distribution of the new sensing signal of the audio sensor in the three frequency bands;

[0030] S336, determining the extended spectrum of the new sensing signal of the audio sensor in the three frequency bands according to the spectrum of the new sensing signal of the audio sensor and the weights of the new sensing signal in the three frequency bands;

[0031] S337, performing inverse Fourier transform on the extended spectrum of the new sensor signal of the audio sensor in three frequency bands to obtain an audio target sensor signal with three channels, and using the audio target sensor signal with three channels as a new sensor signal when the audio sensor performs signal splicing.

[0032] Optionally, in the specific implementation of S4, the multi-channel training sample data is framed and windowed by specifying a sliding window with a length of 128 points, and the moving step length of the sliding window is 50% of the sliding window length.

[0033] Alternatively, assume that the convolutional neural network contains L convolutional layers, and each convolutional layer contains n l neurons, represents the characteristic response of the lth convolutional layer, the ACI module is inserted behind the lth convolutional layer, the ACI module includes an encoder, an information interaction layer and a decoder, and the specific analysis process of the ACI module in the trained coal-rock cutting state recognition network includes:

[0034] S61, the encoder extracts the feature responses of all channels of the l-th convolutional layer as the input feature map, and flattens the input feature map into a one-dimensional form to connect the feature responses of all channels into a long vector;

[0035] S62, the long vector is processed by two fully connected layers in sequence to obtain the channel feature response of the encoder It is expressed as:

[0036]

[0037] in, and Represent the linear functions in the two fully connected layers, σ represents the ReLU activation function, represents the characteristic response of the data after passing through the i-th neuron of the l-th convolutional layer, Represents the first fully connected layer pair The channel characteristic response, Represents the second fully connected layer pair The channel characteristic response of is the channel characteristic response of the encoder;

[0038] S63, the information interaction layer converts the encoder's channel feature response As nodes form an undirected graph, define the node set of the undirected graph with Indicates that the edge strength s between each node is calculated ij :

[0039]

[0040] in, represents the kth element of the long vector, i and j represent any two different neurons in the lth convolutional layer, H and W represent the height and width of the input features of the information interaction layer, respectively. Represents the expected value of the information interaction layer's response to the encoder's channel features;

[0041] S64, the edge strength s between each node ij Perform normalization processing to obtain the normalized processing result a ij , and according to the normalization result a ij Calculate the output node set of the information interaction layer The formula is:

[0042]

[0043] S65, the decoder will output the node set Decoded into a shape that matches the encoder’s input feature map.

[0044] Optionally, the coal-rock cutting state recognition network reaching a preset convergence condition in S5 means that the loss value of the target loss function of the coal-rock cutting state recognition network reaches a set value;

[0045] The target loss function of the coal-rock cutting state recognition network is expressed as: α and β represent and L(y)’s respective set weights;

[0046]

[0047] Among them, i and j represent any two different neurons in the lth convolutional layer, m and n represent the horizontal and vertical coordinates of the neuron channel response matrix, and y i and They represent the output value and expected value of the coal-rock cutting state recognition network respectively, and N represents the number of iterations.

[0048] Optionally, during specific implementation, S23 is implemented by the following formula:

[0049]

[0050] Among them, N i represents the number of original sampling points of the i-th type of sensor, targetSR represents the target sampling rate, sr i represents the original sampling rate of the i-th type of sensor, N i-target Represents the number of samples of the resampled signal of the i-th type of sensor.

[0051] All the above optional technical solutions can be combined arbitrarily, and the present invention does not provide detailed descriptions of the structures after the combinations.

[0052] By means of the above scheme, the beneficial effects of the present invention are as follows:

[0053] By constructing a convolutional neural network with an ACI module inserted as the coal-rock cutting state recognition network, the ACI module can ensure the interaction of information between different channels, and then perceive the correlation and context information between channels, thereby improving the expression ability and performance of the coal-rock cutting state recognition network, and then improving the accuracy of coal-rock cutting state recognition.

[0054] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1It is a flow chart of a method for identifying coal-rock cutting status with full-channel response of multi-sensor data provided by an embodiment of the present invention.

[0056] Figure 2 This is a visualization diagram of the ACI module in an embodiment of the present invention for changing the field of view of the convolutional neural network architecture.

[0057] Figure 3 Schematic diagram of the computational accuracy when the ACI module is added after different convolutional layers of the basic convolutional neural network architecture. DETAILED DESCRIPTION

[0058] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0059] The coal-rock cutting state identification method with full-channel response of multi-sensor data provided in the embodiment of the present invention can be implemented by any electronic device with computing function, such as a PC, a mobile terminal or a server. Figure 1 As shown, the coal-rock cutting state identification method based on full-channel response of multi-sensor data provided by the embodiment of the present invention includes the following steps S1 to S6:

[0060] S1, obtaining historical original signals collected by various types of sensors under various coal and rock cutting conditions when the roadheader cut coal and rock in the past.

[0061] The embodiment of the present invention does not specifically limit the type of sensor. During specific implementation, several sensors that are more relevant to the coal and rock cutting state can be selected according to the type of sensor installed on the tunnel boring machine when it cuts the coal and rock. For example, various types of sensors may include acceleration sensors, current sensors, and audio sensors. Among them, the acceleration sensor is installed on the cutting arm of the tunnel boring machine, the current sensor is connected to the motor of the tunnel boring machine, and the audio sensor is placed on the tunnel boring machine or connected to the tunnel boring machine and is used to collect audio data when the tunnel boring machine cuts. On this basis, the historical original signal includes the historical original acceleration signal, the historical original current signal, and the historical original audio signal. For the convenience of subsequent description, the embodiment of the present invention uses various types of sensors including acceleration sensors, current sensors, and audio sensors to illustrate the method provided by the embodiment of the present invention.

[0062] Specifically, various types of sensors are electrically connected to the electronic device. After collecting their corresponding historical original signals, the various types of sensors send them to the electronic device. The electronic device obtains the historical original signals collected by the various types of sensors by receiving the data sent by the various types of sensors.

[0063] In addition, the coal-rock cutting state includes whole coal, whole rock or coal-rock segmentation cross section.

[0064] S2, adaptively resample the historical original signals collected by each type of sensor to obtain the resampled signals of each type of sensor.

[0065] In a specific embodiment, the S2 may include the following steps S21 to S25 when implemented:

[0066] S21, determining the number of channels, the number of original sampling points, and the original sampling rate of the historical original signals collected by each type of sensor.

[0067] Specifically, the historical raw acceleration signal can be defined as N 1 Indicates the number of original sampling points. Each original sampling point has three channels of data, representing the data of the three coordinate axes at this moment. The original sampling rate is sr 1 ; The historical original current signal is defined as N 2 Indicates the number of original sampling points. Each original sampling point has three channels of data, which represent the data of the motor in three-phase AC at this moment. The original sampling rate is sr 2 ; The historical original audio signal is defined as N 3 Indicates the number of original sampling points, 1 indicates single-channel data, and the original sampling rate is sr 3 .

[0068] S22, selecting the minimum original sampling rate as the target sampling rate.

[0069] The step S22 is expressed by the formula: targetSR = min(sr 1 ,sr 2 ,sr 3 ), targetSR represents the target sampling rate.

[0070] S23, calculating the number of samples of the resampled historical original signal of each type of sensor according to the ratio of the target sampling rate to the original sampling rate of each type of sensor and the number of original sampling points.

[0071] Specifically, step S23 can be implemented by the following formula:

[0072]

[0073] Among them, N i represents the number of original sampling points of the i-th type of sensor, targetSR represents the target sampling rate, sr i represents the original sampling rate of the i-th type of sensor, N i-targetRepresents the number of samples of the resampled historical original signal of the i-th type of sensor.

[0074] S24, performing low-pass filtering on the historical original signals collected by each type of sensor to obtain filtered signals of each type of sensor.

[0075] Specifically, step S24 can be implemented by the following formula:

[0076] y(n)=s(n)×h(n);

[0077]

[0078] Among them, s(n) represents the historical original signal collected by any sensor; h(n) is the impulse response of the low-pass filter, which can be expressed as a sinc function; y(n) represents the signal after filtering any historical original signal; f c represents the low-pass cutoff frequency. In the embodiment of the present invention, its value is set to targetSR / 2. w(n) represents the window function, which is used to truncate the signal. In the embodiment of the present invention, a rectangular window function is used. The value of M is N 1 、N 2 or N 3 .

[0079] By low-pass filtering, aliasing can be prevented during the resampling process.

[0080] S25, interpolating and decimating the filtered signal of each type of sensor according to the number of samples of the resampled signal of each type of sensor and its original sampling rate, to obtain a resampled signal of each type of sensor at a target sampling rate.

[0081] Specifically, step S25 can be implemented by the following formula:

[0082]

[0083] Where m = 0, 1, 2, ..., N target -1, z(n) represents the resampled signal of any type of sensor at the target sampling rate. The three resampled signals are represented by z 1 、z 2 and z 3 .

[0084] S3, processes the resampled signals of each type of sensor into multi-channel training sample data.

[0085] In a specific embodiment, the S3 may include the following steps S31 to S34 during implementation:

[0086] S31, find the minimum length of the resampled signal of various types of sensors.

[0087] Specifically, when the step S31 is implemented, it can be expressed by the following formula:

[0088] N min =min{length(z 1 ),length(z 2 ),length(z 3 )};

[0089] Among them, length(z i ) represents the length of the resampled signal of the i-th type sensor, N min Indicates the minimum length of the resampled signal of various types of sensors.

[0090] S32, aligning the resampled signals of various types of sensors according to the minimum length of the resampled signals and the signal timestamp, to obtain a new sensing signal of each type of sensor.

[0091] Specifically, the new sensing signals of the three types of sensors It can be expressed as:

[0092]

[0093] Furthermore, in order to enrich the features of the new sensor signal of the audio sensor, the method provided in the embodiment of the present invention may further include: S33, expanding the new sensor signal of the audio sensor with one channel to an audio target sensor signal with three channels.

[0094] When the step S33 is implemented specifically, it may include the following steps S331 to S337:

[0095] S331, analyzing the new sensing signal of the audio sensor by fast Fourier transform to obtain the frequency spectrum of the new sensing signal of the audio sensor.

[0096] Specifically, the new sensor signal of the audio sensor Perform spectrum analysis and use fast Fourier transform (FFT) to obtain the performance of the new sensor signal of the audio sensor in the frequency domain. Assume that the spectrum of the new sensor signal of the audio sensor is Z 3 (f), f represents the frequency component, Z 3 (f) represents the amplitude of the new sensing signal of the audio sensor at each frequency component. This step is expressed by the formula:

[0097] S332, calculating the energy of the new sensor signal of the audio sensor and its energy distribution in three frequency bands according to the amplitude of the new sensor signal of the audio sensor in different frequency bands, wherein the three frequency bands include low frequency, middle frequency and high frequency.

[0098] Specifically, when calculating the energy E(f) of the new sensing signal of the audio sensor, step S332 can be implemented by the following formula:

[0099] E(f)=|Z 3 (f)| 2 .

[0100] Furthermore, when calculating the energy distribution of the new sensing signal of the audio sensor in the three frequency bands, it can be achieved by the following formula:

[0101] E(f i )=|Z 3 (f i )| 2 ;

[0102] Among them, E(f i ) represents the energy distribution of the new sensor signal of the audio sensor in frequency band i, Z 3 (f i ) shows the amplitude of the new sensing signal of the audio sensor in frequency band i.

[0103] By analyzing the energy distribution of components in different frequency bands of the new sensing signal of the audio sensor, the main frequency range in the new sensing signal of the audio sensor and its characteristics in different frequency bands can be identified.

[0104] S333: Calculate the probability distribution of the new sensing signal of the audio sensor in the three frequency bands according to the energy of the new sensing signal of the audio sensor and the energy distribution in the three frequency bands.

[0105] The probability distribution is formed by normalizing the energy distribution. Specifically, step S333 can be implemented by the following formula:

[0106]

[0107] Among them, P(f i ) represents the probability distribution of the new sensing signal of the audio sensor in frequency band i.

[0108] S334, calculating the entropy H of the new sensing signal of the audio sensor according to its probability distribution in the three frequency bands. Specifically, step S334 can be implemented by the following formula:

[0109] H=-∑P(f i )log(P(f i )).

[0110] S335 , calculating weights of the three frequency bands according to the inverse of the entropy and the probability distribution of the new sensing signal of the audio sensor in the three frequency bands.

[0111] Specifically, step S335 can be implemented by the following formula:

[0112]

[0113] Among them, w(f i ) represents the weight of the new sensor signal of the audio sensor in frequency band i.

[0114] S336: Determine the extended spectrum of the new sensing signal of the audio sensor in the three frequency bands according to the spectrum of the new sensing signal of the audio sensor and the weights of the three frequency bands.

[0115] Specifically, step S336 can be implemented by the following formula:

[0116] S A (f) = w A (f)·Z 3 (f)

[0117] S B (f) = w B (f)·Z 3 (f)

[0118] S C (f) = w C (f)·Z 3 (f);

[0119] Among them, S A (f) S B (f) and S C (f) represents the extended spectrum of the new sensing signal of the audio sensor in the low frequency band, medium frequency band and high frequency band, respectively. A (f) w B (f) and w C (f) represents the weights of the new sensing signal of the audio sensor in the low frequency band, medium frequency band and high frequency band respectively.

[0120] By calculating the energy distribution, entropy and normalization of each frequency band, we can assign reasonable weights to different frequency bands. This method not only takes into account the energy distribution of the signal, but also reflects the importance of frequency information, thus improving the effect of signal processing.

[0121] S337, performing inverse Fourier transform on the extended spectrum of the new sensor signal of the audio sensor in three frequency bands to obtain an audio target sensor signal with three channels, and using the audio target sensor signal with three channels as a new sensor signal when the audio sensor performs signal splicing.

[0122] Specifically, step S337 can be implemented by the following formula:

[0123] z 3 (A) (n) = IFFT(S A (f));

[0124] z 3 (B) (n) = IFFT(S B (f));

[0125] z 3 (C) (n) = IFFT(S C (f));

[0126] z 3 new =Concat(z 3 (A) (n),z 3 (B) (n),z 3 (C) (n));

[0127] Among them, z 3 (A) (n), z 3 (B) (n) and z 3 (C) (n) represents the inverse Fourier transform results of the extended spectrum of the new sensing signal of the audio sensor in three frequency bands, z 3 new represents the audio target sensor signal, and Concat represents the concatenation operation.

[0128] Specifically, during splicing, the inverse Fourier transform results of the spread spectrum of the three frequency bands can be directly combined together according to the frequency.

[0129] S34, splicing new sensor signals of each type of sensor according to the number of channels of each type of sensor to obtain multi-channel training sample data.

[0130] Specifically, step S34 can be expressed as the following formula:

[0131]

[0132] Wherein, dataS represents multi-channel training sample data. The multi-channel training sample data here is nine-channel training sample data.

[0133] Specifically, when splicing, the new sensor signal of the acceleration sensor, the new sensor signal of the current sensor, and the audio target sensor signal are arranged in sequence into a 9-row N min A matrix of columns is sufficient.

[0134] S4, frame and window the multi-channel training sample data through a sliding window of a preset size to divide the multi-channel training sample data into multi-channel training data of a specified length, and mark the coal-rock cutting state of each multi-channel training data to obtain a training data set.

[0135] The number of the specified length can be set as needed. For example, the embodiment of the present invention sets the specified length to include 128 points. That is to say, in the specific implementation of S4, the multi-channel training sample data is framed and windowed by a sliding window with a length of 128 points, and the moving step length of the sliding window is 50% (64 points) of the sliding window length. By setting the moving step length of the sliding window to 50% of the sliding window length, the local features of the multi-channel training sample data can be better captured.

[0136] Specifically, when marking the coal-rock cutting status of each multi-channel training data, it can be achieved with the help of labeling software or through manual labeling.

[0137] S5, training the coal-rock cutting state recognition network through the training data set, and when the coal-rock cutting state recognition network reaches a preset convergence condition, obtaining a trained coal-rock cutting state recognition network, wherein the coal-rock cutting state recognition network is a convolutional neural network with an ACI module inserted therein, and the ACI module is inserted behind at least one convolutional layer of the convolutional neural network.

[0138] In a specific embodiment, the S5 coal-rock cutting state identification network reaches a preset convergence condition means that the loss value of the target loss function of the coal-rock cutting state identification network reaches a set value.

[0139] Specifically, the target loss function of the coal-rock cutting state recognition network is expressed as: α and β represent and L(y)’s respective set weights;

[0140]

[0141] Among them, i and j represent any two different neurons in the lth convolutional layer, m and n represent the horizontal and vertical coordinates of the neuron channel response matrix, and y i and They represent the output value and expected value of the coal-rock cutting state recognition network, respectively, and N represents the number of iterations; and Represents characteristic response The mean and variance of . Represents the correlation between channel response graphs, which can be used to know the relationship between the features expressed by different channels in a specific layer. The correlation between features can help understand whether the information encoded by different channels in a specific layer of the coal-rock cutting state recognition network is overlapping or diverse. If the correlation is high, that is, the correlation value is large, it means that there is a large overlap or redundancy in the feature expressions between different channels, which means that the features learned by the network at this layer may lack diversity and have a high degree of information repetition. On the contrary, if the correlation is low, that is, the correlation value is small, it means that the feature expressions between different channels are more diverse, the features learned by the network at this layer are richer and more diverse, and the information repetition is low, which is exactly what is needed for training the coal-rock cutting state recognition network in the embodiment of the present invention.

[0142] The embodiment of the present invention does not specifically limit the number of ACI modules inserted in the convolutional neural network. During specific implementation, the number of ACI modules to be inserted can be determined based on the number of convolutional layers required to be included in the convolutional neural network and the recognition requirements.

[0143] S6, obtain the current original signals collected by various types of sensors when the roadheader is currently cutting coal and rock, and process the current original signals collected by each type of sensor into multi-channel target data, input the multi-channel target data into the trained coal and rock cutting state recognition network, and output the current coal and rock cutting state of the roadheader after analysis by the ACI module and other component structures in the trained coal and rock cutting state recognition network.

[0144] It should be noted that the method of processing the current original signal collected by each type of sensor into multi-channel target data is the same as the method of processing the historical original signal collected by each type of sensor into multi-channel training data as described in S2 to S4 in the above embodiments. For details, please refer to the contents in the above embodiments, which will not be repeated here.

[0145] In a specific embodiment, it is assumed that the convolutional neural network includes L convolutional layers, and each convolutional layer includes n l neurons, represents the characteristic response of the l-th convolutional layer, the ACI module is inserted after the l-th convolutional layer, the ACI module includes an encoder, an information interaction layer and a decoder, and the specific analysis process of the ACI module in the trained coal-rock cutting state recognition network includes the following steps S61 to S65:

[0146] S61, the encoder extracts the feature responses of all channels of the lth convolutional layer as the input feature map, and flattens the input feature map into a one-dimensional form to connect the feature responses of all channels into a long vector.

[0147] In this way, the channel information of the feature responses of all channels of the l-th convolutional layer can be maintained and the spatial information can be removed.

[0148] S62, the long vector is processed by two fully connected layers in sequence to obtain the channel feature response of the encoder It is expressed as:

[0149]

[0150] in, and Represent the linear functions in the two fully connected layers respectively; σ represents the ReLU activation function, which enables the network to learn nonlinear relationships and better fit complex signal distributions; represents the characteristic response of the data after passing through the i-th neuron of the l-th convolutional layer, Represents the first fully connected layer pair The channel characteristic response, Represents the second fully connected layer pair The channel characteristic response of is the channel characteristic response of the encoder.

[0151] Optionally, the first fully connected layer has a linear function The bottleneck layer is also included, which is used to compress the dimension of the input feature by γ times while ensuring the validity of the feature, where γ>1.

[0152] S63, the information interaction layer converts the encoder's channel feature response As nodes form an undirected graph, define the node set of the undirected graph with Indicates that the edge strength s between each node is calculated ij :

[0153]

[0154] in, represents the kth element of the long vector, i and j represent any two different neurons in the lth convolutional layer, H and W represent the height and width of the input features of the information interaction layer, respectively. Represents the expected value of the information interaction layer's response to the encoder's channel features.

[0155] Edge strength represents the strength of connection or relationship between channels, which defines the degree of information propagation in an undirected graph. When calculating edge strength, the embodiment of the present invention can increase the robustness of information interaction by seeking expectations for the encoder according to the output shape. At the same time, the embodiment of the present invention defines edge strength by calculating the negative square distance, and uses it as an evaluation indicator of the degree of channel interaction, so that channels that capture similar features will have more interactions, thereby making the features more responsive. In the subsequent recognition process, graph nodes with close distances will be given greater weights, thereby improving the recognition effect.

[0156] S64, the edge strength s between each node ij Perform normalization processing to obtain the normalized processing result a ij , and according to the normalization result a ij Calculate the output node set of the information interaction layer The formula is:

[0157]

[0158] In summary, the processing of the information interaction layer can ensure that the signals collected by various types of sensors make the greatest contribution to the recognition of cutting states. The ACI module transmits information between all channels and updates the feature responses in a timely manner to ensure that the weights between channels are shared and exchanged.

[0159] S65, the decoder will output the node set Decoded into a shape that matches the encoder’s input feature map.

[0160] At this point, the decoder obtains the channel information containing weights.

[0161] In summary, the processing of the ACI module can be expressed as follows:

[0162]

[0163] Among them, the characteristic response of all filters is given by f l i The function performs cumulative calculations, and only the response of the i-th channel is updated each time during the calculation.

[0164] So far, The features are passed to the deeper convolutional layer of the coal-rock cutting state recognition network for conventional convolution operations. The features after the encoder, information interaction layer and decoder operations contain more advanced semantic information, while making the channel capture richer information.

[0165] In summary, when processing data, the ACI module first maps the input features to a long vector through the encoder's multi-layer perceptron and fully connected layer to achieve comprehensive perception and compression of signal features. Then, the information interaction layer uses GNN (graph neural network) to ensure information interaction between different channels, which is used to perceive the correlation and context information between channels, thereby improving the expression ability and performance of the coal-rock cutting state recognition network. Finally, the decoder corrects the weights according to the shape of the encoder's original input. In general, compared with ordinary convolutional layers, the ACI module is closer to the way the human brain processes information, allowing information to be widely communicated and shared in the network.

[0166] It should be noted that the ACI module described in the embodiment of the present invention is plug-and-play and can be used in almost all convolutional neural network architectures. The embodiment of the present invention specifically describes the three most common convolutional neural network architectures, which are CNN3 (3-layer CNN), ResNet and DenseNet. Specifically, CNN3 contains 3 convolutional layers, ResNet contains 6 convolutional layers, and DenseNet contains three dense blocks, each of which contains a convolutional layer, and the dense blocks are connected by transition layers. To verify its effect, the embodiment of the present invention experimented with three convolutional neural network architectures with the ACI module added. The experiment used the Adam optimizer, the learning rate was set to 0.0005, the batch size was 128, and CrossEntropyLoss was selected as the loss function. The detailed model architecture and the properties of each layer are shown in Table 1, where C(Ls) represents the number of feature maps of each convolutional layer.

[0167]

[0168]

[0169] Figure 2 This is a visualization of the effect of adding the ACI module to CNN3 on changing the convolutional neural network architecture. Figure 2 It can be seen that the use of ACI modules activates more channels ( Figure 2 Data from different channels are displayed in different colors), which enables multi-sensor data to make a more comprehensive joint decision-making and recognition result, reducing channel redundancy.

[0170] In addition, the ACI module can be added after any convolutional layer of the commonly used convolutional neural network architecture. Experiments have shown that adding the ACI module after the deeper convolutional layer of the network is more effective, because deeper convolutional layers can extract more advanced semantic features, and channel interactions can obtain richer feature responses. Moreover, experiments have shown that there is no need to add ACI modules after each convolutional layer without limit, which is conducive to lightweight networks and speeding up data calculations. Figure 3 As shown in Figure 1, it is a schematic diagram of the computational accuracy when the ACI module is added after different convolutional layers of the basic convolutional neural network architecture (ResNet). Figure 3 It can be seen that adding the ACI module to the convolutional neural network can significantly improve the accuracy of the coal-rock cutting state recognition network. Figure 3 While showing the experimental results, the response speed of the coal-rock cutting state recognition network was statistically analyzed, and the statistical results are shown in Table 2.

[0171] Table 2

[0172]

[0173] It can be seen from Table 2 that after adding the ACI module after the convolutional neural network, the response speed of the coal-rock cutting state recognition network is not greatly affected compared with the case where the ACI module is not added.

[0174] It should be noted that the specific process of analyzing other component structures in the trained coal-rock cutting state recognition network is the same as the specific process of processing data by the convolutional neural network without the ACI module inserted, and the embodiment of the present invention will not elaborate on this.

[0175] In summary, the method provided by the embodiment of the present invention improves the recognition accuracy of the coal and rock cutting state recognition network while hardly increasing the computational burden, while ensuring a rapid response, and has proved its applicability and robustness in practical applications through experiments. The method provided by the embodiment of the present invention has broad application prospects.

[0176] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the technical principles of the present invention, and these improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A coal-rock cutting state recognition method with full channel response of multi-sensor data, characterized in that: include: S1, obtaining historical original signals collected by various types of sensors under various coal and rock cutting conditions when the roadheader cut coal and rock in history; S2, adaptively resampling the historical original signals collected by each type of sensor to obtain the resampled signals of each type of sensor; S3, processing the resampled signals of each type of sensor into multi-channel training sample data; S4, performing frame windowing on the multi-channel training sample data through a sliding window of a preset size, so as to divide the multi-channel training sample data into multi-channel training data of a specified length, and marking the coal-rock cutting state of each multi-channel training data, so as to obtain a training data set; S5, training the coal-rock cutting state recognition network through the training data set, and obtaining a trained coal-rock cutting state recognition network when the coal-rock cutting state recognition network reaches a preset convergence condition, wherein the coal-rock cutting state recognition network is a convolutional neural network with an ACI module inserted therein, and the ACI module is inserted behind at least one convolutional layer of the convolutional neural network; S6, obtain the current original signals collected by various types of sensors when the roadheader is currently cutting coal and rock, and process the current original signals collected by each type of sensor into multi-channel target data, input the multi-channel target data into the trained coal and rock cutting state recognition network, and output the current coal and rock cutting state of the roadheader after analysis by the ACI module and other component structures in the trained coal and rock cutting state recognition network.

2. The coal-rock cutting state identification method based on multi-sensor data full-channel response according to claim 1 is characterized in that: When S2 is specifically implemented, it includes: S21, determining the number of channels, the number of original sampling points, and the original sampling rate of the historical original signals collected by each type of sensor; S22, selecting the minimum original sampling rate as the target sampling rate; S23, calculating the number of samples of the resampled historical original signal of each type of sensor according to the ratio of the target sampling rate to the original sampling rate of each type of sensor and the number of original sampling points; S24, performing low-pass filtering on the historical original signals collected by each type of sensor to obtain filtered signals of each type of sensor; S25, interpolating and decimating the filtered signal of each type of sensor according to the number of samples of the resampled signal of each type of sensor and its original sampling rate, to obtain a resampled signal of each type of sensor at a target sampling rate.

3. The coal-rock cutting state identification method based on full-channel response of multi-sensor data according to claim 1 or 2, characterized in that: When S3 is specifically implemented, it includes: S31, finding the minimum length of the resampled signals of various types of sensors; S32, aligning the resampled signals of various types of sensors according to the minimum length of the resampled signals and the signal timestamp, to obtain a new sensor signal of each type of sensor; S34, splicing new sensor signals of each type of sensor according to the number of channels of each type of sensor to obtain multi-channel training sample data.

4. The coal-rock cutting state identification method based on multi-sensor data full-channel response according to claim 3 is characterized in that: Various types of sensors include acceleration sensors, current sensors and audio sensors. The historical original signals include historical original acceleration signals, historical original current signals and historical original audio signals. The number of channels of the historical original acceleration signals and the historical original current signals is three, and the number of channels of the historical original audio signals is one.

5. The coal-rock cutting state identification method based on full-channel response of multi-sensor data according to claim 4 is characterized in that: When S3 is specifically implemented, it further includes: S33, expanding the new sensing signal of the audio sensor with one channel into an audio target sensing signal with three channels, specifically including: S331, analyzing a new sensing signal of the audio sensor by fast Fourier transform to obtain a frequency spectrum of the new sensing signal of the audio sensor; S332, calculating the energy of the new sensor signal of the audio sensor and its energy distribution in three frequency bands according to the amplitudes of the new sensor signal of the audio sensor in different frequency bands, wherein the three frequency bands include a low frequency, a medium frequency, and a high frequency; S333, calculating the probability distribution of the new sensor signal of the audio sensor in the three frequency bands according to the energy of the new sensor signal of the audio sensor and the energy distribution in the three frequency bands; S334, calculating the entropy of the new sensing signal of the audio sensor according to its probability distribution in three frequency bands; S335, calculating weights of the three frequency bands according to the inverse of entropy and the probability distribution of the new sensing signal of the audio sensor in the three frequency bands; S336, determining the extended spectrum of the new sensing signal of the audio sensor in the three frequency bands according to the spectrum of the new sensing signal of the audio sensor and the weights of the new sensing signal in the three frequency bands; S337, performing inverse Fourier transform on the extended spectrum of the new sensor signal of the audio sensor in three frequency bands to obtain an audio target sensor signal with three channels, and using the audio target sensor signal with three channels as a new sensor signal when the audio sensor performs signal splicing.

6. The coal-rock cutting state identification method based on multi-sensor data full-channel response according to claim 1 is characterized in that: In the specific implementation of S4, the multi-channel training sample data is framed and windowed by specifying a sliding window with a length of 128 points, and the moving step length of the sliding window is 50% of the sliding window length.

7. The coal-rock cutting state identification method based on multi-sensor data full-channel response according to claim 1 is characterized in that: Assume that the convolutional neural network contains L convolutional layers, and each convolutional layer contains n l neurons, represents the characteristic response of the lth convolutional layer, the ACI module is inserted behind the lth convolutional layer, the ACI module includes an encoder, an information interaction layer and a decoder, and the specific analysis process of the ACI module in the trained coal-rock cutting state recognition network includes: S61, the encoder extracts the feature responses of all channels of the l-th convolutional layer as the input feature map, and flattens the input feature map into a one-dimensional form to connect the feature responses of all channels into a long vector; S62, the long vector is processed by two fully connected layers in sequence to obtain the channel feature response of the encoder It is expressed as: in, and Respectively represent the linear functions in the two fully connected layers, σ represents the ReLU activation function, represents the characteristic response of the data after passing through the i-th neuron of the l-th convolutional layer, Represents the first fully connected layer pair The channel characteristic response, Represents the second fully connected layer pair The channel characteristic response of is the channel characteristic response of the encoder; S63, the information interaction layer converts the encoder's channel feature response As nodes form an undirected graph, define the node set of the undirected graph with Indicates that the edge strength s between each node is calculated ij : in, represents the kth element of the long vector, i and j represent any two different neurons in the lth convolutional layer, H and W represent the height and width of the input features of the information interaction layer, respectively. Represents the expected value of the information interaction layer's response to the encoder's channel features; S64, the edge strength s between each node ij Perform normalization processing to obtain the normalized processing result a ij , and according to the normalization result a ij Calculate the output node set of the information interaction layer The formula is: S65, the decoder will output the node set Decoded into a shape that matches the encoder’s input feature map.

8. The coal-rock cutting state identification method based on full-channel response of multi-sensor data according to claim 7 is characterized in that: The coal-rock cutting state recognition network reaches a preset convergence condition in S5, which means that the loss value of the target loss function of the coal-rock cutting state recognition network reaches a set value; The target loss function of the coal-rock cutting state recognition network is expressed as: α and β represent and L(y)’s respective set weights; Among them, i and j represent any two different neurons in the lth convolutional layer, m and n represent the horizontal and vertical coordinates of the neuron channel response matrix, and y i and They represent the output value and expected value of the coal-rock cutting state recognition network respectively, and N represents the number of iterations.

9. The coal-rock cutting state identification method based on multi-sensor data full-channel response according to claim 3 is characterized in that: In the specific implementation, the S23 is realized by the following formula: Among them, N i represents the number of original sampling points of the i-th type of sensor, targetSR represents the target sampling rate, sr i represents the original sampling rate of the i-th type of sensor, N i-target Represents the number of samples of the resampled signal of the i-th type of sensor.

Images