Coal rock identification method and device thereof
By combining signals acquired from sound and vibration sensors and using the Transformer model for cross-modal feature fusion, the limitations and lag issues of existing coal and rock identification technologies are resolved, achieving higher-precision coal and rock category identification.
Patent Information
- Application Number
- CN202211334641.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-10-28
- Publication Date
- 2025-11-11
- Estimated Expiration
- 2042-10-28
AI Technical Summary
Existing coal and rock identification methods rely on a single monitoring means, which has limitations and lags, making it difficult to accurately identify the types of coal and rock cut by coal mining machines.
By combining sound and vibration sensors to acquire audio and vibration signals from the coal mining machine, and performing time synchronization processing, a Transformer-based acoustic-vibration fusion classification model is used to perform cross-modal feature fusion to determine the predicted category of coal and rock.
It improves the accuracy of coal and rock identification, avoids the limitations and lag of a single monitoring signal, and enhances the prediction accuracy of the coal and rock categories cut by the coal mining machine.
Smart Images
Figure CN115910103B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of coal mining technology, and in particular to a method and apparatus for coal and rock identification. Background Technology
[0002] Coal and rock identification refers to the automatic identification of coal or rock objects using a single method. The operating audio of coal mining equipment is closely related to its mechanical structure and working condition, making it a crucial indicator for analyzing equipment operation. Many related coal and rock identification methods, such as radar detection, stress-cutting tooth method, and infrared detection, require the installation of various sensors on equipment like coal mining machine drums and robotic arms to directly acquire operating information. Furthermore, these methods are primarily based on one-way monitoring, exhibiting certain limitations and time lags. Summary of the Invention
[0003] This application aims to at least partially address one of the technical problems in the related art.
[0004] Therefore, one objective of this application is to propose a coal and rock identification method, which involves monitoring a coal mining machine using a sound sensor to obtain the machine's audio signal; monitoring the coal mining machine using a vibration sensor to obtain its vibration signal; acquiring the audio time spectrum corresponding to the audio signal and the vibration time spectrum corresponding to the vibration signal, wherein the audio signal and the vibration signal are in a time-synchronized state; and determining the predicted category of the coal and rock cut by the coal mining machine based on the audio time spectrum and the vibration time spectrum.
[0005] The second objective of this application is to propose a coal and rock identification device.
[0006] The third objective of this application is to propose an electronic device.
[0007] The fourth objective of this application is to provide a non-transitory computer-readable storage medium.
[0008] The fifth objective of this application is to provide a computer program product.
[0009] To achieve the above objectives, the first aspect of this application proposes a coal and rock identification method, comprising: monitoring a coal mining machine using a sound sensor to obtain an audio signal of the coal mining machine; monitoring the coal mining machine using a vibration sensor to obtain a vibration signal of the coal mining machine; obtaining the audio time spectrum corresponding to the audio signal and the vibration time spectrum corresponding to the vibration signal, wherein the audio signal and the vibration signal are in a time-synchronized state; and determining the predicted category of the coal and rock cut by the coal mining machine based on the audio time spectrum and the vibration time spectrum.
[0010] This application determines the predicted category of coal and rock cut by the coal mining machine by fusing and analyzing the audio and vibration signals of the coal mining machine. This avoids the limitations and lag of a single monitoring signal and improves the accuracy of the predicted category of coal and rock cut by the coal mining machine.
[0011] According to one embodiment of this application, determining the predicted category of coal and rock cut by a coal mining machine based on audio time-frequency spectrum and vibration time-frequency spectrum includes: preprocessing the audio time-frequency spectrum and vibration time-frequency spectrum respectively to obtain an audio vector sequence corresponding to the preprocessed audio time-frequency spectrum and a vibration vector sequence corresponding to the vibration time-frequency spectrum; extracting modal features on the audio signal side based on the audio vector sequence to obtain an extracted audio feature vector sequence, and extracting modal features on the vibration signal side based on the vibration vector sequence to obtain an extracted vibration feature vector sequence; performing cross-modal feature fusion on the audio signal side and the vibration signal side based on the audio feature vector sequence, the vibration feature vector sequence, and a preset fusion vector to obtain a fused target fusion vector; and determining the predicted category of coal and rock cut by the coal mining machine based on the target fusion vector.
[0012] According to one embodiment of this application, the audio time spectrum and the vibration time spectrum are preprocessed to obtain the audio vector sequence corresponding to the preprocessed audio time spectrum and the vibration vector sequence corresponding to the vibration time spectrum. The preprocessing includes: inputting the audio time spectrum into a first preprocessing layer for linear transformation to obtain the audio vector sequence corresponding to the audio time spectrum output after linear transformation; and inputting the vibration time spectrum into a second preprocessing layer for linear transformation to obtain the vibration vector sequence corresponding to the vibration time spectrum output after linear transformation.
[0013] According to one embodiment of this application, modal feature extraction on the audio signal side is performed based on an audio vector sequence to obtain an extracted audio feature vector sequence, including: concatenating the audio vector sequence with a first classification vector and adding it to a preset position encoding matrix to obtain an audio classification vector sequence obtained after addition; inputting the audio classification vector sequence into an audio single-modal feature extraction layer to extract modal features on the audio signal side to obtain an extracted audio feature vector sequence.
[0014] According to one embodiment of this application, modal feature extraction of the vibration signal side based on the vibration vector sequence to obtain the extracted vibration feature vector sequence includes: concatenating the vibration vector sequence with a second classification vector and adding it to a preset position encoding matrix to obtain the vibration classification vector sequence obtained after addition; inputting the vibration classification vector sequence into a vibration single-mode feature extraction layer to extract modal features of the vibration signal side to obtain the extracted vibration feature vector sequence.
[0015] According to one embodiment of this application, determining the predicted category of coal and rock cut by the coal mining machine based on the target fusion vector includes: obtaining a first target classification vector corresponding to the audio signal side and a second target classification vector corresponding to the vibration signal side based on the target fusion vector; inputting the first target classification vector and the second target classification vector into the classifier layer to obtain the predicted category of coal and rock cut by the coal mining machine.
[0016] According to one embodiment of this application, the predicted categories include: coal category; rock category; coal plus rock category; and non-coal and non-rock category.
[0017] To achieve the above objectives, a second aspect of this application provides a coal and rock identification device, comprising: a first monitoring module for monitoring a coal mining machine via a sound sensor to acquire an audio signal from the coal mining machine; a second monitoring module for monitoring the coal mining machine via a vibration sensor to acquire a vibration signal from the coal mining machine; an acquisition module for acquiring the audio time spectrum corresponding to the audio signal and the vibration time spectrum corresponding to the vibration signal, wherein the audio signal and the vibration signal are time-synchronized; and a prediction module for determining the predicted category of the coal and rock cut by the coal mining machine based on the audio time spectrum and the vibration time spectrum.
[0018] According to one embodiment of this application, the prediction module is further configured to: preprocess the audio time spectrum and the vibration time spectrum respectively to obtain an audio vector sequence corresponding to the preprocessed audio time spectrum and a vibration vector sequence corresponding to the vibration time spectrum; extract modal features of the audio signal side based on the audio vector sequence to obtain an extracted audio feature vector sequence, and extract modal features of the vibration signal side based on the vibration vector sequence to obtain an extracted vibration feature vector sequence; perform cross-modal feature fusion on the audio signal side and the vibration signal side based on the audio feature vector sequence, the vibration feature vector sequence and a preset fusion vector to obtain a fused target fusion vector; and determine the predicted category of the coal and rock cut by the coal mining machine based on the target fusion vector.
[0019] According to one embodiment of this application, the prediction module is further configured to: input the audio time spectrum into a first preprocessing layer for linear transformation, and obtain an audio vector sequence corresponding to the audio time spectrum output after linear transformation; input the vibration time spectrum into a second preprocessing layer for linear transformation, and obtain a vibration vector sequence corresponding to the vibration time spectrum output after linear transformation.
[0020] According to one embodiment of this application, the prediction module is further configured to: concatenate the audio vector sequence with the first classification vector and add it to a preset position encoding matrix to obtain the audio classification vector sequence obtained after addition; input the audio classification vector sequence into the audio single-modal feature extraction layer to extract modal features on the audio signal side to obtain the extracted audio feature vector sequence.
[0021] According to one embodiment of this application, the prediction module is further configured to: concatenate the vibration vector sequence with the second classification vector and add it to a preset position encoding matrix to obtain the vibration classification vector sequence obtained after addition; input the vibration classification vector sequence into the vibration single-mode feature extraction layer to extract the modal features on the vibration signal side to obtain the vibration feature vector sequence generated after extraction.
[0022] According to one embodiment of this application, the prediction module is further configured to: obtain a first target classification vector corresponding to the audio signal side and a second target classification vector corresponding to the vibration signal side based on the target fusion vector; input the first target classification vector and the second target classification vector into the classifier layer to obtain the predicted category of the coal and rock cut by the coal mining machine.
[0023] According to one embodiment of this application, the predicted categories include: coal category; rock category; coal plus rock category; and non-coal and non-rock category.
[0024] To achieve the above objectives, a third aspect of this application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to implement the coal and rock identification method as described in the first aspect of this application.
[0025] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to implement the coal and rock identification method as described in the first aspect of this application.
[0026] To achieve the above objectives, a fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the coal and rock identification method as described in the first aspect of this application. Attached Figure Description
[0027] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:
[0028] Figure 1 This is an exemplary implementation of a coal and rock identification method shown in one embodiment of this application.
[0029] Figure 2 This is a schematic diagram illustrating an embodiment of the present application, showing how to determine the predicted category of coal and rock cut by a coal mining machine based on audio time spectrum and vibration time spectrum.
[0030] Figure 3This is a schematic diagram of a Transformer-based acoustic-vibration fusion classification model, as shown in one embodiment of this application.
[0031] Figure 4 This is a schematic diagram of the structure of a self-attention encoder layer shown in one embodiment of this application.
[0032] Figure 5 This is a schematic diagram of the structure of a multi-head self-attention module shown in one embodiment of this application.
[0033] Figure 6 This is a general flowchart illustrating a coal and rock identification method according to one embodiment of this application.
[0034] Figure 7 This is a schematic diagram of a coal and rock identification device according to one embodiment of this application.
[0035] Figure 8 This is a schematic diagram of an electronic device according to one embodiment of this application. Detailed Implementation
[0036] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.
[0037] Figure 1 This is an exemplary implementation of a coal and rock identification method proposed in this application, such as... Figure 1 As shown, the coal and rock identification method includes the following steps:
[0038] S101 monitors the coal mining machine using a sound sensor to obtain the machine's audio signal.
[0039] The coal mining machine is monitored using sound sensors to obtain its audio signals. Optionally, the audio signals can be acquired based on the sound signals generated when the cutter head cuts coal and rock, which are collected by a microphone array installed on the forearm of the coal mining machine.
[0040] S102 monitors the coal mining machine using vibration sensors to obtain vibration signals from the coal mining machine.
[0041] The coal mining machine is monitored using vibration sensors to obtain its vibration signals. Optionally, the vibration signals of the coal mining machine's forearm can be collected based on an acceleration vibration sensor installed near the microphone array.
[0042] S103, acquire the audio time spectrum corresponding to the audio signal and the vibration time spectrum corresponding to the vibration signal, wherein the audio signal and the vibration signal are in a time-synchronized state.
[0043] For the audio signal obtained above, a short-time Fourier transform is used to convert it into an initial time spectrum. The initial time spectrum is then multiplied by a set of Mel filter banks to convert the linear frequency scale into a logarithmic Mel scale that is closer to the human ear's pitch perception. The logarithm of the energy of the Mel spectrum is then used to obtain the logarithmic Mel spectrum, and the obtained logarithmic Mel spectrum is used as the audio time spectrum corresponding to the audio signal.
[0044] The vibration signal obtained above is subjected to continuous wavelet transform to obtain the vibration time spectrum of the vibration signal.
[0045] It is important to note that the time window length of the continuous wavelet transform of the vibration signal and the time window length of the short-time Fourier transform of the acoustic signal can be different, but the two signals need to be synchronized in time.
[0046] Optionally, timestamps can be added to the acquired audio and vibration signals, and the two signals can be synchronized in time using the timestamps.
[0047] S104, based on the audio time spectrum and vibration time spectrum, determine the predicted category of coal and rock cut by the coal mining machine.
[0048] Based on the audio time spectrum corresponding to the audio signal and the vibration time spectrum corresponding to the vibration signal obtained above, modal features are extracted from the audio signal and vibration signal sides respectively. Cross-modal feature fusion is performed on the outputs after modal feature extraction from both sides. Based on the target fusion vector obtained after fusion, the predicted category of coal and rock cut by the coal mining machine is determined.
[0049] The predicted categories of coal and rock cut by the coal mining machine can include four categories: coal category, that is, the object cut by the coal mining machine is coal; rock category, that is, the object cut by the coal mining machine is rock; coal and rock category, that is, the object cut by the coal mining machine is both coal and rock; and non-coal and non-rock category, that is, the object cut by the coal mining machine is neither coal nor rock.
[0050] This application proposes a coal and rock identification method. The specific implementation scheme is as follows: A sound sensor monitors the coal mining machine to obtain its audio signal; a vibration sensor monitors the coal mining machine to obtain its vibration signal; the audio time spectrum corresponding to the audio signal and the vibration time spectrum corresponding to the vibration signal are obtained, wherein the audio signal and vibration signal are time-synchronized; based on the audio time spectrum and vibration time spectrum, the predicted category of the coal and rock cut by the coal mining machine is determined. This application determines the predicted category of the coal and rock cut by the coal mining machine by fusing and analyzing the audio and vibration signals, avoiding the limitations and lag of a single monitoring signal, and improving the accuracy of the predicted category of the coal and rock cut by the coal mining machine.
[0051] Figure 2 This is an exemplary implementation of a coal and rock identification method proposed in this application, such as... Figure 2 As shown, based on the above embodiments, the predicted category of coal and rock cut by the coal mining machine is determined according to the audio time spectrum and vibration time spectrum, including the following steps:
[0052] S201, preprocess the audio time spectrum and the vibration time spectrum respectively to obtain the audio vector sequence corresponding to the preprocessed audio time spectrum and the vibration vector sequence corresponding to the vibration time spectrum.
[0053] The audio time-frequency spectrum and vibration time-frequency spectrum are used as the audio signal side and vibration signal side inputs to a pre-trained Transformer-based acoustic-vibration fusion classification model, respectively. Figure 3 This is a schematic diagram of the Transformer-based acoustic-vibration fusion classification model in this application, as shown below. Figure 3 As shown, the Transformer-based acoustic-vibration fusion classification model includes a preprocessing layer, a Transformer encoder layer, and a classification layer. The first few layers of the Transformer encoder layer are single-modal feature extraction layers, while the later layers are cross-modal feature fusion layers.
[0054] In this application, the preprocessing layer on the audio signal side is referred to as the first preprocessing layer, and the preprocessing layer on the vibration signal side is referred to as the second preprocessing layer, such as... Figure 3 As shown, the audio time spectrum is input into the first preprocessing layer for linear projection. The first preprocessing layer divides and reduces the dimension of the input audio time spectrum, transforming it into a two-dimensional data block sequence that slightly overlaps on both the time and frequency axes. The two-dimensional data blocks are flattened and transformed into audio vectors of a specific length through a linear transformation. All the audio vectors form an audio vector sequence.
[0055] Similarly, the vibration time spectrum is input into the second preprocessing layer for linear projection. The second preprocessing layer divides and reduces the dimension of the input vibration time spectrum, transforming it into a two-dimensional data block sequence that slightly overlaps on both the time and frequency axes. The two-dimensional data blocks are flattened and transformed into vibration vectors of a specific length through a linear transformation. All the vibration vectors form a vibration vector sequence.
[0056] In this application, the Transformer-based acoustic-vibration fusion classification model is pre-trained and saved. When training the Transformer-based acoustic-vibration fusion classification model, the audio and vibration signals of the coal mining machine can be sampled when the predicted category of the coal and rock cut by the coal mining machine is coal, rock, coal plus rock, or non-coal and non-rock. The sampled audio and vibration signals are used as the training set to train the initial model to obtain the final trained Transformer-based acoustic-vibration fusion classification model.
[0057] S202, Modal feature extraction of the audio signal side based on the audio vector sequence to obtain the extracted audio feature vector sequence, and modal feature extraction of the vibration signal side based on the vibration vector sequence to obtain the extracted vibration feature vector sequence.
[0058] The audio vector sequence is concatenated with the first classification vector and then added to the preset position encoding matrix to obtain the audio classification vector sequence. The audio classification vector sequence is then input into a multi-layer audio single-modal feature extraction layer to extract modal features on the audio signal side, thereby obtaining the extracted audio feature vector sequence.
[0059] The vibration vector sequence is concatenated with the second classification vector and then added to the preset position encoding matrix to obtain the vibration classification vector sequence. The vibration classification vector sequence is then input into the multi-layer vibration single-mode feature extraction layer to extract the modal features of the vibration signal side, so as to obtain the vibration feature vector sequence generated after extraction.
[0060] Based on the above, the general formula for concatenating the vibration vector sequence with the second classification vector and then adding it to the preset position encoding matrix, or concatenating the audio vector sequence with the first classification vector and then adding it to the preset position encoding matrix, is as follows:
[0061] z = [z cls Ex1, Ex2, ... Ex N ]+p
[0062] In the above formula, x i Represents a one-dimensional matrix after the data block is flattened; E represents linear projection; z clsrepresents the classification vector used for classification; p represents the preset position encoding matrix.
[0063] The audio single-modal feature extraction layer and the vibration single-modal feature extraction layer are located on either side of the single-modal feature extraction layer. Figure 4 This is a schematic diagram of the structure of a self-attention encoder layer, as shown below. Figure 4 As shown, each Transformer encoder layer, that is, each self-attention encoder layer, is a residual structure, consisting of a multi-head self-attention module, a layer normalization module, and a multilayer perceptron module.
[0064] The above audio classification vector sequence and vibration classification vector sequence are input into the corresponding self-attention encoder layer on their respective sides, and the output of the self-attention encoder layer is obtained. The formula for the self-attention encoder layer is as follows:
[0065] y l =MSA(LN(z) l ))+z l
[0066] z l+1 =MLP(LN(y l ))+y l
[0067] In the above formula, MSA represents a multilayer self-attention module; LN represents a layer normalization module; MLP represents a multilayer perceptron module; and l represents the layer number of the current self-attention encoder layer.
[0068] in, Figure 5 This is a structural diagram of a multi-head self-attention module, such as... Figure 5 As shown, the multi-head self-attention module contains multiple sets of independent learnable fully connected layers to perform linear projection to transform queries, keys, and values. The queries, keys, and values come from the same set of inputs. The transformed queries, keys, and values are fed into attention convergence in parallel. Finally, the outputs of multiple attention convergences are concatenated together and transformed through another learnable fully connected layer to produce the final output.
[0069] Based on the above, the audio feature vector sequence and the vibration feature vector sequence generated after modal extraction can be expressed by the following formulas:
[0070]
[0071] In the above formula, aud represents the audio signal; vib represents the vibration signal; L f θ represents the starting layer number of the cross-modal feature fusion layer, l represents the current layer number of the self-attention encoder layer, and θ represents other parameters in the model.
[0072] S203, based on the audio feature vector sequence, the vibration feature vector sequence and the preset fusion vector, performs cross-modal feature fusion on the audio signal side and the vibration signal side to obtain the fused target fusion vector.
[0073] Based on the above, the obtained audio feature vector sequence, vibration feature vector sequence, and preset fusion vector are concatenated using the formula z = [z aud , z fus , z vib ] indicates that in the above formula, z aud Represents an audio feature vector sequence; z vib Represents the sequence of vibrational eigenvectors; z fus This represents a preset fusion vector used for multimodal fusion.
[0074] For the audio signal side, a multi-layer cross-modal feature fusion layer takes the audio feature vector sequence and the preset fusion vector as input to obtain the output vector of the audio signal side output by the self-attention encoder layer of the audio signal side in the cross-modal feature fusion layer and the intermediate fusion vector that is biased towards the audio signal side.
[0075] For the vibration signal side, a multi-layer cross-modal feature fusion layer takes the vibration feature vector sequence and the preset fusion vector as input to obtain the output vector of the vibration signal side output by the self-attention encoder layer of the vibration signal side in the cross-modal feature fusion layer and the intermediate fusion vector that is biased towards the vibration signal side.
[0076] The process of multi-layer cross-modal feature fusion layers on the audio signal side and the vibration signal side can be expressed by the following formula:
[0077]
[0078] In the above formula, z fus This represents the preset fusion vector; z fus_aud This represents an intermediate fusion vector that prioritizes the audio signal; z fus_vib θ represents the intermediate fusion vector that is biased towards vibration signals, θ represents other parameters in the model, and l represents the layer number of the current self-attention encoder layer.
[0079] The average of the intermediate fusion vectors of the audio signal and the vibration signal is taken as the average fusion vector of the final output of this layer, expressed by the formula:
[0080]
[0081] In the above formula, z represents the average fusion vector; fus_aud This represents an intermediate fusion vector that prioritizes the audio signal; z fus_vid This represents an intermediate fusion vector that prioritizes vibration signals.
[0082] The final target fusion vector output by the last self-attention encoder layer is expressed by the formula:
[0083]
[0084] In the above formula, z l+1 This represents the final target fusion vector output by the last self-attention encoder layer. Represents the average fusion vector. This represents the target audio signal vector output by the final self-attention encoder layer. This represents the target vibration signal vector that is finally output by the last self-attention encoder layer.
[0085] In this application, the existence of the average fusion vector enables the feature extraction of audio and vibration signals to obtain contextual information of the other side of the signal. Furthermore, since intermodal information fusion must be achieved through a small number of fusion vectors, only the information most relevant to the most abstract task will be fused.
[0086] S204, based on the target fusion vector, determines the predicted category of the coal and rock cut by the coal mining machine.
[0087] Based on the target fusion vector, the first target classification vector corresponding to the audio signal side and the second target classification vector corresponding to the vibration signal side are obtained. The first and second target classification vectors are input into the classifier layer. The classifier layer contains two layers: a fully connected layer and a softmax layer. The values obtained by performing the same linear transformation (fully connected layer) on the first and second target classification vectors are averaged, and then used as the input of the softmax layer. Finally, a value between 0 and 1 is output to obtain the predicted category of the coal and rock cut by the coal mining machine. The predicted category may be: [0, 0] indicating no coal seam or rock strata are cut; [1, 0] indicating coal seam is cut; [0, 1] indicating rock strata are cut; [1, 1] indicating both coal seam and rock strata (coal gangue) are cut.
[0088] In this embodiment, a neural network based on the Transformer architecture performs multimodal fusion on the audio signals and vibration signals of the robotic arm simultaneously acquired during coal mining machine cutting coal and rock to identify the working state of the coal mining machine. The single-modal feature extraction mechanism uses the same neural network framework to extract and fuse features from both the audio and vibration signals, eliminating the need for a modality-specific feature extraction network based on prior knowledge. A cross-modal attention mechanism is used to ensure that the single-modal signal fully utilizes the contextual information of other modal signals during processing. The first few layers of the model use only self-attention for single-channel feature extraction, eliminating redundant information. The later layers introduce a cross-modal attention mechanism to fuse the simplified multimodal information, reducing model complexity.
[0089] Figure 6 This is a flowchart of the overall process for a coal and rock identification method proposed in this application, as follows: Figure 6 As shown, the coal and rock identification method includes the following steps:
[0090] The S601 monitors the coal mining machine using a sound sensor to obtain its audio signals.
[0091] S602 monitors the coal mining machine using vibration sensors to obtain vibration signals from the coal mining machine.
[0092] S603, obtain the audio time spectrum corresponding to the audio signal.
[0093] S604, acquire the vibration time spectrum corresponding to the vibration signal, wherein the audio signal and the vibration signal are in a time-synchronized state.
[0094] For details on the specific implementation of steps S601 to S604, please refer to the relevant parts of the above embodiments, which will not be repeated here.
[0095] S605, input the audio time spectrum into the first preprocessing layer for linear transformation, and obtain the audio vector sequence corresponding to the audio time spectrum output after linear transformation.
[0096] S606, input the vibration time spectrum into the second preprocessing layer for linear transformation, and obtain the vibration vector sequence corresponding to the vibration time spectrum output after linear transformation.
[0097] S607, the audio vector sequence is concatenated with the first classification vector and then added to the preset position encoding matrix to obtain the audio classification vector sequence obtained after addition.
[0098] S608, input the audio classification vector sequence into the audio single-modal feature extraction layer to extract modal features on the audio signal side, so as to obtain the extracted audio feature vector sequence.
[0099] S609, the vibration vector sequence is concatenated with the second classification vector and then added to the preset position encoding matrix to obtain the vibration classification vector sequence obtained after addition.
[0100] S610, input the vibration classification vector sequence into the vibration single-mode feature extraction layer to extract the modal features of the vibration signal side, so as to obtain the vibration feature vector sequence generated after extraction.
[0101] S611 performs cross-modal feature fusion on the audio signal side and the vibration signal side based on the audio feature vector sequence, the vibration feature vector sequence and the preset fusion vector to obtain the target fusion vector after fusion.
[0102] S612, based on the target fusion vector, obtain the first target classification vector corresponding to the audio signal side and the second target classification vector corresponding to the vibration signal side.
[0103] S613, input the first target classification vector and the second target classification vector into the classifier layer to obtain the predicted category of the coal and rock cut by the coal mining machine.
[0104] For details on the specific implementation of steps S605 to S613, please refer to the relevant parts of the above embodiments, which will not be repeated here.
[0105] This application proposes a coal and rock identification method. The specific implementation scheme is as follows: A sound sensor monitors the coal mining machine to obtain its audio signal; a vibration sensor monitors the coal mining machine to obtain its vibration signal; the audio time spectrum corresponding to the audio signal and the vibration time spectrum corresponding to the vibration signal are obtained, wherein the audio signal and vibration signal are time-synchronized; based on the audio time spectrum and vibration time spectrum, the predicted category of the coal and rock cut by the coal mining machine is determined. This application determines the predicted category of the coal and rock cut by the coal mining machine by fusing and analyzing the audio and vibration signals, avoiding the limitations and lag of a single monitoring signal, and improving the accuracy of the predicted category of the coal and rock cut by the coal mining machine.
[0106] Figure 7 This is a schematic diagram of a coal and rock identification device proposed in this application, such as... Figure 7 As shown, the coal and rock identification device 700 includes a first monitoring module 701, a second monitoring module 702, an acquisition module 703, and a prediction module 704, wherein:
[0107] The first monitoring module 701 is used to monitor the coal mining machine through a sound sensor to obtain the audio signal of the coal mining machine;
[0108] The second monitoring module 702 is used to monitor the coal mining machine through a vibration sensor to obtain the vibration signal of the coal mining machine;
[0109] The acquisition module 703 is used to acquire the audio time spectrum corresponding to the audio signal and the vibration time spectrum corresponding to the vibration signal, wherein the audio signal and the vibration signal are in a time-synchronized state.
[0110] The prediction module 704 is used to determine the predicted category of the coal and rock cut by the coal mining machine based on the audio time spectrum and the vibration time spectrum.
[0111] This application proposes a coal and rock identification method, comprising: a first monitoring module for monitoring a coal mining machine using a sound sensor to acquire its audio signal; a second monitoring module for monitoring the coal mining machine using a vibration sensor to acquire its vibration signal; an acquisition module for acquiring the audio time spectrum corresponding to the audio signal and the vibration time spectrum corresponding to the vibration signal, wherein the audio signal and vibration signal are time-synchronized; and a prediction module for determining the predicted category of the coal and rock cut by the coal mining machine based on the audio time spectrum and vibration time spectrum. This application determines the predicted category of the coal and rock cut by the coal mining machine by fusing and analyzing the audio and vibration signals, avoiding the limitations and lag of a single monitoring signal, and improving the accuracy of the predicted category of the coal and rock cut by the coal mining machine.
[0112] Furthermore, the prediction module 704 is also used to: preprocess the audio time spectrum and the vibration time spectrum respectively to obtain the audio vector sequence corresponding to the preprocessed audio time spectrum and the vibration vector sequence corresponding to the vibration time spectrum; extract modal features on the audio signal side based on the audio vector sequence to obtain the extracted audio feature vector sequence, and extract modal features on the vibration signal side based on the vibration vector sequence to obtain the extracted vibration feature vector sequence; perform cross-modal feature fusion on the audio signal side and the vibration signal side based on the audio feature vector sequence, the vibration feature vector sequence and the preset fusion vector to obtain the fused target fusion vector; and determine the predicted category of the coal and rock cut by the coal mining machine based on the target fusion vector.
[0113] Furthermore, the prediction module 704 is also used to: input the audio time spectrum into the first preprocessing layer for linear transformation, and obtain the audio vector sequence corresponding to the audio time spectrum output after linear transformation; input the vibration time spectrum into the second preprocessing layer for linear transformation, and obtain the vibration vector sequence corresponding to the vibration time spectrum output after linear transformation.
[0114] Furthermore, the prediction module 704 is also used to: concatenate the audio vector sequence with the first classification vector and add it to the preset position encoding matrix to obtain the audio classification vector sequence obtained after addition; input the audio classification vector sequence into the audio single-modal feature extraction layer to extract the modal features on the audio signal side to obtain the extracted audio feature vector sequence.
[0115] Furthermore, the prediction module 704 is also used to: concatenate the vibration vector sequence with the second classification vector and add it to the preset position encoding matrix to obtain the vibration classification vector sequence obtained after addition; input the vibration classification vector sequence into the vibration single-mode feature extraction layer to extract the modal features on the vibration signal side to obtain the vibration feature vector sequence generated after extraction.
[0116] Furthermore, the prediction module 704 is also used to: obtain a first target classification vector corresponding to the audio signal side and a second target classification vector corresponding to the vibration signal side based on the target fusion vector; input the first target classification vector and the second target classification vector into the classifier layer to obtain the predicted category of the coal and rock cut by the coal mining machine.
[0117] Furthermore, the predicted categories of coal and rock cut by the aforementioned coal mining machine include: coal category; rock category; coal plus rock category; and non-coal and non-rock category.
[0118] To implement the above embodiments, this application also proposes an electronic device 800, such as... Figure 8 As shown, the electronic device 800 includes a processor 801 and a memory 802 communicatively connected to the processor. The memory 802 stores instructions that can be executed by at least one processor. The instructions are executed by at least one processor 801 to implement the coal and rock identification method as shown in the above embodiment.
[0119] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to implement the coal and rock identification method as shown in the above embodiments.
[0120] To implement the above embodiments, this application also proposes a computer program product, including a computer program that, when executed by a processor, implements the coal and rock identification method as shown in the above embodiments.
[0121] In the description of this application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicating the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.
[0122] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.
[0123] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0124] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.
Claims
1. A method for coal and rock identification, characterized in that, include: The coal mining machine is monitored by a sound sensor to obtain the audio signal of the coal mining machine; The coal mining machine is monitored by vibration sensors to obtain its vibration signals; The audio time spectrum corresponding to the audio signal and the vibration time spectrum corresponding to the vibration signal are obtained, wherein the audio signal and the vibration signal are in a time-synchronized state; The audio time spectrum is input into the first preprocessing layer for linear transformation to obtain the audio vector sequence corresponding to the audio time spectrum after linear transformation; The vibration time spectrum is input into the second preprocessing layer for linear transformation to obtain the vibration vector sequence corresponding to the vibration time spectrum after linear transformation; Modal feature extraction is performed on the audio signal side based on the audio vector sequence to obtain the extracted audio feature vector sequence, and modal feature extraction is performed on the vibration signal side based on the vibration vector sequence to obtain the extracted vibration feature vector sequence. Based on the audio feature vector sequence, the vibration feature vector sequence, and the preset fusion vector, cross-modal feature fusion is performed on the audio signal side and the vibration signal side to obtain the fused target fusion vector; Based on the target fusion vector, the predicted category of the coal and rock cut by the coal mining machine is determined.
2. The method according to claim 1, characterized in that, The step of extracting modal features from the audio signal based on the audio vector sequence to obtain the extracted audio feature vector sequence includes: The audio vector sequence is concatenated with the first classification vector and then added to a preset position encoding matrix to obtain the audio classification vector sequence obtained after the addition. The audio classification vector sequence is input into the audio single-modal feature extraction layer to extract modal features on the audio signal side, so as to obtain the audio feature vector sequence generated after extraction.
3. The method according to claim 2, characterized in that, The modal feature extraction based on the vibration vector sequence to obtain the extracted vibration feature vector sequence includes: The vibration vector sequence is concatenated with the second classification vector and then added to the preset position encoding matrix to obtain the vibration classification vector sequence obtained after addition. The vibration classification vector sequence is input into the vibration single-mode feature extraction layer to extract the modal features on the vibration signal side, so as to obtain the vibration feature vector sequence generated after extraction.
4. The method according to claim 3, characterized in that, The step of determining the predicted category of coal and rock cut by the coal mining machine based on the target fusion vector includes: Based on the target fusion vector, a first target classification vector corresponding to the audio signal side and a second target classification vector corresponding to the vibration signal side are obtained; The first target classification vector and the second target classification vector are input into the classifier layer to obtain the predicted category of the coal and rock cut by the coal mining machine.
5. The method according to any one of claims 1-4, characterized in that, The prediction categories include: Coal categories; Rock categories; Coal plus rock categories; Non-coal, non-rock category.
6. A coal and rock identification device, characterized in that, include: The first monitoring module is used to monitor the coal mining machine through a sound sensor to obtain the audio signal of the coal mining machine; The second monitoring module is used to monitor the coal mining machine through vibration sensors to obtain the vibration signal of the coal mining machine; The acquisition module is used to acquire the audio time spectrum corresponding to the audio signal and the vibration time spectrum corresponding to the vibration signal, wherein the audio signal and the vibration signal are in a time-synchronized state; The prediction module is used to input the audio time spectrum into a first preprocessing layer for linear transformation to obtain an audio vector sequence corresponding to the audio time spectrum after linear transformation; input the vibration time spectrum into a second preprocessing layer for linear transformation to obtain a vibration vector sequence corresponding to the vibration time spectrum after linear transformation; perform modal feature extraction on the audio signal side based on the audio vector sequence to obtain an extracted audio feature vector sequence, and perform modal feature extraction on the vibration signal side based on the vibration vector sequence to obtain an extracted vibration feature vector sequence; perform cross-modal feature fusion on the audio signal side and the vibration signal side based on the audio feature vector sequence, the vibration feature vector sequence, and a preset fusion vector to obtain a fused target fusion vector; and determine the predicted category of the coal and rock cut by the coal mining machine based on the target fusion vector.
7. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.
8. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-5.