Coal mining machine cutting state recognition method and device based on multi-modal data perception

Abstract

The present disclosure relates to the technical field of coal mining machine, and particularly provides a coal mining machine cutting state recognition method and device based on multi-modal data perception. The method comprises: collecting multi-modal data including cutting visual data, cutting audio data and cutting load time series data of the drum cutting part of the coal mining machine during the cutting process of the drum cutting part; respectively extracting features of the cutting visual data, the cutting audio data and the cutting load time series data to obtain image features, audio features and load time series features of the drum cutting part; fusing the image features, the audio features and the load time series features to generate multi-modal fusion features; inputting the multi-modal fusion features into a cutting state recognition model to output a recognition result of the cutting state of the drum cutting part; and adjusting the cutting height of the drum cutting part based on the recognition result of the cutting state. The present disclosure can realize remote control and adaptive adjustment of the height of the drum cutting part, and improve the cutting efficiency of the coal mining machine.

Description

Technical Field

[0001] This disclosure relates to the field of coal mining machine technology, and in particular to a method and apparatus for identifying the cutting status of a coal mining machine based on multimodal data perception. Background Technology

[0002] With the advancement of intelligent coal mining and the introduction of the concept of "unmanned operation for safety," high-tech and artificial intelligence (AI) empowering safe and efficient coal mine production is an inevitable trend. As one of the three core pieces of equipment in a fully mechanized coal mining face, and an indispensable piece of mining equipment, the ability of coal mining machines to achieve intelligent operation, remote coal cutting, and autonomous cutting directly impacts the implementation of the "unmanned operation for safety" concept, and indirectly affects coal mine capacity and energy output.

[0003] In related technologies, the control of the drum height of underground coal mining machines is mostly based on manual judgment. The drum height and speed are adjusted in real time by combining the actual coal and rock cutting conditions of the coal mining machine, years of accumulated human experience, geological conditions and mine pressure, etc., in order to reduce the wear and tear on the coal mining equipment and the physical injury to personnel caused by the complex underground geological environment. Summary of the Invention

[0004] This disclosure is made in view of the above-mentioned problems. This disclosure provides a method and apparatus for identifying the cutting status of a coal mining machine based on multimodal data perception.

[0005] According to one aspect of this disclosure, a method for identifying the cutting state of a coal mining machine based on multimodal data perception is provided, comprising: During the cutting process of the drum cutting section of the coal mining machine, multimodal data of the drum cutting section is collected; wherein, the multimodal data includes at least cutting visual data, cutting audio data, and cutting load time series data; Feature extraction is performed on the cutting visual data, the cutting audio data, and the cutting load time-series data respectively to obtain the image features, audio features, and load time-series features of the drum cutting section; The image features, audio features, and load temporal features are fused to generate multimodal fusion features; The multimodal fusion features are input into a pre-constructed cutting state recognition model, and the recognition result of the cutting state of the drum cutting section is output; wherein, the cutting state includes at least coal cutting state and rock cutting state; Based on the identification result of the cutting state, the sampling height of the drum cutting section is adjusted.

[0006] According to another aspect of this disclosure, a coal mining machine cutting status identification device based on multimodal data perception is provided, comprising: The acquisition module is used to acquire multimodal data of the drum cutting section during the cutting process of the coal mining machine; wherein the multimodal data includes at least cutting visual data, cutting audio data, and cutting load time-series data. The processing module is used to extract features from the cutting visual data, the cutting audio data, and the cutting load time-series data respectively, to obtain the image features, audio features, and load time-series features of the drum cutting section; The processing module is also used to fuse the image features, the audio features, and the load time-series features to generate multimodal fusion features; The processing module is also used to input the multimodal fusion features into a pre-constructed cutting state recognition model and output the recognition result of the cutting state of the drum cutting section; wherein, the cutting state includes at least coal cutting state and rock cutting state; An adjustment module is used to adjust the sampling height of the drum cutting section based on the identification result of the cutting state.

[0007] In another aspect of exemplary embodiments of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement the methods described in exemplary embodiments of this disclosure.

[0008] In another aspect of exemplary embodiments of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the methods described in exemplary embodiments of the present disclosure.

[0009] In another aspect of the exemplary embodiments of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods described in the exemplary embodiments of this disclosure.

[0010] As will be described in detail below, the coal mining machine cutting state recognition method based on multimodal data perception according to embodiments of this disclosure involves collecting multimodal data of the drum cutting section during the cutting process of the coal mining machine; wherein the multimodal data includes at least cutting visual data, cutting audio data, and cutting load time-series data; feature extraction is performed on the cutting visual data, cutting audio data, and cutting load time-series data respectively to obtain image features, audio features, and load time-series features of the drum cutting section; and the image features, audio features, and load time-series features are fused to generate multimodal fusion features. The multimodal fusion features are input into a pre-constructed cutting state recognition model, and the recognition results of the cutting state of the drum cutting section are output. The cutting state includes at least coal cutting state and rock cutting state. Based on the recognition results of the cutting state, the mining height of the drum cutting section is adjusted. The cutting state recognition model can be used to identify the cutting state of the drum cutting section in real time based on the multimodal data of the drum cutting section, thereby realizing remote control and adaptive adjustment of the height of the drum cutting section. This achieves the goal of improving the cutting efficiency of the coal mining machine and reducing manpower, while also improving the overall recovery rate of the coal mining face and increasing resource utilization.

[0011] It should be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further illustration of the claimed technology. Attached Figure Description

[0012] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0013] Figure 1 This illustration shows the equipment layout diagram of the fully mechanized mining face where the coal mining machine is located, provided by an exemplary embodiment of this disclosure. Figure 2 A schematic diagram of the structure of a coal mining machine provided in an exemplary embodiment of this disclosure is shown; Figure 3 A flowchart illustrating the coal mining machine cutting status identification method based on multimodal data perception provided in an exemplary embodiment of this disclosure is shown. Figure 4 A schematic diagram of the framework of a Transformer network provided in an exemplary embodiment of this disclosure is shown; Figure 5 A schematic diagram illustrating the principle of the Transformer network provided in an exemplary embodiment of this disclosure is shown. Figure 6A schematic diagram of the data acquisition and transmission architecture provided by an exemplary embodiment of this disclosure is shown; Figure 7 A schematic diagram illustrating the framing process of an audio signal provided in an exemplary embodiment of this disclosure is shown. Figure 8 A schematic diagram illustrating the framing process of an audio signal provided in an exemplary embodiment of this disclosure is shown. Figure 9 This illustration shows a schematic diagram of the structure of a coal mining machine cutting status identification device based on multimodal data perception provided in an exemplary embodiment of the present disclosure; Figure 10 A schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this disclosure is shown; Figure 11 A schematic diagram of the structure of a computer system provided in an exemplary embodiment of this disclosure is shown.

[0014] Figure label: 101-Coal mining machine, 102-Scraper conveyor, 103-Hydraulic support, 104-Lower end hydraulic support, 105-Upper end hydraulic support, 106-Transfer conveyor, 107-Belt conveyor, 108-Distribution box, 109-Emulsion pump station, 110-Equipment train, 111-Mobile substation, 113-Hydraulic safety winch, 211-Cutting section, 2111-Left cutting section, 2112-Right cutting section, 212-Rocker arm, 213-Electrical control box, 214-Camera. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0016] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0017] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc., used in this disclosure are only used to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0018] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0019] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0020] Currently, methods for identifying the height of coal mining machine drums mainly include process signal monitoring, infrared thermal imaging, image recognition, spectral analysis, and ultrasonic and electromagnetic wave detection. Except for image feature recognition, other methods rely on the differences in the physical properties of coal and rock. Image feature recognition, on the other hand, distinguishes between coal and rock by comparing differences in brightness, texture, and shape in images through model learning. By sensing the distribution and variation curves of coal and rock in advance, the height of the coal mining machine drum can be adjusted and controlled.

[0021] The limitations of the above methods are that, although the process monitoring signals have a certain degree of universality, they cannot overcome the randomness of the signals caused by factors such as excessive reliance on the differences in coal and rock hardness, limitations of the geological environment, high requirements for the timeliness of real-time data transmission, and non-uniformity of coal seam media. This results in certain limitations in the monitoring of cutting process signals.

[0022] Infrared thermal imaging identification identifies coal and rock by stimulating them with infrared light to change their temperature. Although it is more accurate than other methods, it has drawbacks such as long excitation time, inability to quickly identify coal and rock, significant safety hazards, and susceptibility of identification accuracy to external factors, which greatly limits its application in engineering.

[0023] Image feature recognition has high requirements for image quality, water mist, lighting, and lamp reflection in complex coal mining environments, and its accuracy is easily affected by these factors. Reflectance spectral recognition distinguishes coal and rock by comparing their absorption and reflectance. This method is currently in its early stages and lacks significant results and industrial-scale experimental cases.

[0024] Ultrasonic detection and identification methods are mostly used for pre-mining exploration, determining coal thickness based on differences in acoustic impedance between coal and rock. However, this method is susceptible to interference from echo data generated by interbedded materials such as gangue and iron sulfide within the coal seam. Furthermore, intermittent detection during the cutting process consumes significant manpower and resources, reducing mining efficiency.

[0025] Electromagnetic wave detection and identification is currently mainly used for coal seam thickness detection. However, it is easily affected by factors such as medium homogeneity, geological faults, metals, and impurities, making it difficult to promote its application.

[0026] Therefore, in order to solve the above problems, this disclosure provides a method for identifying the cutting status of a coal mining machine based on multimodal data perception. By introducing multimodal data such as images, sounds, current and torque during the production process, the method performs pre-model fusion and establishes a cutting status identification model to identify the cutting status of the coal mining machine drum in real time. This enables remote control and adaptive adjustment of the drum height, thereby improving the cutting efficiency of the coal mining machine and reducing manpower. At the same time, it can improve the overall recovery rate of the coal mining face and increase resource utilization.

[0027] Figure 1 This illustration shows the equipment layout of the fully mechanized mining face where the coal mining machine is located, as provided in an exemplary embodiment of this disclosure. Figure 1 As shown, a fully mechanized longwall mining face is generally composed of a coal mining machine 101, a scraper conveyor 102, a hydraulic support 103, a lower end hydraulic support 104, an upper end hydraulic support 105, a transfer conveyor 106, a crusher, a belt conveyor 107, a power distribution box 108, an emulsion pump station 109, an equipment train 110, a mobile substation 111, and a hydraulic safety winch 113, etc.

[0028] Figure 2 A schematic diagram of the structure of a coal mining machine provided in an exemplary embodiment of this disclosure is shown. Figure 2As shown, the coal mining machine mainly consists of a cutting section 211 (left cutting section 2111 and right cutting section 2112), left and right traction sections, an intermediate electrical system, and auxiliary devices. The cutting section 211 includes the connecting lugs of the rocker arm 212, a cutting motor, a cutting gear reducer, a auger drum, an auger drum height adjustment device, and a coal retaining plate. The cutting section 211 primarily functions to cut, break, and load coal. The traction section includes a traction gear reducer and a traction mechanism. The traction mechanism is the actuator for moving the coal mining machine and can be further divided into chain traction and chainless traction. The main function of the traction section is to control the coal mining machine to run along the working face as required and to provide overload protection for the coal mining machine. The intermediate electrical system, i.e., the electrical control box 213, includes power electrical components, traction speed control system electrical components, various protection and fault diagnosis control, status display, and alarm devices. The main function of this system is to provide power to the coal mining machine. The auxiliary equipment includes a crushing mechanism, a tilting coal retaining plate mechanism (additional), a cable dragging device, a body tilting hydraulic cylinder for the coal mining machine (additional), a drum height adjustment hydraulic cylinder, a coal mining machine guiding device, an oil tank, a cooling system, a spray dust suppression system, etc. The main function of this device is to form a complete functional system of the coal mining machine together with the main components to meet the requirements of efficient and safe coal mining.

[0029] The coal mining machine cutting status recognition method based on multimodal data perception provided in this disclosure can be executed by a terminal or by a chip applied to the terminal.

[0030] For example, the terminal may include one or more of the following: mobile phone, tablet computer, wearable device, in-vehicle device, laptop computer, ultra-mobile personal computer (UMPC), netbook, PDA, and wearable device based on augmented reality (AR) and / or virtual reality (VR) technology. The exemplary embodiments disclosed herein do not impose specific limitations on these.

[0031] Figure 3 A flowchart illustrating the coal mining machine cutting status identification method based on multimodal data perception provided in an exemplary embodiment of this disclosure is shown. Figure 3 As shown, the coal mining machine cutting status identification method based on multimodal data perception includes: S301, during the cutting process of the drum cutting section of the coal mining machine, multimodal data of the drum cutting section is collected; wherein, the multimodal data includes at least cutting visual data, cutting audio data and cutting load time series data; S302, feature extraction is performed on the cutting visual data, cutting audio data and cutting load time series data respectively to obtain the image features, audio features and load time series features of the drum cutting section; S303, which fuses image features, audio features, and load time-series features to generate multimodal fusion features; S304, input the multimodal fusion features into the pre-constructed cutting state recognition model, and output the recognition result of the cutting state of the drum cutting section; wherein, the cutting state includes at least coal cutting state and rock cutting state; S305, based on the identification results of the cutting status, adjusts the sampling height of the drum cutting section.

[0032] Specifically, the drum cutting section of the coal mining machine can be as follows: Figure 2 The cutting section 211 shown (including the left cutting section 2111 and the right cutting section 2112) contains left and right drums, which are devices used for cutting and breaking coal within the cutting section 211. During the cutting process of the drum cutting section of the coal mining machine, cutting visual data, cutting audio data, and cutting load timing data are collected in real time. Here, the cutting visual data, cutting audio data, and cutting load timing data constitute multimodal data due to their different physical properties. The cutting load timing data, also known as process signal data, can include timing data of signals such as current, torque, and vibration.

[0033] Feature extraction for each modality is performed on the truncated visual data, truncated audio data, and truncated load time-series data to obtain corresponding image features, audio features, and load time-series features. Then, the image features, audio features, and load time-series features are fused to generate multimodal fused features.

[0034] Here, fusion can include dimensionality reduction and concatenation. Because the modal data after feature extraction are inconsistent in terms of data dimensions, it is necessary to reduce and process the multidimensional data to transform it to the same dimension; then, the image features, audio features, and load time-series features of the same dimension are concatenated to obtain multimodal fused features.

[0035] For example, the above-mentioned fusion method can be early feature fusion, also known as pre-fusion. Feature-level fusion is a shallow fusion after feature extraction, which directly connects the features of multiple modalities, i.e., shallow concatenation, addition, and weighted summation. Feature fusion aims to integrate multiple features from different modalities into a common space. Due to the differences between the various modalities, it often contains a large amount of redundant information. Dimensionality reduction methods are used to eliminate redundant information, typically employing methods such as Principal Component Analysis (PCA).

[0036] After the feature extraction described above, we can obtain a one-dimensional matrix of image features, a two-dimensional matrix of audio features, and a one-dimensional matrix of load temporal features. It is necessary to unify the dimensions of the feature matrices for each modality. To minimize modifications, we consider reducing the two-dimensional audio feature matrix to a one-dimensional matrix, which facilitates fusion with image features and load temporal features. A one-dimensional matrix can represent the maximum amount of information from each modality with the smallest dimension, reducing the computational cost of the model while allowing the modality to carry the maximum amount of information into the model.

[0037] Principal Component Analysis (PCA) is a commonly used linear dimensionality reduction method. It reduces dimensionality by identifying the most important principal components in the data. PCA works by finding the directions with the largest variance (i.e., principal components), projecting the data onto these principal components, and then selecting the most important components to represent the data. As a general dimensionality reduction method, it is applicable to most datasets, regardless of whether the relationship is linear or non-linear. Its computational efficiency is usually very high, especially for large-scale datasets, where it can significantly reduce the dimensionality of the data while retaining most of the original information.

[0038] Then, the multimodal fusion features are input into a pre-built cutting state recognition model to predict the recognition result of the cutting state of the drum cutting section, thereby realizing real-time prediction of the cutting state of the coal mining machine's drum cutting section. Here, the cutting state of the drum cutting section refers to whether the drum cutting section is in a coal cutting state or a rock cutting state during the coal cutting process, and the drum cutting state recognition corresponds to the judgment of whether the drum is cutting coal or rock.

[0039] In the process of constructing the cutting state recognition model, multimodal fusion features can be input into the deep learning model for learning. By learning the features corresponding to coal cutting and rock cutting, the deep learning model is converged to obtain the cutting state recognition model.

[0040] This disclosure embodiment can also, based on the identification results of the cutting status, realize the real-time adjustment of the mining height of the drum cutting section (which can be understood as the height of the drum cutting section) through the programmable logic controller (PLC) industrial control system, reduce the involvement of human experience, thereby removing personnel from the production environment, achieving the purpose of remote coal cutting and reducing manpower, while improving the overall recovery rate of the coal mining face and increasing resource utilization.

[0041] According to the technical solution of the exemplary embodiments of this disclosure, multimodal data of the drum cutting section of the coal mining machine is collected during the cutting process; wherein, the multimodal data includes at least cutting visual data, cutting audio data, and cutting load time-series data; feature extraction is performed on the cutting visual data, cutting audio data, and cutting load time-series data respectively to obtain image features, audio features, and load time-series features of the drum cutting section; the image features, audio features, and load time-series features are fused to generate multimodal fused features; and the multimodal fused features are input to... A pre-built cutting state recognition model outputs the recognition results of the cutting state of the drum cutting section; wherein, the cutting state includes at least coal cutting state and rock cutting state; based on the recognition results of the cutting state, the mining height of the drum cutting section is adjusted. The cutting state recognition model can be used to identify the cutting state of the drum cutting section in real time based on the multimodal data of the drum cutting section, thereby realizing remote control and adaptive adjustment of the height of the drum cutting section, achieving the goal of improving the cutting efficiency of the coal mining machine and reducing manpower, while also improving the overall recovery rate of the coal mining face and increasing resource utilization.

[0042] The aforementioned deep learning model can be a Transformer network based on an attention mechanism. It is a deep learning model used to process sequential data. It uses a self-attention mechanism to capture the dependencies between different positions in the input sequence, thereby better understanding sequential data such as text, speech, or video.

[0043] Figure 4 A schematic diagram of the framework of a Transformer network provided in an exemplary embodiment of this disclosure is shown. Figure 4 As shown, the Transformer network mainly consists of two parts: Encoder and Decoder. The Encoder converts the input sequence into a hidden representation, while the Decoder generates the corresponding output sequence based on the encoder's output. Both the Encoder and Decoder contain six blocks. Each encoder contains a feedforward neural network and a self-attention mechanism. Each decoder is structurally similar to the encoder, but differs in that after the self-attention mechanism is applied, the output of the self-attention is compared with the output of the Decoder module to calculate the attention score before entering the feedforward neural network module.

[0044] The Transformer workflow is roughly as follows: 1. Convert the structured data of the feature matrix of multimodal fusion features into text form: Structured data usually needs to be converted into a text input form that Transformer can understand, forming a corresponding text sequence as the input of the model.

[0045] 2. Obtain the representation vector X of each input word. X is obtained by adding the word's embedding (the embedding is the feature extracted from the original data) and the word's position embedding.

[0046] 3. Pass the obtained word representation vector matrix into the Encoder. After passing through 6 Encoder blocks, you can obtain the encoding information matrix C of all words in the sentence.

[0047] 4. Pass the encoded information matrix C output by the Encoder to the Decoder, and the Decoder decodes the semantic information, contextual relationships, etc. in the structured data in sequence.

[0048] 5. Combining the specific encoded information matrix C and the given data labels, we learn the mapping relationship between the feature matrix and the labels, which serves as the standard for the final model's inference judgment. During the learning process, we divide the data into training, validation, and test sets, using the cross-entropy loss function as the loss function for model training and the F1 score as the evaluation metric for the model's performance on the validation set. After training for each batch size, we compare the relationship between the loss function and the F1 score to determine the direction of parameter optimization and model iteration.

[0049] Based on the representation of the model output, the model output is converted into a understandable structured data result format. Depending on the model task, the output can be the classification, generation, prediction, or other results of structured data. In this specific task, which is the drum cutting state recognition scenario, the final output will pass through the final Softmax layer to output the predicted probability value of the corresponding cutting state, which will be used as the final output of the model.

[0050] Figure 5 A schematic diagram illustrating the principle of a Transformer network provided in an exemplary embodiment of this disclosure is shown. Figure 5 As shown, Add is a residual connection, typically used to solve the problem of training multi-layer networks, allowing the network to focus only on the currently differing parts. Norm refers to Layer Normalization, commonly used in RNN structures. It transforms the input of each neuron in each layer to have the same mean and variance, which can speed up convergence. The Feed Forward layer is relatively simple, a two-layer fully connected layer. The first layer uses ReLU as its activation function, while the second layer does not use an activation function.

[0051] Multi-Head Attention is a mechanism formed by combining multiple Self-Attention mechanisms. In practice, Self-Attention receives either the input (a matrix X composed of word representation vectors x) or the output of the previous encoder block. After calculation using a linear transformation formula, an attention matrix is ​​obtained for each word with respect to other words, where each value represents the attention strength between pairs of words. Multi-Head Attention extends the model's ability to focus on different positions, capturing the correlation coefficients (attention scores) between words across multiple dimensions. Mask Multi-Head Attention works similarly to Multi-Head Attention, but adds a mask operation. It can be observed that word 0 can only use information from word 0, while word 1 can use information from words 0 and 1; that is, it can only use information from previous words and cannot use information from later words.

[0052] The training steps for the truncated state recognition model are as follows: First, collect sample data from each modality and extract features; second, associate and label the multimodal data on timestamps to construct a sample dataset; third, divide the sample dataset into training, validation, and test sets according to time sequence; fourth, train the model based on the training and validation sets until it converges to within the error range; fifth, test the trained truncated state recognition model using the test set. If the accuracy requirement is met, the model training is complete; otherwise, add the test set to the sample set, update the test set data to ensure that the test set data has not appeared in the training and validation sets, and return to step one to repeat the above training process until the model converges.

[0053] In some embodiments, such as Figure 1 and Figure 2 As shown, the cutting visual data is collected by a camera 214 arranged on the hydraulic support 103; the hydraulic support 103 is located in the fully mechanized mining face where the coal mining machine is located, and the camera 214 is pointed towards the drum cutting part of the coal mining machine to collect real-time visual data of the drum cutting part during the coal cutting process. The audio data of the cutting process is collected by a microphone built into the electrical control box 213 of the coal mining machine; The cutting load timing data includes the cutting motor current signal, the cutting motor torque signal, and the rocker arm vibration signal. The cutting motor current signal is collected by a sensor installed on the drum cutting section, the cutting motor torque signal is collected by a sensor installed on the traction section of the coal mining machine, and the rocker arm vibration signal is collected by a sensor installed on the rocker arm between the coal mining machine body and the drum cutting section.

[0054] Figure 6 A schematic diagram of the data acquisition and transmission architecture provided by an exemplary embodiment of this disclosure is shown. For example... Figure 6 As shown, the cutting visual data (also known as video image data) is collected by a camera mounted on the hydraulic support, while the cutting audio data is collected by a microphone device installed in the electrical control box. The video and audio streams pass through a 10 Gigabit industrial ring network to the network hard drive camera. The streaming media server then retrieves the audio and video stream data from the network hard drive recorder and sends it to the algorithm server. The cutting motor current signal, cutting motor torque signal, and rocker arm vibration signal, corresponding to the cutting load timing data, are received by the electrical control box, converted via an interface, and then uploaded to the 10 Gigabit industrial ring network. The data acquisition server collects current, torque, and vibration data from the 10 Gigabit industrial ring network and sends them to the algorithm server. The algorithm server utilizes the uploaded cutting visual data, cutting audio data, and cutting load timing data, among other multi-dimensional data, to perform real-time multimodal pre-fusion inference, enabling real-time identification and judgment of the coal cutting and rock cutting status of the coal mining machine's cutting drum.

[0055] In some embodiments, feature extraction is performed on the truncated visual data, truncated audio data, and truncated load time-series data respectively to obtain corresponding image features, audio features, and load time-series features, including: Obtain a pre-constructed multimodal feature extraction model; wherein, the multimodal feature extraction model includes an image feature extractor, an audio feature extractor, and a temporal feature extractor; Image features of the drum cutting section are obtained by using an image feature extractor to extract image features from the visual data of the cutting section. Audio features of the cut audio data are extracted using an audio feature extractor to obtain the audio features of the drum cut section. The timing features of the cutting load time series data are extracted using a timing feature extractor to obtain the timing features of the load of the drum cutting section.

[0056] Specifically, embodiments of this disclosure may pre-construct a multimodal feature extraction model, which may include an image feature extractor, an audio feature extractor, and a temporal feature extractor.

[0057] Then, an image feature extractor is used to extract image features from the cutting visual data to obtain the image features of the drum cutting section; an audio feature extractor is used to extract audio features from the cutting audio data to obtain the audio features of the drum cutting section; and a time-series feature extractor is used to extract time-series features from the cutting load time-series data to obtain the load time-series features of the drum cutting section.

[0058] For example, the image feature extractor is a convolutional neural network-based model, and the image features are depth features related to coal and rock texture and contour extracted from truncated visual data.

[0059] Specifically, a convolutional neural network is a deep neural network specifically designed for image and multidimensional data processing, and is widely used in fields such as image recognition, speech recognition, and natural language processing.

[0060] Convolutional neural networks (CNNs) consist of multiple convolutional layers, pooling layers, and fully connected layers. These different network structures play different roles in network learning, achieving different functions. The input to convolutional layers is often 3D image data. The number and size of the convolutional kernels determine the size and depth of the output feature map of that layer. Each convolutional kernel performs a convolution operation on the input data, generating a feature map, which is equivalent to feature extraction in the processing, enabling the learning of features such as image edges and contours. A non-linear function, such as sigmoid, tanh, or ReLU, is usually added after the convolutional layer. Through non-linear transformation, the non-linear relationships between data are learned, thereby improving the model's performance.

[0061] Pooling layers are used to reduce the size of the feature map. By using a sliding window selection method, they reduce the number of model parameters, thereby reducing the computational cost and complexity of the model, while improving its robustness. Common pooling methods include average pooling and max pooling, which take the average and maximum values ​​from the feature map as outputs, respectively. The size of the sliding window and the pooling method ultimately determine the number of parameters. The final fully connected layer is used to convert the feature map into probability or regression outputs using methods such as Softmax. The number of output neurons determines the output dimension of the network. Typically, assuming the input image size is (Height, Width, Channels), where Height is the image height, Width is the image width, and Channels is the number of image channels, after convolution and pooling operations, the feature map size is (Height_out, Width_out, Channels_out). The feature map size may vary, depending on the size of the convolution kernel, stride, padding method, and the type and parameter settings of the pooling operation. After passing through the fully connected layer, the feature map is converted into one-dimensional data for output, serving as the final feature matrix, thus obtaining the image features.

[0062] For example, an audio feature extractor is used to extract audio features from the cut audio data to obtain the audio features of the drum cut section, including: The truncated audio data is preprocessed to obtain preprocessed audio data; the preprocessing includes pre-emphasis operation, frame segmentation operation and windowing operation. Mel frequency cepstral coefficients are extracted from the preprocessed audio data to obtain the audio characteristics of the drum cutting section.

[0063] Specifically, audio feature extractors, also known as sound classification models, first need to convert the raw sound signal into a suitable signal for feature extraction and modeling. The preprocessing process mainly includes pre-emphasis, framing, windowing, data annotation, and label verification.

[0064] Pre-emphasis: Audio signals have high energy in the high-frequency band and low energy in the low-frequency band. However, the power spectral density of the discriminator output noise increases with the square of the frequency (low-frequency noise is low, high-frequency noise is high), resulting in a high low-frequency signal-to-noise ratio (SNR) and a significantly insufficient high-frequency SNR. Therefore, the high-frequency components of the signal are emphasized before transmission to increase the resolution of the high-frequency audio band and make the signal spectrum smoother, facilitating subsequent feature extraction and analysis. Pre-emphasis is typically achieved using a first-order FIR digital high-pass filter.

[0065] Framing: In audio signal analysis, a key technique is called "short-time analysis." Audio signals are often non-stationary, especially abnormal sound signals, whose waveforms and inherent characteristics change significantly over time, making direct analysis of the entire audio segment difficult. However, audio signals maintain a relatively stable state within a very short period. To effectively analyze audio signals, framing is necessary, dividing the audio signal along the time axis into segments of 10-30ms each. This allows each segment to be considered a stationary signal. Considering the correlation between audio frames, adjacent frames overlap to achieve a smooth transition between frames, commonly known as frame shift. The framing process of an audio signal is as follows: Figure 7 As shown. Figure 7 A schematic diagram illustrating the framing process of an audio signal provided in an exemplary embodiment of this disclosure is shown.

[0066] Windowing: Because the audio signal is truncated in the time domain during framing, the edges of each frame's audio signal exhibit abrupt transitions from zero to signal or vice versa, resulting in significant high-frequency noise at the frame edges. Windowing maintains the continuity between the beginning and end of adjacent frames, making the overall signal more continuous, avoiding the Gibbs effect, and reducing signal leakage caused by truncation to some extent. Different window functions result in different time-domain shapes and frequency-domain characteristics in the processed signal. Commonly used window functions include rectangular windows, Hanning windows, and Hamming windows.

[0067] Among these, rectangular windows are suitable for simple spectrum analysis, mainly used to display the results of short-time Fourier transform (STFT) or Fourier transform. They are generally not recommended for spectrum analysis due to their significant truncation effect (spectral leakage). Hanning windows are suitable for spectrum analysis, signal filtering, and other applications requiring good spectral resolution, and are commonly used in spectrum estimation and sound processing. Hamming windows are suitable for spectrum analysis and filtering, and can be used for applications such as spectrum estimation of narrowband signals; however, attention should be paid to their impact on the main lobe and side lobes of the spectrum. In this embodiment, a Hanning window is used for windowing processing.

[0068] Data annotation and label verification: Label annotation involves assigning labels to the data based on the actual working conditions, categorizing each audio segment into a corresponding working condition. The model then learns the data characteristics of each working condition to form its own representation of data under different conditions. After annotation, manual verification of the data labels is required to ensure accuracy before the sample data can be fed into the model for training and learning.

[0069] Mel-frequency cepstrum coefficients (MFCCs) are sound characteristics established based on the characteristics of human auditory perception. The human ear's response to the sound spectrum on a frequency scale is non-linear. Converting a standard frequency scale to a Mel-frequency scale transforms the human ear's response to the spectrum into a linear relationship. The relationship between the actual sound frequency (f) perceived by the human auditory system and the Mel frequency (Mel frequency, Mel(f)) is as follows:

[0070] Figure 8 A schematic diagram illustrating the framing process of an audio signal provided in an exemplary embodiment of this disclosure is shown. Figure 8 As shown, when the sound frequency is low, the Mel frequency and the ordinary frequency have an approximately linear relationship; as the sound frequency increases, the Mel frequency and the ordinary frequency have an approximately logarithmic relationship.

[0071] The process of converting an audio file into a Mel matrix is ​​as follows: 1. First, the speech is pre-emphasized, framed, and windowed; 2. For each short-time analysis window, the corresponding spectrum is obtained through FFT (Fast Fourier Transform); 3. The above spectrum is passed through the Mel filter bank to obtain the Mel spectrum; 4. Cepstral analysis is performed on the Mel spectrum (logarithm, inverse transform, the actual inverse transform is generally implemented by DCT discrete cosine transform, and the 2nd to 13th coefficients after DCT are taken as MFCC coefficients), to obtain Mel frequency cepstral coefficients (MFCC). This MFCC is the feature of this frame of audio, thus obtaining the two-dimensional Mel matrix, i.e., the audio feature.

[0072] For example, a time-series feature extractor is used to extract time-series features from the cutting load time-series data to obtain the load time-series features of the drum cutting section, including: Using a time-series feature extractor, time-domain statistical analysis is performed on truncated load time-series data within a preset sliding time window to extract multiple statistical indicator features; among which, multiple statistical indicator features include, but are not limited to, at least two of the following: maximum value, minimum value, mean, variance, standard deviation, median, maximum absolute value, and minimum absolute value. By splicing together multiple statistical indicators, the load time sequence characteristics of the drum cutting section are obtained.

[0073] Specifically, commonly used time-series signal feature extraction methods include statistical feature extraction, model feature extraction, and time-frequency domain analysis. Statistical feature extraction refers to extracting statistical indicators such as the maximum, minimum, mean, variance, standard deviation, median, maximum absolute value, and minimum absolute value of time-series data to reflect its characteristics. Model feature extraction refers to using models to characterize time-series data and extracting model coefficients as feature vectors. Commonly used models include AR (Autoregressive Model), MA (Moving Average Model), ARMA (Autoregressive Moving Average Model), and ARIMA (Autoregressive Differential Moving Average Model). Time-frequency analysis methods transform time-series data from the time domain to the frequency domain, with common transformation methods including Fourier transform and cepstral transform.

[0074] This embodiment employs the tsfresh statistical feature extraction method for time-series data feature extraction. A time-series feature extractor extracts statistical indicator features from the time-series data within a preset sliding time window to reflect the data's changing characteristics over that short period. Each dimension of the signal can generate multiple statistical indicators, which are then concatenated to form a new one-dimensional feature array. Here, the width parameter of the preset sliding time window is specifically obtained through model parameter tuning of the time-series feature extractor.

[0075] The foregoing mainly describes the solutions provided by the embodiments of this disclosure. It is understood that, in order to achieve the above functions, the electronic device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0076] Based on this, the embodiments of this disclosure introduce multi-dimensional data such as images, audio, current, torque, and vibration, and establish a multi-modal feature extraction model (including an image feature extractor, an audio feature extractor, and a temporal feature extractor) and a cutting state recognition model. This solves the shortcomings of current single-model performance, improves the overall model performance, and ensures the accuracy of the cutting state of the coal mining machine drum cutting section. This method addresses the deficiencies of inaccurate single-dimensional judgment results, single-dimensionality, and susceptibility of vision to environmental conditions. It ensures analysis from multiple angles and perspectives, including visual, auditory, and actual production processes, guaranteeing comprehensive and multi-dimensional model analysis and thus ensuring optimal recognition results. Real-time recognition of the coal mining machine cutting state is achieved, and the PLC industrial control system enables real-time adjustment of the coal mining machine drum height, reducing the involvement of manual experience. This allows personnel to be removed from the production environment, achieving remote coal cutting and reducing manpower. Simultaneously, it improves the overall recovery rate of the coal mining face and increases resource utilization.

[0077] This disclosure embodiment can divide the electronic device into functional units according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this disclosure embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0078] In the case of dividing each function into functional modules, an exemplary embodiment of this disclosure provides a coal mining machine cutting status identification device based on multimodal data perception. This coal mining machine cutting status identification device based on multimodal data perception can be a terminal or a chip applied to the terminal. Figure 9 A schematic diagram of the structure of a coal mining machine cutting status identification device based on multimodal data perception provided in an exemplary embodiment of this disclosure is shown. Figure 9 As shown, the device 900 includes: The acquisition module 901 is used to acquire multimodal data of the drum cutting section during the cutting process of the coal mining machine; wherein the multimodal data includes at least cutting visual data, cutting audio data and cutting load time series data; Processing module 902 is used to extract features from the cutting visual data, the cutting audio data, and the cutting load time series data respectively to obtain the image features, audio features, and load time series features of the drum cutting section; The processing module 902 is further configured to fuse the image features, the audio features, and the load time-series features to generate multimodal fusion features; The processing module 902 is further configured to input the multimodal fusion features into a pre-constructed cutting state recognition model and output the recognition result of the cutting state of the drum cutting section; wherein, the cutting state includes at least coal cutting state and rock cutting state; The adjustment module 903 is used to adjust the sampling height of the drum cutting section based on the identification result of the cutting state.

[0079] In some embodiments, the processing module 902 is further configured to obtain a pre-constructed multimodal feature extraction model; wherein the multimodal feature extraction model includes an image feature extractor, an audio feature extractor, and a temporal feature extractor; The image feature extractor is used to extract image features from the cutting visual data to obtain the image features of the drum cutting section; The audio feature extractor is used to extract audio features from the cut audio data to obtain the audio features of the drum cut section; The timing feature extractor is used to extract timing features from the cutting load timing data to obtain the load timing features of the drum cutting section.

[0080] In some embodiments, the image feature extractor is a convolutional neural network-based model, and the image features are depth features related to coal and rock texture and contour extracted from the truncated visual data.

[0081] In some embodiments, the processing module 902 is further configured to preprocess the truncated audio data to obtain preprocessed audio data; wherein the preprocessing includes pre-emphasis operation, frame segmentation operation, and windowing operation; Mel frequency cepstral coefficients are extracted from the preprocessed audio data to obtain the audio characteristics of the drum cutting section.

[0082] In some embodiments, the processing module 902 is further configured to use the time-series feature extractor to perform time-domain statistical analysis on the truncated load time-series data within a preset sliding time window, and extract multiple statistical indicator features; wherein, the multiple statistical indicator features include, but are not limited to, at least two of the following: maximum value, minimum value, mean, variance, standard deviation, median, maximum absolute value, and minimum absolute value; By concatenating the multiple statistical indicators, the load timing characteristics of the drum cutting section are obtained.

[0083] In some embodiments, the cutting visual data is collected by a camera arranged on a hydraulic support; the hydraulic support is located in the fully mechanized mining face where the coal mining machine is located, and the camera is pointed towards the drum cutting part of the coal mining machine to collect real-time visual data of the drum cutting part during the coal cutting process; The cutting audio data is collected by a microphone built into the electrical control box of the coal mining machine; The cutting load timing data includes cutting motor current signal, cutting motor torque signal and rocker arm vibration signal. The cutting motor current signal is collected by a sensor installed on the drum cutting section, the cutting motor torque signal is collected by a sensor installed on the traction section of the coal mining machine, and the rocker arm vibration signal is collected by a sensor installed on the rocker arm between the coal mining machine body and the drum cutting section.

[0084] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the methods disclosed in this disclosure.

[0085] Figure 10 A schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this disclosure is shown. For example... Figure 10 As shown, the electronic device 1000 includes at least one processor 1001 and a memory 1002 coupled to the processor 1001. The processor 1001 can perform the corresponding steps in the methods disclosed in the embodiments of this disclosure.

[0086] The processor 1001 described above can also be referred to as a Central Processing Unit (CPU), which can be an integrated circuit chip with signal processing capabilities. Each step in the method disclosed in this embodiment can be implemented by the integrated logic circuitry in the processor 1001 or by software instructions. The processor 1001 can be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this embodiment can be directly implemented by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules can be located in the memory 1002, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor 1001 reads information from the memory 1002 and, in conjunction with its hardware, completes the steps of the method described above.

[0087] Furthermore, various operations / processes according to this disclosure, implemented via software and / or firmware, can be transmitted from a storage medium or network to a computer system with a dedicated hardware architecture, for example, Figure 11 The computer system 1100 shown is equipped with the programs that constitute the software. When various programs are installed, the computer system is able to perform various functions, including functions such as those described above. Figure 11 A schematic diagram of the structure of a computer system provided in an exemplary embodiment of this disclosure is shown.

[0088] Computer system 1100 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0089] like Figure 11 As shown, the computer system 1100 includes a computing unit 1101, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded into random access memory (RAM) 1103 from a storage unit 1108. The RAM 1103 may also store various programs and data required for the operation of the computer system 1100. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0090] Multiple components in computer system 1100 are connected to I / O interface 1105, including: input unit 1106, output unit 1107, storage unit 1108, and communication unit 1109. Input unit 1106 can be any type of device capable of inputting information into computer system 1100. Input unit 1106 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 1107 can be any type of device capable of presenting information and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 1108 may include, but is not limited to, hard disks and optical disks. Communication unit 1109 allows computer system 1100 to exchange information / data with other devices via a network such as the Internet, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0091] The computing unit 1101 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above. For example, in some embodiments, the methods disclosed in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on an electronic device via ROM 1102 and / or communication unit 1109. In some embodiments, the computing unit 1101 can be configured to perform the methods disclosed in this disclosure by any other suitable means (e.g., by means of firmware).

[0092] This disclosure also provides a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the methods disclosed in this disclosure.

[0093] The computer-readable storage medium in this disclosure can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. The aforementioned computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specifically, the aforementioned computer-readable storage medium may include electrical connections based on one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0094] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0095] This disclosure also provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, it implements the methods disclosed in the embodiments of this disclosure.

[0096] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer.

[0097] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0098] The modules, components, or units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules, components, or units do not necessarily constitute a limitation on the module, component, or unit itself.

[0099] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0100] The above description is merely an illustration of some embodiments of this disclosure and the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0101] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A method for identifying the cutting state of a coal mining machine based on multimodal data perception, characterized in that, include: During the cutting process of the drum cutting section of the coal mining machine, multimodal data of the drum cutting section is collected; wherein, the multimodal data includes at least cutting visual data, cutting audio data, and cutting load time series data; Feature extraction is performed on the cutting visual data, the cutting audio data, and the cutting load time-series data respectively to obtain the image features, audio features, and load time-series features of the drum cutting section; The image features, audio features, and load temporal features are fused to generate multimodal fusion features; The multimodal fusion features are input into a pre-constructed cutting state recognition model, and the recognition result of the cutting state of the drum cutting section is output; wherein, the cutting state includes at least coal cutting state and rock cutting state; Based on the identification result of the cutting state, the sampling height of the drum cutting section is adjusted.

2. The method as described in claim 1, characterized in that, The step of extracting features from the truncated visual data, the truncated audio data, and the truncated load time-series data to obtain corresponding image features, audio features, and load time-series features includes: Obtain a pre-constructed multimodal feature extraction model; wherein the multimodal feature extraction model includes an image feature extractor, an audio feature extractor, and a temporal feature extractor; The image feature extractor is used to extract image features from the cutting visual data to obtain the image features of the drum cutting section; The audio feature extractor is used to extract audio features from the cut audio data to obtain the audio features of the drum cut section; The timing feature extractor is used to extract timing features from the cutting load timing data to obtain the load timing features of the drum cutting section.

3. The method as described in claim 2, characterized in that, The image feature extractor is a model based on a convolutional neural network, and the image features are depth features related to coal and rock texture and contour extracted from the truncated visual data.

4. The method as described in claim 2, characterized in that, The step of using the audio feature extractor to extract audio features from the cut audio data to obtain the audio features of the drum cut section includes: The truncated audio data is preprocessed to obtain preprocessed audio data; wherein, the preprocessing includes pre-emphasis operation, frame segmentation operation and windowing operation; Mel frequency cepstral coefficients are extracted from the preprocessed audio data to obtain the audio characteristics of the drum cutting section.

5. The method as described in claim 2, characterized in that, The step of using the time-series feature extractor to extract time-series features from the cutting load time-series data to obtain the load time-series features of the drum cutting section includes: Using the time-series feature extractor, time-domain statistical analysis is performed on the truncated load time-series data within a preset sliding time window to extract multiple statistical indicator features; wherein, the multiple statistical indicator features include, but are not limited to, at least two of the following: maximum value, minimum value, mean, variance, standard deviation, median, maximum absolute value, and minimum absolute value; By concatenating the multiple statistical indicators, the load timing characteristics of the drum cutting section are obtained.

6. The method according to any one of claims 1 to 5, characterized in that, The cutting visual data is collected by a camera arranged on the hydraulic support; the hydraulic support is located in the fully mechanized mining face where the coal mining machine is located, and the camera is pointed towards the drum cutting part of the coal mining machine to collect real-time visual data of the drum cutting part during the coal cutting process. The cutting audio data is collected by a microphone built into the electrical control box of the coal mining machine; The cutting load timing data includes cutting motor current signal, cutting motor torque signal and rocker arm vibration signal. The cutting motor current signal is collected by a sensor installed on the drum cutting section, the cutting motor torque signal is collected by a sensor installed on the traction section of the coal mining machine, and the rocker arm vibration signal is collected by a sensor installed on the rocker arm between the coal mining machine body and the drum cutting section.

7. A coal mining machine cutting status identification device based on multimodal data perception, characterized in that, include: The acquisition module is used to acquire multimodal data of the drum cutting section during the cutting process of the coal mining machine; wherein the multimodal data includes at least cutting visual data, cutting audio data, and cutting load time-series data. The processing module is used to extract features from the cutting visual data, the cutting audio data, and the cutting load time-series data respectively, to obtain the image features, audio features, and load time-series features of the drum cutting section; The processing module is also used to fuse the image features, the audio features, and the load time-series features to generate multimodal fusion features; The processing module is also used to input the multimodal fusion features into a pre-constructed cutting state recognition model and output the recognition result of the cutting state of the drum cutting section; wherein, the cutting state includes at least coal cutting state and rock cutting state; An adjustment module is used to adjust the sampling height of the drum cutting section based on the identification result of the cutting state.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method described in any one of claims 1 to 6.

Images