Coal gangue identification method and system based on adaptive multimodal feature fusion

Abstract

An embodiment of the present invention provides a method and system for identifying coal gangue based on adaptive multimodal feature fusion, the method comprising acquiring image data and audio data related to coal gangue production; inputting the image data and audio data into a trained coal gangue identification model based on adaptive multimodal feature fusion, and obtaining a coal gangue identification result output by the coal gangue identification model based on adaptive multimodal feature fusion, wherein the coal gangue identification model based on adaptive multimodal feature fusion is used to adaptively adjust the weights of different modal features based at least on a channel attention mechanism to identify coal gangue, so as to obtain a coal gangue identification result.

Description

Technical Field

[0001] The present invention relates to the field of minerals, and in particular to a method and system for identifying coal gangue based on adaptive multi-modal feature fusion. Background Art

[0002] During the mining process, the effective sorting of coal and gangue is directly related to the utilization rate of coal resources and the subsequent coal washing quality control. In traditional coal mining production, it usually relies on manual sorting or simple physical and mechanical separation methods.

[0003] However, these methods have the following problems: (1) Low efficiency: manual sorting is slow and prone to fatigue, resulting in low overall production efficiency; (2) Poor accuracy: manual sorting is greatly affected by subjective factors and is prone to misjudgment, affecting the purity of coal and the quality of subsequent coal washing; (3) Environmental risks: coal mines have harsh environments and are subject to safety hazards such as hazardous gases and dust, resulting in high operational risks.

[0004] Although deep learning and visual technology have been applied to coal mine production to a certain extent, their application in complex mining environments still has limitations. For example, environmental factors, dust, insufficient light or visual occlusion in mines can affect the accuracy of image recognition. Summary of the invention

[0005] The present invention provides a method and system for identifying coal gangue based on adaptive multimodal feature fusion, aiming to solve at least the following technical problems: (1) Low efficiency: manual sorting is slow and prone to fatigue, resulting in low overall production efficiency; (2) Poor accuracy: manual sorting is greatly affected by subjective factors and is prone to misjudgment, affecting the purity of coal and the quality of subsequent coal washing; (3) Environmental risks: coal mines have a harsh environment, with potential safety hazards such as dangerous gases and dust, and high operational risks; (4) Visual technology has limitations in coal mine production.

[0006] Specifically, the embodiment of the present invention provides the following technical solutions: In a first aspect, an embodiment of the present invention provides a method for identifying coal gangue based on adaptive multimodal feature fusion, comprising: Acquire image data and audio data related to coal gangue production; The image data and the audio data are input into a trained gangue recognition model based on adaptive multimodal feature fusion to obtain a gangue recognition result output by the gangue recognition model based on adaptive multimodal feature fusion. Among them, the gangue recognition model based on adaptive multimodal feature fusion is used to adaptively adjust the weights of different modal features based at least on a channel attention mechanism to identify the gangue to obtain the gangue recognition result.

[0007] Furthermore, the method further comprises: The gangue recognition model based on adaptive multimodal feature fusion includes at least an audio feature extraction module, an image feature extraction module, and an adaptive multimodal feature fusion module.

[0008] Furthermore, the method further comprises: The image feature extraction module, based on the 2D-CNN structure, is used to extract image features from the image data input into the coal gangue recognition model based on adaptive multimodal feature fusion, and enhance the image features through multi-layer convolution kernels and feature downsampling mechanism to obtain enhanced image features, and the enhanced image features are input into the adaptive multimodal feature fusion module.

[0009] Furthermore, the method further comprises: The adaptive multimodal feature fusion module performs fusion processing on the enhanced image features and audio features based on the channel attention mechanism, and the adaptive multimodal feature fusion module includes an input layer, an adaptive weighting layer, and an output layer. Furthermore, the method further comprises: The audio feature extraction module, based on LSTM and 1D-CNN structures, is used to extract audio features from the audio data input to the coal gangue recognition model based on adaptive multimodal feature fusion, and the audio features are input to the adaptive multimodal feature fusion module.

[0010] Furthermore, the method further comprises: Before acquiring the image data and audio data related to coal gangue production, the method further includes: performing preprocessing on the image data and audio data.

[0011] Furthermore, the method further comprises: The pre-processing comprises: For the audio data, at least one of denoising, normalization, framing, pre-emphasis, endpoint detection, gain control, resampling, echo cancellation, sound source separation, and format conversion is performed.

[0012] Furthermore, the method further comprises: The pre-processing further comprises: For the image data, at least one of grayscale processing, filtering processing, image enhancement processing, edge processing, sharpening processing, segmentation processing, size processing, normalization processing, color space conversion processing, format conversion processing, background removal processing, and noise removal processing is performed.

[0013] Furthermore, the method further comprises: The acquisition further includes acquiring vibration data and spectral data related to coal gangue production, In the case of acquiring vibration data related to gangue production, the gangue identification model based on adaptive multimodal feature fusion further includes a vibration feature extraction module, the vibration feature extraction module is used to extract vibration features from the vibration data input to the gangue identification model based on adaptive multimodal feature fusion, the vibration feature extraction module is input to the adaptive multimodal feature fusion module, and When acquiring spectral data related to coal gangue production, the coal gangue identification model based on adaptive multimodal feature fusion further includes a spectral feature extraction module, which is used to extract spectral features from the spectral data input into the coal gangue identification model based on adaptive multimodal feature fusion, and the spectral feature extraction module is input into the adaptive multimodal feature fusion module.

[0014] In a second aspect, an embodiment of the present invention further provides a coal gangue identification system based on adaptive multimodal feature fusion, comprising: A gangue data acquisition module, used to acquire image data and audio data related to gangue production; A gangue recognition module is used to input the image data and the audio data into a trained gangue recognition model based on adaptive multimodal feature fusion to obtain a gangue recognition result output by the gangue recognition model based on adaptive multimodal feature fusion, wherein the gangue recognition model based on adaptive multimodal feature fusion is used to adaptively adjust the weights of different modal features at least based on a channel attention mechanism to identify the gangue to obtain the gangue recognition result.

[0015] Furthermore, in the third aspect, an embodiment of the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned coal gangue identification method based on adaptive multimodal feature fusion when executing the program.

[0016] Furthermore, in a fourth aspect, an embodiment of the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the above-mentioned coal gangue identification method based on adaptive multimodal feature fusion are implemented.

[0017] It can be seen from the above technical solutions that the embodiments of the present invention provide a coal gangue identification method based on adaptive multimodal feature fusion and a coal gangue identification system based on adaptive multimodal feature fusion, which are intended to adaptively adjust the weights of different modal features based on at least the channel attention mechanism to identify coal gangue, and at least solve (1) low efficiency: manual sorting is slow and prone to fatigue, resulting in low overall production efficiency; (2) poor accuracy: manual sorting is greatly affected by subjective factors and is prone to misjudgment, affecting the purity of coal and the subsequent coal washing quality; (3) environmental risks: the coal mine environment is harsh, there are safety hazards such as dangerous gases and dust, and the operation risk is relatively high; (4) visual technology has limitations in coal mine production. The gangue identification method based on adaptive multimodal feature fusion and the gangue identification system based on adaptive multimodal feature fusion provided by the embodiments of the present invention can at least achieve: (1) real-time monitoring and analysis: by real-time acquisition and analysis of multimodal data, real-time identification and sorting of gangue can be further performed; (2) predictive maintenance is achieved to reduce the risk of gangue production interruption; (3) production efficiency and resource utilization are significantly improved; (4) operational risks in the gangue production process are significantly reduced; and (5) human resource costs are significantly saved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A flow chart of a method for identifying coal gangue based on adaptive multimodal feature fusion provided by an embodiment of the present invention; Figure 2 A schematic diagram of the overall framework of a coal gangue recognition model based on adaptive multimodal feature fusion provided by an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a gangue identification system based on adaptive multimodal feature fusion provided by an embodiment of the present invention; and Figure 4 A schematic diagram of an electronic device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] The various terms or phrases used in the present invention have the general meanings known to those of ordinary skill in the art. Even so, the present invention still hopes to provide a more detailed description and explanation of these terms or phrases. If the terms and phrases involved in this article are inconsistent with the commonly known meanings, the meanings expressed in the present invention shall prevail; and if not defined in this application, they have the meanings commonly understood by those of ordinary skill in the art.

[0022] The existing technology has many problems, including but not limited to the following: (1) Low efficiency: manual sorting is slow and prone to fatigue, resulting in low overall production efficiency; (2) Poor accuracy: manual sorting is greatly affected by subjective factors and is prone to misjudgment, affecting the purity of coal and the quality of subsequent coal washing; (3) Environmental risks: the coal mine environment is harsh, with safety hazards such as dangerous gases and dust, and high operational risks; (4) Visual technology has limitations in coal mine production.

[0023] In view of this, in a first aspect, an embodiment of the present invention proposes a coal gangue identification method based on adaptive multimodal feature fusion.

[0024] Combine the following Figure 1 The invention describes the gangue identification method based on adaptive multimodal feature fusion.

[0025] Figure 1 A flowchart of a method for identifying coal gangue based on adaptive multimodal feature fusion provided by one embodiment of the present invention.

[0026] In this embodiment, it should be noted that the gangue identification method based on adaptive multimodal feature fusion may at least include the following steps: S100: Acquire image data and audio data related to coal gangue production; S200: Input the image data and the audio data into the trained gangue recognition model based on adaptive multimodal feature fusion, and obtain the gangue recognition result output by the gangue recognition model based on adaptive multimodal feature fusion, wherein the gangue recognition model based on adaptive multimodal feature fusion is used to adaptively adjust the weights of different modal features based on at least a channel attention mechanism to identify the gangue, so as to obtain the gangue recognition result.

[0027] In this embodiment, it should be noted that the gangue identification method based on adaptive multimodal feature fusion may also include: before acquiring image data and audio data related to gangue production, further including: performing preprocessing on the image data and audio data.

[0028] In this embodiment, it should be noted that the coal gangue identification method based on adaptive multimodal feature fusion may also include: preprocessing includes: performing at least one of denoising processing, normalization processing, frame processing, pre-emphasis processing, endpoint detection processing, gain control processing, resampling processing, echo cancellation processing, sound source separation processing, and format conversion processing on the audio data.

[0029] Specifically, the embodiments of the present invention can deploy audio collection equipment such as microphones at the coal mine production site to collect audio signals at the site (such as coal gangue falling to the ground, crushing sound, etc.), form a real-time audio data stream, and perform denoising processing on the collected audio signals, such as filtering and normalization operations, to improve the quality of the signal. In addition, the embodiments of the present invention can also divide the processed audio data into several short-time segments of equal length, and each segment is sent to the feature extraction module as independent input data for effective processing by the subsequent neural network.

[0030] More specifically, for the audio data collected in the coal mine production site, the denoising processing in the embodiments of the present invention may include: filtering, that is, using a low-pass, high-pass, band-pass or band-stop filter to remove noise within a specific frequency range; spectral subtraction, that is, reducing noise by estimating the noise spectrum and subtracting it from the power spectrum of the audio data; wavelet denoising, that is, using wavelet transform to identify and remove noise while retaining the important features of the audio data.

[0031] More specifically, for the audio data collected at the coal mine production site, the normalization processing in the embodiments of the present invention may include: amplitude normalization, that is, scaling the amplitude of the audio data to a specific range, such as [-1, 1] or [0, 1], to prevent signal oversaturation; energy normalization, that is, ensuring that all audio data segments have the same energy level.

[0032] More specifically, for audio data collected at a coal mine production site, the frame processing in an embodiment of the present invention may include: dividing the audio data into shorter frames, usually using a sliding window method, so that each frame is processed independently.

[0033] More specifically, for audio data collected at a coal mine production site, the pre-emphasis processing in the embodiment of the present invention may include: enhancing high-frequency components through a high-pass filter to compensate for the attenuation of high-frequency components during data transmission or processing.

[0034] More specifically, for audio data collected in a coal mine production site, the endpoint detection process in an embodiment of the present invention may include: identifying silent segments and valid speech segments in the audio data so as to process only useful audio portions.

[0035] More specifically, for audio data collected in a coal mine production site, the gain control processing in the embodiment of the present invention may include: adjusting the overall volume of the signal to ensure a consistent loudness level under different audio data collection conditions.

[0036] More specifically, for the audio data collected in the coal mine production site, the resampling process in the embodiment of the present invention may include: adjusting the sampling rate of the audio data to match the requirements of a specific system or optimizing the processing speed. More specifically, for the audio data collected at the coal mine production site, the echo cancellation processing in the embodiments of the present invention may include: eliminating or reducing the reverberation and echo in the audio data collected at the production site through an algorithm to improve the clarity of the audio data.

[0037] More specifically, for the audio data collected at the coal mine production site, the sound source separation process in the embodiment of the present invention may include: separating separate audio data with a high correlation with coal gangue production from the audio data collected at the production site.

[0038] More specifically, for the audio data collected at the coal mine production site, the format conversion processing in the embodiments of the present invention may include: converting the audio data into a format suitable for processing or storage by the coal gangue recognition model based on adaptive multimodal feature fusion proposed in the present invention.

[0039] In this embodiment, it should be noted that the coal gangue identification method based on adaptive multimodal feature fusion may also include: preprocessing also includes: for the image data, performing at least one of grayscale processing, filtering processing, image enhancement processing, edge processing, sharpening processing, segmentation processing, size processing, normalization processing, color space conversion processing, format conversion processing, background removal processing, and noise removal processing.

[0040] Specifically, embodiments of the present invention can use image acquisition equipment such as industrial cameras at coal mine production sites to obtain real-time image data at the coal mining site, for example, focusing on the appearance characteristics of coal gangue in the coal flow.

[0041] In addition, specifically, graying, filtering, image enhancement and other methods are used to pre-process the raw image data to enhance key image features (such as gangue edges and gangue textures) and reduce redundant information, providing high-quality gangue image data for subsequent feature extraction.

[0042] More specifically, for the image data collected at the coal mine production site, the grayscale processing in the embodiments of the present invention may include: converting the color image collected at the coal mine production site into a grayscale image, reducing the computational complexity, while retaining sufficient information for subsequent processing.

[0043] More specifically, for the image data collected at the coal mine production site, the filtering processing in the embodiments of the present invention may include: using different filters (such as Gaussian filtering, median filtering, bilateral filtering) to remove noise from the gangue image and smooth the image.

[0044] More specifically, for the image data collected at the coal mine production site, the image enhancement processing in the embodiments of the present invention may include: using histogram equalization, contrast enhancement and other techniques to improve the visual quality of the gangue image, making the appearance features in the gangue image more obvious.

[0045] More specifically, for image data collected at a coal mine production site, edge processing in an embodiment of the present invention may include: using operators such as Canny, Sobel, Laplacian, etc. to detect edges in the gangue image and highlight the contour of the gangue image.

[0046] More specifically, for the image data collected in the coal mine production site, the sharpening processing in the embodiment of the present invention may include: using a sharpening filter to enhance the edge and details of the gangue image and improve the clarity of the gangue image.

[0047] More specifically, for the image data collected in the coal mine production site, the size processing and segmentation processing in the embodiment of the present invention may include: segmenting the coal gangue image into multiple regions or objects so as to process each region or object separately.

[0048] More specifically, for the image data collected at the coal mine production site, the normalization processing in the embodiments of the present invention may include: scaling the pixel values ​​of the gangue image to a specific range (such as [0,] or [-1, 1]) to prevent the gradient disappearance or explosion problem caused by excessively large values.

[0049] More specifically, for the image data collected at the coal mine production site, the color space conversion processing in the embodiments of the present invention may include: converting the gangue image from the RGB color space to other color spaces (such as HSV, Lab) to extract different color features.

[0050] More specifically, for the image data collected at the coal mine production site, the format conversion processing in the embodiments of the present invention may include: converting the gangue image into a format suitable for processing or storage by the gangue recognition model based on adaptive multimodal feature fusion proposed in the present invention.

[0051] More specifically, for image data collected at a coal mine production site, background removal processing in an embodiment of the present invention may include: removing the coal mine production site background in the collected gangue image through image processing, and retaining only the foreground object of interest.

[0052] More specifically, for the image data collected in the coal mine production site, the noise removal process in the embodiment of the present invention may include: using an algorithm such as morphological operation to remove noise points in the coal gangue image.

[0053] Combine the following Figure 2 The method of the present invention describes a trained coal gangue recognition model based on adaptive multimodal feature fusion.

[0054] Figure 2 A schematic diagram of the overall framework of a coal gangue recognition model based on adaptive multimodal feature fusion provided in one embodiment of the present invention.

[0055] like Figure 2 As shown, in this embodiment, it should be noted that the coal gangue identification method based on adaptive multimodal feature fusion may also include: the coal gangue identification model based on adaptive multimodal feature fusion includes at least an audio feature extraction module, an image feature extraction module, and an adaptive multimodal feature fusion module.

[0056] Specifically, in this embodiment, it should be noted that the coal gangue identification method based on adaptive multimodal feature fusion may also include: an audio feature extraction module, based on LSTM and 1D-CNN structure, for extracting audio features from audio data input into the coal gangue identification model based on adaptive multimodal feature fusion, and the audio features are input into the adaptive multimodal feature fusion module.

[0057] More specifically, the audio feature extraction module can model the temporal features of audio data through the LSTM structure, capture the long-range dependencies in the audio signal, and thus obtain the unique audio performance of coal and gangue. In addition, the local features of the audio signal are further extracted through 1D-CNN, and the multi-band feature response of the audio signal is obtained through different convolution kernels to further enrich the audio features. Finally, the audio signal is processed by LSTM and 1D-CNN to form a feature vector, which is input to the subsequent feature fusion module.

[0058] Specifically, in this embodiment, it should be noted that the coal gangue identification method based on adaptive multimodal feature fusion may also include: an image feature extraction module, based on a 2D-CNN structure, for extracting image features from image data input into a coal gangue identification model based on adaptive multimodal feature fusion, and enhancing the image features through multi-layer convolution kernels and feature downsampling mechanisms to obtain enhanced image features, and the enhanced image features are input into the adaptive multimodal feature fusion module.

[0059] More specifically, the image feature extraction module can extract edge, texture and spatial features in the image layer by layer through multi-layer convolution and pooling operations. The image feature extraction module can enhance the recognition of image features through multi-layer convolution kernels and feature downsampling mechanisms to obtain image spatial feature representation based on the visual differences between coal and gangue. The image feature vector extracted by the image feature extraction module may include but is not limited to key information on the shape, texture and structure of gangue.

[0060] Specifically, in this embodiment, it should be noted that the coal gangue identification method based on adaptive multimodal feature fusion may also include: an adaptive multimodal feature fusion module, which performs fusion processing on enhanced image features and audio features based on a channel attention mechanism, and the adaptive multimodal feature fusion module includes an input layer, an adaptive weighted layer, and an output layer.

[0061] More specifically, after the audio and image feature extraction is completed, the adaptive multimodal feature fusion module uses the channel attention mechanism to fuse the two modal features. The adaptive multimodal feature fusion module increases the weight of key features by adaptively adjusting the weight of each channel, thereby paying more attention to important features.

[0062] Furthermore, the adaptive multimodal feature fusion module includes: input layer, adaptive weighting layer, and output layer. The input layer receives the concatenated multimodal features, and the input feature shape is W×H×C, where C is the number of input channels and W×H is the spatial dimension of each channel. The adaptive weighting layer mainly consists of three parts: compression, excitation, and scale.

[0063] Specifically, the compression Squeeze in the coal gangue recognition model based on adaptive multimodal feature fusion uses the global average pooling GAP operation to compress the global spatial information of each feature channel, that is, to compress the two-dimensional features (W×H) of each channel, and the compressed features become 1×1×C. The formula of GAP is: in, is the weight parameter after compression operation; It is a feature compression operation; forU Middle c A two-dimensional matrix, U is a collection of local feature maps; H and W are the height and width of the feature matrix respectively.

[0064] Specifically, the excitation excitation in the coal gangue recognition model based on adaptive multimodal feature fusion is achieved by adjusting the parameters w Continuous learning is used to build the correlation between different feature channels and prediction results, thereby generating a weight with a value range of (0, 1) for each feature channel.

[0065] More specifically, the gangue recognition model based on adaptive multimodal feature fusion uses two layers of full connection (FC-ReLU-FC-Sigmoid) to calculate the weight value. The weight calculation formula is: in, Represents the first full connection operation, The dimension is C / r×C, where r is a scaling parameter that reduces the amount of computation by reducing the number of channels. z The dimension of is 1×1×C, then The calculation result is 1×1×C / r, and then the calculation result is input into the ReLU function for back propagation. The results and Multiply to perform the second full connection operation, The dimension of the final output is 1×1×C. The calculated result is finally activated by the Sigmoid function to obtain the normalized weight. s .

[0066] Specifically, the scaling Scale in the gangue recognition model based on adaptive multimodal feature fusion weights the normalized weights obtained above by multiplying them with the features of each channel, inputs the weighted features into the fully connected layer, and finally outputs the gangue recognition results.

[0067] In addition, this embodiment has strong scalability and can add other modal data (such as vibration sensor data, spectral detection data) or optimize the neural network model structure to further improve the recognition performance and meet the needs of different coal mine sites for gangue recognition. This architecture is suitable for intelligent detection or recognition tasks for other mineral targets in other mine production and has broad application prospects.

[0068] Based on this, the present invention proposes a further embodiment. It should be noted that the coal gangue identification method based on adaptive multimodal feature fusion may also include: the acquisition further includes: acquiring vibration data and spectral data related to coal gangue production, In the case of acquiring vibration data related to gangue production, the gangue identification model based on adaptive multimodal feature fusion further includes a vibration feature extraction module, the vibration feature extraction module is used to extract vibration features from the vibration data input to the gangue identification model based on adaptive multimodal feature fusion, the vibration feature extraction module is input to the adaptive multimodal feature fusion module, and When acquiring spectral data related to coal gangue production, the coal gangue identification model based on adaptive multimodal feature fusion further includes a spectral feature extraction module, which is used to extract spectral features from the spectral data input into the coal gangue identification model based on adaptive multimodal feature fusion, and the spectral feature extraction module is input into the adaptive multimodal feature fusion module.

[0069] It can be seen from the above technical solution that the embodiment of the present invention provides a coal gangue identification method based on adaptive multimodal feature fusion, which is at least based on the channel attention mechanism, adaptively adjusts the weights of different modal features to identify coal gangue, and at least solves (1) low efficiency: manual sorting is slow and prone to fatigue, resulting in low overall production efficiency; (2) poor accuracy: manual sorting is greatly affected by subjective factors and is prone to misjudgment, affecting the purity of coal and the subsequent coal washing quality; (3) environmental risks: the coal mine environment is harsh, there are safety hazards such as dangerous gases and dust, and the operation risk is relatively high; (4) visual technology has limitations in coal mine production. The gangue identification method based on adaptive multimodal feature fusion and the gangue identification system based on adaptive multimodal feature fusion provided by the embodiments of the present invention can at least achieve: (1) real-time monitoring and analysis: by real-time acquisition and analysis of multimodal data, real-time identification and sorting of gangue can be further performed; (2) predictive maintenance is achieved to reduce the risk of gangue production interruption; (3) production efficiency and resource utilization are significantly improved; (4) operational risks in the gangue production process are significantly reduced; and (5) human resource costs are significantly saved.

[0070] In addition, the coal gangue recognition model based on adaptive multimodal feature fusion proposed in various embodiments of the present invention has the following advantages: 1. Improving recognition accuracy. By combining LSTM and 1D-CNN to extract time series features in audio signals, the unique audio features generated by coal gangue can be effectively captured, making the recognition model more sensitive to the temporal changes in coal gangue sound. Not only that, 2D-CNN is also used to extract spatial features in image data, which can capture the subtle differences in appearance between coal and gangue, and provide visual feature support. Based on this, the joint use of multimodal data significantly improves the recognition accuracy of coal gangue. rate; 2. Enhance environmental adaptability. In view of the common problems such as insufficient light and dust interference in mines, the fusion of audio data and image data can make up for the defects of a single data source to a certain extent, enhance the robustness in different environments, and even when the image quality is damaged, the audio features can still provide effective recognition basis; 3. Real-time and high efficiency. The present invention adopts LSTM, 1D-CNN and 2D-CNN to extract audio and image features in parallel, and quickly fuses the features through a lightweight channel attention mechanism. The calculation process is efficient and has low latency, which is suitable for real-time detection needs in mine production. The entire system process can complete the identification and sorting control of coal gangue in a short time, realizing automation and intelligence; 4. Reduce the misjudgment rate. The channel attention mechanism adaptively adjusts the weights of different modal features, so that the system can focus on the most recognizable features in coal gangue identification, reduce the interference of environmental noise and non-target objects, reduce the occurrence of misjudgment and missed judgment, and improve the recognition reliability of the system.

[0071] Based on the same inventive concept, on the other hand, an embodiment of the present invention proposes a coal gangue identification system based on adaptive multimodal feature fusion.

[0072] Combine the following Figure 3 The coal gangue identification system based on adaptive multimodal feature fusion provided by the present invention is described. The coal gangue identification system based on adaptive multimodal feature fusion described below and the coal gangue identification method based on adaptive multimodal feature fusion described above can refer to each other.

[0073] Figure 3 A schematic structural diagram of a coal gangue identification system based on adaptive multimodal feature fusion provided by an embodiment of the present invention.

[0074] In this embodiment, it should be noted that the gangue identification system 100 based on adaptive multimodal feature fusion includes: a gangue data acquisition module 110, used to acquire image data and audio data related to gangue production; a gangue identification module 120, used to input the image data and audio data into a trained gangue identification model based on adaptive multimodal feature fusion, and obtain the gangue identification result output by the gangue identification model based on adaptive multimodal feature fusion, wherein the gangue identification model based on adaptive multimodal feature fusion is used to adaptively adjust the weights of different modal features at least based on the channel attention mechanism, to identify gangue, so as to obtain the gangue identification result.

[0075] It can be seen from the above technical solution that the embodiment of the present invention provides a coal gangue identification system based on adaptive multimodal feature fusion, which is based on at least a channel attention mechanism and adaptively adjusts the weights of different modal features to identify coal gangue, thereby solving at least (1) low efficiency: manual sorting is slow and prone to fatigue, resulting in low overall production efficiency; (2) poor accuracy: manual sorting is greatly affected by subjective factors and prone to misjudgment, affecting the purity of coal and the subsequent coal washing quality; (3) environmental risks: the coal mine environment is harsh, with safety hazards such as dangerous gases and dust, and high operational risks; (4) visual technology has limitations in coal mine production. The gangue identification method based on adaptive multimodal feature fusion and the gangue identification system based on adaptive multimodal feature fusion provided by the embodiments of the present invention can at least achieve: (1) real-time monitoring and analysis: by real-time acquisition and analysis of multimodal data, real-time identification and sorting of gangue can be further performed; (2) predictive maintenance is achieved to reduce the risk of gangue production interruption; (3) production efficiency and resource utilization are significantly improved; (4) operational risks in the gangue production process are significantly reduced; and (5) human resource costs are significantly saved.

[0076] In addition, the coal gangue recognition model based on adaptive multimodal feature fusion proposed in various embodiments of the present invention has the following advantages: 1. Improving recognition accuracy. By combining LSTM and 1D-CNN to extract time series features in audio signals, the unique audio features generated by coal gangue can be effectively captured, making the recognition model more sensitive to the temporal changes in coal gangue sound. Not only that, 2D-CNN is also used to extract spatial features in image data, which can capture the subtle differences in appearance between coal and gangue, and provide visual feature support. Based on this, the joint use of multimodal data significantly improves the recognition accuracy of coal gangue. rate; 2. Enhance environmental adaptability. In view of the common problems such as insufficient light and dust interference in mines, the fusion of audio data and image data can make up for the defects of a single data source to a certain extent, enhance the robustness in different environments, and even when the image quality is damaged, the audio features can still provide effective recognition basis; 3. Real-time and high efficiency. The present invention adopts LSTM, 1D-CNN and 2D-CNN to extract audio and image features in parallel, and quickly fuses the features through a lightweight channel attention mechanism. The calculation process is efficient and has low latency, which is suitable for real-time detection needs in mine production. The entire system process can complete the identification and sorting control of coal gangue in a short time, realizing automation and intelligence; 4. Reduce the misjudgment rate. The channel attention mechanism adaptively adjusts the weights of different modal features, so that the system can focus on the most recognizable features in coal gangue identification, reduce the interference of environmental noise and non-target objects, reduce the occurrence of misjudgment and missed judgment, and improve the recognition reliability of the system.

[0077] Since the gangue identification system based on adaptive multimodal feature fusion provided by the embodiment of the present invention can be used to execute the gangue identification method based on adaptive multimodal feature fusion described in the above embodiment, the specific content of its working principle can be found in the introduction of the above method embodiment.

[0078] In this embodiment, it should be noted that the various modules in the coal gangue identification system based on adaptive multimodal feature fusion of the embodiment of the present invention can be integrated into one or deployed separately. The above modules can be combined into one module or further divided into multiple sub-units.

[0079] On the other hand, based on the same inventive concept, another embodiment of the present invention provides an electronic device.

[0080] Figure 4 A schematic diagram of an electronic device provided according to an embodiment of the present invention.

[0081] In this embodiment, it should be noted that the electronic device may include: a processor (processor) 410, a communication interface (Communications Interface) 420, a memory (memory) 430 and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call the logic instructions in the memory 430 to execute the coal gangue recognition method based on adaptive multimodal feature fusion, and the coal gangue recognition method based on adaptive multimodal feature fusion includes: obtaining image data and audio data related to coal gangue production; inputting the image data and audio data into the trained coal gangue recognition model based on adaptive multimodal feature fusion, and obtaining the coal gangue recognition result output by the coal gangue recognition model based on adaptive multimodal feature fusion, wherein the coal gangue recognition model based on adaptive multimodal feature fusion is used to adaptively adjust the weights of different modal features based on at least a channel attention mechanism to identify coal gangue to obtain a coal gangue recognition result.

[0082] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0083] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute a coal gangue identification method based on adaptive multimodal feature fusion, the coal gangue identification method based on adaptive multimodal feature fusion comprising: acquiring image data and audio data related to coal gangue production; inputting the image data and audio data into a trained coal gangue identification model based on adaptive multimodal feature fusion, and obtaining a coal gangue identification result output by the coal gangue identification model based on adaptive multimodal feature fusion, wherein the coal gangue identification model based on adaptive multimodal feature fusion is used to adaptively adjust the weights of different modal features at least based on a channel attention mechanism, to identify coal gangue, and to obtain a coal gangue identification result.

[0084] The system embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0085] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0086] In addition, in the present invention, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0087] In addition, in the present invention, the description with reference to the terms "embodiment", "present embodiment", "another embodiment", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying coal gangue based on adaptive multimodal feature fusion, characterized in that: include: Acquire image data and audio data related to coal gangue production; The image data and the audio data are input into a trained gangue recognition model based on adaptive multimodal feature fusion to obtain a gangue recognition result output by the gangue recognition model based on adaptive multimodal feature fusion. Among them, the gangue recognition model based on adaptive multimodal feature fusion is used to adaptively adjust the weights of different modal features based at least on a channel attention mechanism to identify the gangue to obtain the gangue recognition result.

2. The method for identifying coal gangue based on adaptive multimodal feature fusion according to claim 1, characterized in that: The gangue recognition model based on adaptive multimodal feature fusion includes at least an audio feature extraction module, an image feature extraction module, and an adaptive multimodal feature fusion module.

3. The method for identifying coal gangue based on adaptive multimodal feature fusion according to claim 2, characterized in that: The image feature extraction module, based on the 2D-CNN structure, is used to extract image features from the image data input into the coal gangue recognition model based on adaptive multimodal feature fusion, and enhance the image features through multi-layer convolution kernels and feature downsampling mechanism to obtain enhanced image features, and the enhanced image features are input into the adaptive multimodal feature fusion module.

4. The method for identifying coal gangue based on adaptive multimodal feature fusion according to claim 3 is characterized in that: The adaptive multimodal feature fusion module performs fusion processing on the enhanced image features and audio features based on the channel attention mechanism, and the adaptive multimodal feature fusion module includes an input layer, an adaptive weighting layer, and an output layer.

5. The method for identifying coal gangue based on adaptive multimodal feature fusion according to claim 2, characterized in that: The audio feature extraction module, based on LSTM and 1D-CNN structures, is used to extract audio features from the audio data input to the coal gangue recognition model based on adaptive multimodal feature fusion, and the audio features are input to the adaptive multimodal feature fusion module.

6. The method for identifying coal gangue based on adaptive multimodal feature fusion according to claim 1, characterized in that: Before acquiring the image data and audio data related to coal gangue production, the method further includes: performing preprocessing on the image data and audio data.

7. The method for identifying coal gangue based on adaptive multimodal feature fusion according to claim 6, characterized in that: The pre-processing comprises: For the audio data, at least one of denoising, normalization, framing, pre-emphasis, endpoint detection, gain control, resampling, echo cancellation, sound source separation, and format conversion is performed.

8. The method for identifying coal gangue based on adaptive multimodal feature fusion according to claim 6, characterized in that: The pre-processing further comprises: For the image data, at least one of grayscale processing, filtering processing, image enhancement processing, edge processing, sharpening processing, segmentation processing, size processing, normalization processing, color space conversion processing, format conversion processing, background removal processing, and noise removal processing is performed.

9. The method for identifying coal gangue based on adaptive multimodal feature fusion according to any one of claims 1 to 8, characterized in that: The acquisition further includes acquiring vibration data and spectral data related to coal gangue production, In the case of acquiring vibration data related to gangue production, the gangue identification model based on adaptive multimodal feature fusion further includes a vibration feature extraction module, the vibration feature extraction module is used to extract vibration features from the vibration data input to the gangue identification model based on adaptive multimodal feature fusion, the vibration feature extraction module is input to the adaptive multimodal feature fusion module, and When acquiring spectral data related to coal gangue production, the coal gangue identification model based on adaptive multimodal feature fusion further includes a spectral feature extraction module, which is used to extract spectral features from the spectral data input into the coal gangue identification model based on adaptive multimodal feature fusion, and the spectral feature extraction module is input into the adaptive multimodal feature fusion module.

10. A coal gangue identification system based on adaptive multimodal feature fusion, characterized in that: include: A gangue data acquisition module, used to acquire image data and audio data related to gangue production; A gangue recognition module is used to input the image data and the audio data into a trained gangue recognition model based on adaptive multimodal feature fusion to obtain a gangue recognition result output by the gangue recognition model based on adaptive multimodal feature fusion, wherein the gangue recognition model based on adaptive multimodal feature fusion is used to adaptively adjust the weights of different modal features at least based on a channel attention mechanism to identify the gangue to obtain the gangue recognition result.

Images