Intelligent identification method and system of coal gangue based on multimodal visual fusion

Abstract

The present invention provides a method and system for intelligent identification of coal gangue based on multimodal visual fusion, which relates to the field of coal sorting. The method includes: collecting multimodal image data of coal gangue mixture; preprocessing the multimodal image data; aligning visible light image, hyperspectral image and depth of field image; identifying coal gangue according to a deep learning model; fusing visible light identification results and hyperspectral identification results; estimating the volume according to the depth of field height and the coal gangue identification results to obtain the volume of coal and gangue; and calculating the coal gangue content of the coal gangue mixture. The embodiment of the present invention is based on the comprehensive utilization of multimodal visual information, which enhances the ability to capture coal gangue features and improves the accuracy and robustness of identification; based on the use of fusion strategy, the information from different image modalities is effectively integrated, which improves the recognition performance of the system; the high degree of automation can greatly reduce labor costs and improve sorting efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of coal sorting, and in particular to a method and system for intelligently identifying coal gangue based on multimodal visual fusion. Background Art

[0002] Gangue is a solid waste generated during coal mining. It has complex composition and various shapes. Intelligent identification of gangue has always been an important demand of the coal industry. Traditional gangue identification methods mainly rely on manual vision and simple image processing technology, which has the problems of low recognition efficiency and low accuracy. Therefore, the use of image processing technology for gangue identification has become a research hotspot. However, single-modal image recognition methods are often limited by factors such as lighting conditions and occlusion, and the recognition accuracy needs to be improved. Summary of the invention

[0003] The invention solves the problem of low recognition accuracy of the existing coal gangue recognition method based on image processing.

[0004] To solve the above problems, an embodiment of the present invention provides a method for intelligent identification of coal gangue based on multimodal visual fusion, the method comprising: collecting multimodal image data of a coal gangue mixture, the multimodal image data comprising a visible light image, a hyperspectral image and a depth of field image; preprocessing the collected multimodal image data; aligning the preprocessed visible light image, the hyperspectral image and the depth of field image based on an alignment algorithm; identifying coal gangue on the aligned visible light image and the hyperspectral image respectively according to a deep learning model; fusing the visible light identification result and the hyperspectral identification result according to a fusion algorithm, and outputting the coal gangue identification result of each pixel; determining the depth of field height of the pixel according to the depth of field image, and estimating the volume according to the depth of field height and the coal gangue identification result to obtain the volume of coal and gangue; and calculating the coal gangue content of the coal gangue mixture according to the volume of the coal and gangue and the corresponding density information.

[0005] Optionally, the visible light recognition result includes first coal gangue confidence data and first coal gangue identification data of each pixel point; the hyperspectral recognition result includes second coal gangue confidence data and second coal gangue identification data of each pixel point; the coal gangue confidence data is a value between 0 and 1, and the coal gangue identification data is coal or gangue.

[0006] Optionally, the visible light recognition result and the hyperspectral recognition result are fused according to the fusion algorithm to output the coal gangue recognition result of each pixel, including: calculating the first product of the first coal gangue confidence data, the first coal gangue recognition data and the current weight of the current pixel, the second product of the second coal gangue confidence data, the second coal gangue recognition data and the current weight, and calculating the sum of the first product and the second product to obtain the current fusion value; respectively calculating the third product of the first coal gangue confidence data, the first coal gangue recognition data and the adjacent weight of each adjacent pixel, the fourth product of the second coal gangue confidence data, the second coal gangue recognition data and the adjacent weight, and calculating the sum of the third product and the fourth product to obtain the adjacent fusion value; summing the current fusion value and all the adjacent fusion values ​​to obtain the coal gangue recognition fusion value of the current pixel; judging the coal gangue recognition fusion value according to preset conditions to obtain the coal gangue recognition result of each pixel.

[0007] Optionally, the sum of the current weight and all the adjacent weights is equal to one, and the current weight is greater than each of the adjacent weights.

[0008] Optionally, the gangue confidence data of each pixel point in the visible light image is XZ(i,j), and the gangue identification data is XD(i,j); the gangue confidence data of each pixel point in the hyperspectral image is YZ(i,j), and the gangue data is YD(i,j), where i and j are the coordinates of each point; Use the following method to fuse pixel points (i, j): (i,j) is the result of gangue identification:

[0009] in , is the weight of the visible light image and hyperspectral image at the pixel point, which needs to satisfy:

[0010]

[0011] If Z(i,j)<0.5, it is determined to be gangue; if Z(i,j)≥0.5, it is determined to be coal.

[0012] Optionally, the depth of field height of the pixel point is H(I,j), and the volume of the coal is obtained by the following formula:

[0013] The volume of gangue is obtained by the following formula:

[0014] Among them, Zm(i,j) is the pixel point determined to be coal, To determine the pixels as gangue, is the pixel volume factor.

[0015] Optionally, the deep learning model is a convolutional neural network or a recurrent neural network.

[0016] Optionally, the fusion algorithm is early fusion, mid-term fusion or late fusion.

[0017] Optionally, the preprocessing includes at least one of the following: denoising, contrast enhancement, and grayscale.

[0018] An embodiment of the present invention provides a coal gangue intelligent recognition system based on multimodal visual fusion, the system comprising: a depth of field camera, a visible light camera, a hyperspectral camera and an industrial computer; the industrial computer is used to execute the above-mentioned coal gangue intelligent recognition method based on multimodal visual fusion.

[0019] The coal gangue intelligent identification method and system based on multimodal visual fusion provided in the embodiment of the present invention enhance the ability to capture coal gangue characteristics and improve the accuracy and robustness of identification based on the comprehensive utilization of multimodal visual information; based on the use of fusion strategy, information from different image modalities is effectively integrated, thereby improving the recognition performance of the system; the high degree of automation can greatly reduce labor costs and improve sorting efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0021] Figure 1 A system installation block diagram of a coal gangue intelligent identification system based on multimodal visual fusion provided by an embodiment of the present invention; Figure 2 A schematic diagram of a flow chart of a method for intelligently identifying coal gangue based on multimodal visual fusion provided by an embodiment of the present invention; Figure 3 A schematic diagram of a feature fusion strategy provided by an embodiment of the present invention; Figure 4 A schematic diagram of the multimodal visual fusion process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0023] The embodiment of the present invention provides a method and system for intelligent identification of coal gangue based on multimodal visual fusion to solve the problems of low recognition rate and great environmental influence in the prior art. For example, in the process of top coal caving, the coal gangue can be intelligently identified based on the above method to obtain a highly accurate recognition result.

[0024] The coal gangue intelligent identification system based on multimodal visual fusion provided by the embodiment of the present invention has a hardware part including a depth of field camera, a visible light camera, a hyperspectral camera and an industrial computer. It should be noted that the depth of field camera, the visible light camera and the hyperspectral camera can be integrated into the same camera or implemented by different cameras.

[0025] The depth of field camera is used to collect the depth of field image of the coal-gangue mixture on the scraper conveyor, the visible light camera is used to collect the visible light image of the coal-gangue mixture, and the hyperspectral camera is used to collect the hyperspectral image of the coal-gangue mixture, and transmit the image to the above-mentioned industrial computer. The industrial computer can execute the coal-gangue intelligent recognition method based on multimodal visual fusion provided by the embodiment of the present invention.

[0026] Figure 1 The system installation block diagram of the coal gangue intelligent identification system based on multimodal visual fusion provided in an embodiment of the present invention shows an industrial computer 1, a depth of field, visible light, and hyperspectral camera 2, and a scraper conveyor 3.

[0027] Figure 2 A flow chart of a method for intelligently identifying coal gangue based on multimodal visual fusion provided by an embodiment of the present invention, the method comprising the following steps: S202, collecting multimodal image data of the coal-gangue mixture. The multimodal image data includes visible light images, hyperspectral images and depth of field images.

[0028] The coal gangue mixture may be in a moving state, for example, on a scraper conveyor.

[0029] S204: pre-process the collected multimodal image data.

[0030] The above-mentioned preprocessing operation may include at least one of the following: denoising, contrast enhancement, grayscale, etc.

[0031] S206, aligning the pre-processed visible light image, hyperspectral image and depth of field image based on an alignment algorithm.

[0032] Since the positions of the visible light camera, depth of field camera and hyperspectral camera are not exactly the same, the images will have different angles, so a photo alignment algorithm is needed to align the three images. After alignment, the points with the same coordinates in each image are the same point.

[0033] S208, performing coal gangue identification on the aligned visible light image and hyperspectral image respectively according to the deep learning model.

[0034] Optionally, the visible light recognition result includes the first coal gangue confidence data and the first coal gangue recognition data of each pixel point; the hyperspectral recognition result includes the second coal gangue confidence data and the second coal gangue recognition data of each pixel point. The coal gangue confidence data is a value between 0 and 1, and the coal gangue recognition data is the category obtained by recognition, which can be coal or gangue.

[0035] The deep learning model can be a convolutional neural network, a recurrent neural network, or other neural networks suitable for image feature extraction.

[0036] S210, fusing the visible light recognition result and the hyperspectral recognition result according to the fusion algorithm, and outputting the gangue recognition result of each pixel.

[0037] The fusion algorithm can be early fusion, mid-term fusion or late fusion. Taking the late fusion method as an example, it is more conducive to retaining the unique information of each modality.

[0038] Exemplarily, the fusion is performed based on the following method: First, the first product of the first gangue confidence data, the first gangue identification data and the current weight of the current pixel point is calculated, the second product of the second gangue confidence data, the second gangue identification data and the current weight is calculated, and the sum of the first product and the second product is calculated to obtain the current fusion value; Secondly, the third product of the first gangue confidence data, the first gangue identification data and the adjacent weight of each adjacent pixel point, the fourth product of the second gangue confidence data, the second gangue identification data and the adjacent weight, and the sum of the third product and the fourth product are calculated to obtain the adjacent fusion value; Then, the current fusion value is summed with all adjacent fusion values ​​to obtain the gangue recognition fusion value of the current pixel point; Finally, the gangue identification fusion value is determined according to the preset conditions to obtain the gangue identification result of each pixel.

[0039] Among them, the sum of the current weight and all adjacent weights is equal to one, and the current weight is greater than each adjacent weight.

[0040] Figure 3 The feature fusion strategy provided by the embodiment of the present invention is shown, including the following processes: The preprocessed visible light image data is used for deep learning to obtain the gangue confidence data XZ(i,j) and the gangue data XD(i,j) of each point; the preprocessed hyperspectral image data is used for deep learning to obtain the gangue confidence data YD(i,j) and the gangue data YZ(i,j) of each point; data fusion is performed to obtain the gangue data of each pixel.

[0041] Specifically, the gangue confidence data of each pixel point in the visible light image is XZ(i,j), and the gangue identification data is XD(i,j); the gangue confidence data of each pixel point in the hyperspectral image is YZ(i,j), and the gangue data is YD(i,j), where i and j are the coordinates of each point; Use the following method to fuse pixel points (i, j): (i,j) is the result of gangue identification:

[0042] in , is the weight of the visible light image and the hyperspectral image at the pixel point. Generally speaking, when calculating the pixel point (i, j), δx(p,q), The highest weight must meet the following requirements:

[0043]

[0044] Generally can be taken , is 0.5, and the other 8 adjacent pixels such as (i-1, j) are lower, which can generally be taken as 0.0625.

[0045] If Z(i,j)<0.5, it is determined to be gangue; if Z(i,j)≥0.5, it is determined to be coal.

[0046] S212, determining the depth height of the pixel point according to the depth image, and performing volume estimation according to the depth height and the coal gangue recognition result to obtain the volume of the coal and gangue.

[0047] Specifically, if the depth of field height of the pixel point is H(I,j), the volume of the coal is obtained by the following formula:

[0048] The volume of gangue is obtained by the following formula:

[0049] Among them, Zm(i,j) is the pixel point determined to be coal, To determine the pixels as gangue, is the pixel volume factor.

[0050] S214, calculating the coal gangue content of the coal gangue mixture according to the volumes of the coal and gangue and the corresponding density information.

[0051] In this embodiment, the height information H(i, j) of the depth of field camera can be used to calculate the volume of the gangue, and then the density can be used to obtain the weight information of the gangue. In an image of size m×n, the mass of the coal is obtained by the following formula: Fm=

[0052] in for The pixels identified as coal, is the pixel volume coefficient, Dm is the density of coal; In an image of size m×n, the mass of the gangue is obtained by the following formula: Fg=

[0053] in for The pixels identified as gangue, is the pixel volume coefficient, and Dg is the density of gangue.

[0054] The coal gangue intelligent recognition method based on multimodal visual fusion provided by the embodiment of the present invention enhances the ability to capture coal gangue characteristics and improves the accuracy and robustness of recognition based on the comprehensive utilization of multimodal visual information; based on the use of fusion strategy, information from different image modalities is effectively integrated, thereby improving the recognition performance of the system; it has a high degree of automation, can greatly reduce labor costs, and improves sorting efficiency.

[0055] Figure 4 A schematic diagram of a multimodal visual fusion process provided by an embodiment of the present invention is shown, including the following processes: The visible light vision sensor acquires the image, and the image is preprocessed; The hyperspectral vision sensor acquires images, and the images are preprocessed; The depth-of-field camera acquires an image, and the image is preprocessed; The above images are subjected to graphic alignment; Deep learning and deep fusion of visible light images and hyperspectral images; Calculate the volume of coal gangue using point-by-point coal gangue data and point-by-point height data; The gangue ratio is calculated based on the gangue density.

[0056] Compared with the prior art, the embodiments of the present invention have the following significant advantages: 1. The comprehensive use of multimodal visual information, which utilizes the visual information of visible light and non-visible light, enhances the ability to capture the characteristics of coal gangue and improves the accuracy and robustness of recognition.

[0057] 2. The use of fusion strategy enables the information from different image modalities to be effectively integrated, further improving the recognition performance of the system.

[0058] 3. High degree of automation can greatly reduce labor costs and improve sorting efficiency.

[0059] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the control device through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it may include the processes of the above-mentioned method embodiments, wherein the storage medium may be a memory, a disk, an optical disk, etc.

[0060] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements. The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for intelligent identification of coal gangue based on multimodal visual fusion, characterized in that: The method comprises: Collecting multimodal image data of the coal-gangue mixture, wherein the multimodal image data includes a visible light image, a hyperspectral image, and a depth of field image; Preprocessing the collected multimodal image data; Aligning the preprocessed visible light image, the hyperspectral image and the depth of field image based on an alignment algorithm; Respectively performing coal gangue recognition on the aligned visible light image and the hyperspectral image according to the deep learning model; The visible light recognition results and the hyperspectral recognition results are fused according to the fusion algorithm, and the gangue recognition results of each pixel are output; Determine the depth of field height of the pixel point according to the depth of field image, and perform volume estimation according to the depth of field height and the coal gangue recognition result to obtain the volume of coal and gangue; The coal gangue content of the coal gangue mixture is calculated according to the volumes of the coal and gangue and the corresponding density information.

2. The method according to claim 1, characterized in that: The visible light recognition result includes the first coal gangue confidence data and the first coal gangue recognition data of each pixel point; the hyperspectral recognition result includes the second coal gangue confidence data and the second coal gangue recognition data of each pixel point; The coal gangue confidence data is a value between 0 and 1, and the coal gangue identification data is coal or gangue.

3. The method according to claim 2, characterized in that The method of fusing the visible light recognition result and the hyperspectral recognition result according to the fusion algorithm and outputting the gangue recognition result of each pixel point includes: Calculating a first product of the first coal gangue confidence data, the first coal gangue identification data and a current weight of a current pixel point, a second product of the second coal gangue confidence data, the second coal gangue identification data and the current weight, and calculating the sum of the first product and the second product to obtain a current fusion value; The first gangue confidence data, the first gangue identification data and the third product of the adjacent weight of each adjacent pixel point, the second gangue confidence data, the second gangue identification data and the fourth product of the adjacent weight, and the sum of the third product and the fourth product are calculated to obtain an adjacent fusion value; Summing the current fusion value and all the adjacent fusion values ​​to obtain the gangue recognition fusion value of the current pixel point; The gangue identification fusion value is determined according to preset conditions to obtain the gangue identification result of each pixel point.

4. The method according to claim 3, characterized in that The sum of the current weight and all the adjacent weights is equal to one, and the current weight is greater than each of the adjacent weights.

5. The method according to claim 3, characterized in that: The gangue confidence data of each pixel point in the visible light image is XZ(i,j), and the gangue identification data is XD(i,j). The gangue confidence data of each pixel point in the hyperspectral image is YZ(i,j), and the gangue data is YD(i,j), where i and j are the coordinates of each point. Use the following method to fuse pixel points (i, j): (i,j) is the result of gangue identification: in , is the weight of the visible light image and hyperspectral image at the pixel point, which needs to satisfy: If Z(i,j)<0.5, it is determined to be gangue; if Z(i,j)≥0.5, it is determined to be coal.

6. The method according to claim 1, characterized in that The depth of field height of the pixel point is H(I,j), and the volume of the coal is obtained by the following formula: The volume of gangue is obtained by the following formula: Among them, Zm(i,j) is the pixel point determined to be coal, To determine the pixels as gangue, is the pixel volume factor.

7. The method according to claim 1, characterized in that The deep learning model is a convolutional neural network or a recurrent neural network.

8. The method according to claim 1, characterized in that The fusion algorithm is early fusion, mid-term fusion or late fusion.

9. The method according to claim 1, characterized in that: The preprocessing includes at least one of the following: denoising, contrast enhancement, and grayscale conversion.

10. A coal gangue intelligent recognition system based on multimodal visual fusion, characterized in that: The system includes: a depth of field camera, a visible light camera, a hyperspectral camera and an industrial computer; The industrial computer is used to execute the coal gangue intelligent identification method based on multimodal visual fusion as described in any one of claims 1-9.

Images