Coal briquette and gangue global visual identification method and system based on multi-modal fusion

Abstract

The invention relates to a coal briquette and gangue global visual identification method and system based on multi-modal fusion, and the method comprises the following steps: a plurality of RGB-D cameras distributed above a conveyor belt carry out the collection of RGB images and point cloud images, the preprocessing of point cloud data, the making of a data set, and the synchronous reasoning of a training model for the training of a model, and a fusion result. And identifying the type and position information of the target to be identified, and realizing global target detection through an SORT tracking algorithm. According to the method, multi-depth-camera point cloud calibration splicing, point cloud background filtering and depth camera installation deviation correction are realized, the flow in the multi-depth-camera point cloud acquisition and preprocessing process is simplified, and the multi-depth-camera point cloud processing efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and more particularly to a method and system for global visual recognition of coal blocks and gangue based on multimodal fusion. Background Art

[0002] During the coal mining process, due to the influence of mining technology and geological conditions, some coal blocks and gangue (coal gangue) or other foreign matter are usually mixed into the raw coal mined and transported, which will have an adverse effect on subsequent transportation, washing and other links.

[0003] Among existing technologies, Chinese Patent Publication No. CN119741581A provides a method and system for intelligently identifying coal and gangue based on multimodal visual fusion. The system collects visible light images, hyperspectral images, and depth-of-field images, identifies visible light and hyperspectral results, and fuses the results, improving recognition accuracy and robustness. Another invention patent, Chinese Patent Publication No. CN119180974A, provides an intelligent coal and gangue recognition and analysis system based on machine vision and image processing. By comprehensively considering image color, texture, and morphological features, thermal distribution, thermal boundaries, and thermal conductivity effects in thermal images, and internal uniformity in X-ray imaging, the system evaluates targets and identifies and detects coal and gangue targets. Chinese Patent Publication No. CN117036953A discloses a multi-feature fusion coal and gangue recognition method. This method uses a depth camera's two-dimensional image for target detection, three-dimensional point cloud volume calculation, and weight-based density calculation to achieve coal and gangue recognition.

[0004] Although existing technologies use multimodal fusion to identify and detect coal and gangue, the following problems still exist: 1. Patents CN119741581A and CN119180974A do not take into account the three-dimensional geometric features of coal and gangue. The data obtained from multimodal acquisition is usually two-dimensional information, and the identification and detection of coal and gangue are achieved by using two-dimensional information, ignoring the geometric features of coal and gangue. Patent CN117036953A uses three-dimensional geometric features, but only for calculating volume dimensions. The identification of coal and gangue still requires external weight sensing to calculate density. 2. Existing technologies do not take into account factors such as long conveyor belt transportation distances and unstable lighting conditions in mines. They rely on a large number of sensors with different modalities for real-time monitoring, resulting in poor overall system robustness. Furthermore, they do not consider the movement of gangue targets when being processed on long conveyor belts and the need for global tracking in poor lighting conditions. Summary of the Invention

[0005] To solve the above problems, the present invention provides a global visual recognition method and system for coal blocks and gangue based on multimodal fusion, which realizes multi-depth camera point cloud calibration and splicing, point cloud background filtering, and depth camera installation deviation correction, simplifies the process of multi-depth camera acquisition and preprocessing of point clouds, and improves the efficiency of multi-depth camera point cloud processing.

[0006] The present invention provides a method for global visual recognition of coal blocks and gangue based on multimodal fusion, comprising the following steps:

[0007] S1 uses multiple RGB-D cameras distributed above the conveyor belt to perform multimodal acquisition of RGB images and depth images;

[0008] S2, pre-processing of the collected data, including registration, noise filtering, correction and conversion;

[0009] S3, respectively, creates a multi-type target detection dataset for RGB images and a target detection dataset for deep images, and trains a target detection model based on YOLOv5 based on the above datasets;

[0010] S4 deploys the algorithm and model to the computing unit to perform real-time inference results, and performs multimodal fusion of multi-type target detection results of RGB images and depth image target detection results to identify coal blocks and gangue and their coordinate position information, and combines Kalman filtering and Hungarian matching tracking algorithms to track the results of depth image target detection.

[0011] Among them, in step S2, the conversion and preprocessing of the collected point cloud data specifically includes:

[0012] S2.1, multi-RGB-D camera calibration:

[0013] Multiple calibration objects are placed in the overlapping detection area between the two cameras to be calibrated, and point clouds are recorded. The range of the calibration objects is manually selected, and the point clouds of the non-overlapping parts are filtered out. Only the calibration object point clouds of the two cameras that are completely in the overlapping detection area are retained. This point cloud is then used to perform multi-camera registration and calibration.

[0014] S2.2, point cloud background filtering:

[0015] Background filtering uses straight-through filtering for elevation coarse filtering and slope filtering for fine filtering to jointly process background filtering, and further calculates the conveyor belt horizontal plane through the filtered background points, and performs horizontal rotation correction and correction on the target point cloud;

[0016] S2.3, point cloud posture correction:

[0017] By filtering out the conveyor belt background, fitting the plane, calculating the correction matrix, and performing rotational posture correction on all point clouds;

[0018] S2.4, Bird's-eye view depth image conversion:

[0019] Determine the projection interval range and spatial resolution parameters, normalize the stored results, and store the normalized results as 8-bit depth images;

[0020] S2.5, depth image hole and crack repair:

[0021] During the bird's-eye view depth image conversion process, in order to preserve the geometric features as much as possible, the resolution of the depth image conversion process is set to the threshold where holes and cracks will be generated after the conversion. The holes and cracks in the converted depth image are repaired by image dilation to repair the holes and cracks in the depth image caused by high resolution.

[0022] Among them, in step S3, the preparation of data sets and training of models specifically include

[0023] S3.1, coal and gangue object detection model for RGB images:

[0024] Using the RGB data collected by the first RGB-D camera with fill light as samples, we trained a YOLOv5 object detection model for classifying and identifying coal lumps and gangue. This model enabled the first RGB-D camera with fill light to identify the target type as coal lumps or gangue, as well as its XY coordinate location.

[0025] S3.2, point cloud bird's-eye view depth image model:

[0026] After filtering the conveyor belt background from the original point cloud, the remaining target point cloud is clustered using the DBSCAN clustering algorithm to obtain each gangue and coal point cloud cluster, and its three-dimensional bounding box is calculated. The calculated bounding box result is corrected by the posture correction matrix and projected into the coordinate system of the two-dimensional bird's-eye view depth image as the label for bird's-eye view depth image target detection, and a global target detection model based on the depth map is trained.

[0027] Among them, in step S4, the training model is used to perform reasoning synchronously, and the results are integrated to identify the type and location information of the target to be identified. The global target detection is achieved through the SORT tracking algorithm. Specifically,

[0028] S4.1, Reasoning of the model:

[0029] After the model is deployed, the first RGB-D camera collects RGB image data and uses the RGB image model for inference to identify coal and gangue within the RGB detection range, obtaining the target type label and coordinate position in the XY plane. All RGB-D cameras collect global depth point cloud information. After calibration and stitching, background filtering, posture correction, depth image conversion, and hole repair in step S2, the bird's-eye view depth image object detection model is used to obtain the global XY coordinates of each target.

[0030] 4.2, Multimodal detection result fusion:

[0031] Based on the RGB image detection results of the first RGB-D camera and the depth detection results of all RGB-D cameras, the RGB image detection frame is mapped to the depth image detection frame, and the type label information is passed;

[0032] Perform IOU matching between the converted RGB image detection frame and the depth image detection frame;

[0033] After the matching is completed, the label result in the RGB-D camera detection result corresponding to the matching result is assigned to the depth image detection frame, and the label type information is passed;

[0034] 4.3, Depth Camera Target Tracking:

[0035] In order to realize the process of global target detection with labels, Kalman filtering and Hungarian matching are used to achieve tracking, and target tracking is performed on the results of depth camera target detection with matched labels.

[0036] The present invention also provides a global visual recognition system for coal blocks and gangue based on multimodal fusion, which is used for the above-mentioned global visual recognition method for coal blocks and gangue based on multimodal fusion, including several RGB-D cameras arranged on a conveyor belt.

[0037] Furthermore, multiple RGB-D cameras are distributed one by one in the conveying direction of the conveyor belt, and a fill light device for fill light is set up near the first RGB-D camera.

[0038] The beneficial effects of the present invention are:

[0039] The present invention provides a global visual recognition system for coal and gangue based on multimodal fusion. In response to poor lighting conditions in mines, the advantages of different sensing devices for different tasks are brought into play. RGB-D cameras can capture clear visible light images and rich target surface texture and color information, but they have high requirements for lighting conditions and require additional fill light to better capture and identify target features. Depth cameras can capture geometric information about the target surface. At the same time, depth cameras are active sensing devices with little dependence on lighting conditions. However, depth cameras cannot capture color images and are less sensitive to information such as the target surface's material texture.

[0040] The classification of coal and gangue usually relies on information such as surface material texture collected by RGB-D cameras, while the point cloud collected by depth cameras is difficult to distinguish. The conveyor belts transporting coal and gangue are affected by factors such as unstable lighting, and the detection range of a single RGB-D camera cannot fully cover the entire conveyor belt system. To this end, this invention fully considers the characteristics of both sensors, using RGB-D cameras for type recognition and depth cameras for global tracking to achieve target detection in the global scene.

[0041] The present invention provides a global visual recognition method for coal blocks and gangue based on multimodal fusion, which realizes the calibration and splicing of multi-depth camera point clouds, point cloud background filtering, and depth camera installation deviation correction, simplifies the process of multi-depth camera point cloud acquisition and preprocessing, and improves the efficiency of multi-depth camera point cloud processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Schematic diagram of the structure of the global visual recognition system for coal blocks and gangue based on multimodal fusion of the present invention;

[0043] Figure 2 This is a flow chart of the method for global visual recognition of coal blocks and gangue based on multimodal fusion of the present invention;

[0044] Figure 3 Schematic diagram of the structure after repairing holes and cracks in the depth image in the present invention;

[0045] Figure 4 Flowchart of global target detection achieved by SORT tracking algorithm in the present invention;

[0046] Figure 5 Schematic diagram of the structure after the state matrix is ​​updated in the present invention. DETAILED DESCRIPTION

[0047] The preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more precise definition of the protection scope of the present invention.

[0048] Example 1

[0049] like Figure 1 The system is shown as a global visual recognition system for coal and gangue based on multimodal fusion, including multiple RGB-D cameras 1, 2, ..., N distributed above a conveyor belt. The multiple RGB-D cameras are distributed above the conveyor belt and are distributed one by one in the conveying direction of the conveyor belt. The detection range of a single RGB-D camera can cover both ends of the conveyor belt. At the same time, the coverage detection range of the two RGB-D cameras is wide enough to accommodate the detection objects used for calibration, and the target will not be lost when passing through the intersection between the two RGB-D cameras.

[0050] A fill light device is set up near the first RGB-D camera 1 to ensure that there is sufficient light within the detection range of the first RGB-D camera 1. This ensures that the RGB image of the first RGB-D camera 1 is clearly visible, and subsequent RGB-D cameras do not need fill light.

[0051] Example 2

[0052] like Figure 2 The global visual recognition method of coal blocks and gangue based on multimodal fusion shown in the figure specifically includes the following steps:

[0053] S1, multiple RGB-D cameras distributed above the conveyor belt collect RGB images and depth images (point cloud images):

[0054] An RGB-D camera is used to capture RGB images, and a subsequent RGB-D camera is used to capture depth images (point cloud images).

[0055] S2, preprocessing of point cloud data:

[0056] S2.1 Multi-RGB-D Camera Calibration:

[0057] Multiple calibration objects are placed in the overlapping detection area between the two RGB-D cameras to be calibrated, and point clouds are recorded. The range of the calibration objects is manually selected, and the point clouds in the non-overlapping areas are filtered out, retaining only the calibration object point clouds of the two cameras that are completely in the overlapping detection area.

[0058] The registration process uses the Iterative Closest Point (ICP) method to perform point cloud registration. First, through some prior knowledge and manual adjustment, the source point cloud P = {p1, p2, ..., p n} and target point cloud The transformation matrix between the initial pose estimate and, secondly, for each point in the original point cloud, find the nearest point And through the minimum mean square error optimization, the error is minimized Implements the rotation and translation between two point clouds that minimizes the error.

[0059] The calibration of multiple RGB-D cameras uses the coordinate system of the first camera 1 as the fill light as the world coordinate system. The second camera 2 is used as the source coordinate system, and the coordinate system of the first camera 1 is used as the target coordinate system for calibration; the third camera 3 is used as the source coordinate system, and the second camera 2 is used as the target coordinate system for calibration, and so on, until all cameras N are calibrated. The calibration results are: {[R1 T1], [R2 T2], ... [R n-1 T n-1 ]}, where [R k T k ] represents the rigid transformation matrix for calibration with camera k-1 as the target coordinate system and camera k as the source coordinate system. The kth transformation is Then a point in camera k Its second expression in the coordinate system of camera k is: The transformation of this point from the camera k coordinate system to the world coordinate system is

[0060] S2.2 Point cloud background filtering:

[0061] The background filtering method uses straight-through filtering for elevation coarse filtering and slope filtering for fine filtering to jointly process the background filtering, and further calculates the conveyor belt horizontal plane through the filtered background points, and performs horizontal rotation correction and correction on the target point cloud.

[0062] The straight-through filtering takes the Z axis as the reference and assumes that the initial point cloud is The height threshold interval is z min ≤z i ≤z max , then the straight-through filtering result is By artificially selecting z min With z max , determine the initial height range of the conveyor belt background for coarse filtering. The selected range should be slightly larger to include all conveyor belt background point clouds as much as possible.

[0063] The background point cloud filtered by the straight-through filter is finely filtered using the slope filtering method. i =(x i ,y i ,z i ) and its neighbor point p j =(x j ,y j,z j ), the slope between the two points is The slope angle is By setting the threshold, the slope angle θ ij <θ th Point p i Filter out as conveyor belt background points.

[0064] S2.3 Point cloud posture correction:

[0065] By filtering out the conveyor belt background, fitting the plane, and calculating the correction matrix. The background point cloud set is Calculate its three-dimensional centroid coordinates The point set in the background point cloud is decentralized by the three-dimensional centroid coordinates to obtain the point set Q and calculate its covariance matrix And perform principal component analysis to obtain its eigenvector n = (a, b, c), which is the plane normal vector of the conveyor belt background point cloud. Calculate its rotation matrix on the z-axis, which is the rotation matrix R required for point cloud posture correction. align , perform rotational posture correction on all point clouds.

[0066] S2.4 Bird’s-eye view depth image conversion:

[0067] Determine the projection interval x min ,x max ,y min ,y max And the spatial resolution r parameter. For any point p i =(x i ,y i ,z i ), calculate its pixel position index (u i ,v i ) is calculated as follows: When there are multiple point projections at the location of a pixel index, the average value strategy is used where n u,v is the number of points indexed to the position (u, v), z i The elevation of the point indexed to (u,v).

[0068] Normalize the stored results. Among them D min ,D max The minimum and maximum depths of all points after indexing and averaging are stored as 8-bit depth images after normalization.

[0069] S2.5 depth image hole and crack repair:

[0070] During the bird's-eye view depth image conversion process, in order to preserve the geometric features as much as possible, the resolution of the depth image conversion process is set to the threshold where holes and cracks will be generated after the conversion. The converted depth image needs to be repaired for holes and cracks. The repair adopts image expansion, setting a 3*3 structure element S, considering the area covered by the structure element u for each pixel in the image, calculating the maximum value in the area, and assigning the maximum value to the center element. This process is repeated step by step to achieve image expansion, so as to repair the holes and cracks in the depth image caused by high resolution. Figure 3 shown.

[0071] S3, data set preparation and model training:

[0072] S3.1 Coal and gangue object detection model for RGB images:

[0073] Using RGB data collected from an RGB-D camera with fill light as samples, we trained a YOLOv5 object detection model for classifying and identifying coal lumps and gangue. This model can identify the target type as coal lumps or gangue, and also identify its XY coordinate location, using an RGB-D camera with fill light.

[0074] S3.2 Point cloud bird’s-eye view depth image model:

[0075] The data annotation of the bird's-eye view depth image of the point cloud is done through an automated annotation process based on DBSCAN clustering. After the original point cloud is filtered out by the conveyor belt background, the remaining target point cloud is clustered using the DBSCAN clustering algorithm. The eps neighborhood of a point p is defined as N eps (p)={q∈D|distance(p,q)≤eps}, which represents the set of points whose distance from a certain point p is less than eps. If the points in the eps neighborhood of point p are greater than the threshold min_Pts, then point p is called a core point; if a point is not a core point, but is located in the neighborhood of a certain point, it is called a boundary point; if a point does not belong to the neighborhood of any core point, nor is it a core point itself, it is marked as a noise point. DBSCAN clustering traverses and visits each point, calculates and determines whether it is a noise point or a point within the cluster, until all points are divided into corresponding clusters or are noise points. Through the above algorithm process, all the gangue and coal block point clouds in the sample are clustered, and each gangue and coal block point cloud cluster is obtained, and its three-dimensional bounding box is calculated. The calculated bounding box result is further passed through the posture correction matrix R align The image is rectified and projected into the coordinate system of the two-dimensional bird's-eye view depth image. The annotation box is refined by manual inspection and used as the label for target detection in the bird's-eye view depth image.

[0076] Use the automatically annotated dataset to train the YOLOv5 target detection model for bird's-eye view depth images to achieve target detection of global targets.

[0077] S4 uses the training model to perform reasoning simultaneously and fuses the results to identify the type and location information of the target to be identified, and realizes global target detection through the SORT tracking algorithm, such as Figure 4 shown.

[0078] 4.1 Model Reasoning:

[0079] After the model is deployed, camera 1 collects RGB image data and uses the RGB image model for inference to identify coal and gangue within the detection range of camera 1, obtaining the target type label and coordinate position in the XY plane. Cameras 1, 2, ..., N collect point cloud information of the depth distance of the targets on the conveyor belt. After calibration and stitching, background filtering, posture correction, depth image conversion, and hole repair in S2, the target detection model of the bird's-eye view depth image is used to obtain the global XY coordinates of each target.

[0080] 4.2 Multimodal detection result fusion:

[0081] The detection results of the RGB-D camera in camera 1 are in

[0082] The detection result includes the center XY coordinates and width and height of a detection frame, and the type of the target is coal or gangue; the result of the depth camera detection in camera 1 is in The result of a detection box includes the XY coordinates of the target center point and the width and height of the bounding box.

[0083] Through mapping Realize the mapping from RGB-D camera detection frame to depth camera detection frame. Its projection transformation is is the transformation result of the RGB-D camera calibration, where K d ,R,K r It is the internal and external parameters that have been calibrated in advance for the RGB-D camera.

[0084] Perform IOU matching between the converted RGB image detection frame and the depth image detection frame. Find the result with the largest IOU matching result and greater than the matching threshold θ, that is and satisfy

[0085] After the matching is completed, the label result c in the RGB-D camera detection result corresponding to the matching result jAssign a depth image detection frame, i.e. in and

[0086] The final depth image target detection result is in is the detection box for target detection, Type label data for target detection.

[0087] 4.3 Depth Camera Target Tracking:

[0088] In order to realize the process of global target detection with labels, Kalman filtering and Hungarian matching are used to achieve tracking, and target tracking is performed on the results of depth camera target detection with matched labels.

[0089] The result obtained by the depth camera detection is B = (x, y, w, h), which are the center coordinates and width and height of the detection box respectively. The state vector of the Kalman filter is

[0090]

[0091] Contains the current detection box center coordinates and speed; its prediction result is Where F is the state transfer matrix; the update process of the state matrix is where z k is the current detection frame, K is the Kalman gain, which is used to dynamically adjust the balance between the predicted value and the measured value. Further, the detection frame of each frame is calculated by calculating the difference between the detection frame of the previous frame and the Use the Hungarian algorithm to associate the detection frame of the current frame with the prediction frame of the previous frame, select the best match, and finally update the state matrix to achieve continuous tracking of the target, such as Figure 5 shown.

[0092] Finally, it should be noted that the above embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A global visual recognition method for coal blocks and gangue based on multimodal fusion, characterized in that: The steps include: S1 uses multiple RGB-D cameras distributed above the conveyor belt to perform multimodal acquisition of RGB images and depth images; S2, preliminary conversion and preprocessing of the collected data; S3, respectively, creates a multi-type target detection dataset for RGB images and a target detection dataset for deep images, and trains a target detection model based on YOLOv5 based on the above datasets; S4 deploys the algorithm and model to the computing unit to perform real-time inference results, and performs multimodal fusion of multi-type target detection in RGB images and target detection in depth images to identify coal blocks and gangue, and detect their coordinate position information. It also tracks the results of target detection in depth images by combining the Kalman filter and Hungarian matching tracking algorithm.

2. The method for global visual recognition of coal blocks and gangue based on multimodal fusion according to claim 1 is characterized in that: Step S2 specifically includes: S2.1, multi-RGB-D camera calibration: Multiple calibration objects are placed in the overlapping detection area between the two RGB-D cameras to be calibrated, and point clouds are recorded. The range of the calibration objects is manually selected, and the point clouds in the non-overlapping part are filtered out. Only the calibration object point clouds of the two cameras that are completely in the overlapping detection area are retained. This point cloud is then used to perform multi-camera registration and calibration. S2.2, point cloud background filtering: Through-through filtering is used for coarse elevation filtering and slope filtering is used for fine filtering to remove background. The conveyor belt horizontal plane is further calculated based on the filtered background points, and the target point cloud is corrected and corrected for horizontal rotation. S2.3, point cloud posture correction: By filtering out the conveyor belt background, fitting the plane, calculating the correction matrix, and performing rotational posture correction on all point clouds; S2.4, Bird's-eye view depth image conversion: Determine the projection interval range and spatial resolution parameters, normalize the stored results, and store the normalized results as 8-bit depth images; S2.5, depth image hole and crack repair: The converted depth image is repaired for holes and cracks. Image dilation is used to repair holes and cracks in the depth image caused by high resolution.

3. The method for global visual recognition of coal blocks and gangue based on multimodal fusion according to claim 2 is characterized in that: Step S3 specifically includes: S3.1, coal and gangue object detection model for RGB images: Using the RGB data collected by the first RGB-D camera as samples, we trained a YOLOv5 object detection model for classifying and identifying coal lumps and gangue. This model enabled the first RGB-D camera to identify the target type as coal lumps or gangue, as well as its XY coordinate location. S3.2, point cloud bird's-eye view depth image model: After filtering out the conveyor belt background from the original point cloud, the remaining target point cloud is clustered using the DBSCAN clustering algorithm to obtain each gangue and coal block point cloud cluster, and its three-dimensional bounding box is calculated. The calculated bounding box result is corrected by the posture correction matrix and projected into the coordinate system of the two-dimensional bird's-eye view depth image. It is used as the label for target detection in the bird's-eye view depth image to achieve automatic labeling. After the labeling is completed, the label is manually inspected and refined. This dataset is used to train a model for global positioning of conveyor belt targets.

4. The method for global visual recognition of coal blocks and gangue based on multimodal fusion according to claim 3 is characterized in that: Step S4 specifically includes: S4.1, Reasoning of the model: After the model is deployed, the first RGB-D camera collects RGB image data and uses the RGB image model for inference to identify coal and gangue within the detection range of the first RGB-D camera, obtaining the target type label and coordinate position in the XY plane. All RGB-D cameras collect global depth point cloud information. After calibration and stitching, background filtering, posture correction, depth image conversion, and hole repair in step S2, the bird's-eye view depth image target detection model is used to obtain the global XY coordinates of each target. 4.2, Multimodal detection result fusion: Based on the RGB image detection results of the first RGB-D camera and the depth detection results of the subsequent RGB-D cameras, the RGB image detection frame is mapped to the depth image detection frame; Perform IOU matching between the converted RGB image detection frame and the depth image detection frame; After the matching is completed, the label result in the RGB-D camera detection result corresponding to the matching result is assigned to the depth image detection frame, and the label type information is passed; 4.3, Depth Camera Target Tracking: Kalman filtering and Hungarian matching are used to achieve tracking, and target tracking is performed on the results of depth camera target detection with matched labels.

5. A global visual recognition system for coal and gangue based on multimodal fusion, used in the global visual recognition method for coal and gangue based on multimodal fusion according to any one of claims 1 to 4, characterized in that: It includes several RGB-D cameras installed on the conveyor belt.

6. The coal block and gangue global visual recognition system based on multimodal fusion according to claim 5 is characterized in that: Multiple RGB-D cameras are distributed one by one in the conveying direction of the conveyor belt, and a fill light device is set up near the first RGB-D camera.

Images