Coal gangue classification and identification method and system based on image vision

Abstract

The invention relates to the technical field of coal gangue recognition, in particular to a coal gangue classification recognition method and system based on image vision, and the method comprises the steps: obtaining a mixed image of coal and coal gangue through employing a camera array, and dividing a plurality of candidate regions; based on the overlapping position between the candidate areas, the mixing state of the coal and the coal gangue on the overlapping position is determined, and feature loss is used for distinguishing the sheltered area and the unsheltered area under coal gangue mixing; on the basis of continuous frame images of the occlusion area, synchronizing feature loss of the occlusion area to an adjacent non-occlusion area, and determining a spatial similarity threshold value of the adjacent non-occlusion area; performing multi-target cooperative processing according to the spatial position of the sheltered area, and unifying classification labels of coal gangue in each candidate area; and defining region features of each candidate region according to the value range of the classification tag, and taking the region features as synchronously output data. The accuracy and efficiency of coal gangue recognition are realized.

Description

Technical Field

[0001] This invention relates to the field of coal gangue identification technology, specifically to a coal gangue classification and identification method and system based on image vision. Background Technology

[0002] Coal gangue is a typical associated solid waste generated during coal mining and washing, exhibiting significant regional genetic differences and mineral diversity. Its mineral composition mainly consists of two categories: one is clay minerals, such as kaolinite, illite, and chlorite; the other is gangue minerals, typically represented by quartz and feldspar. In addition, it often contains a certain proportion of non-mineral phases such as pyrite, calcite, and residual carbonaceous matter.

[0003] The identification process between coal and gangue is generally based on image acquisition and analysis to distinguish between them. When the images are inconsistent, they are marked and the corresponding gangue is removed. However, this method is difficult to identify the mixed state of coal and gangue, resulting in reduced image recognition accuracy due to the occlusion of coal and gangue, thus wasting coal resources.

[0004] For example, Chinese Patent Publication No. CN119180974A discloses an intelligent identification and analysis system for coal gangue based on machine vision and image processing. It analyzes the fit score of each target object relative to coal gangue by combining the first, second and third order matching degrees of each target object relative to coal gangue, thereby identifying whether each target object is coal gangue. The first order matching degree comprehensively considers the color features, texture features and morphological features in the appearance image of each target object. The second order matching degree comprehensively considers the heat distribution effect, heat boundary effect and thermal conductivity effect fed back in the thermal image of each target object. The third order matching degree comprehensively considers the uniformity within the X-ray imaging partition of each target object and the uniformity between partitions. The matching degree of different orders evaluates the target object from different angles, making the identification of coal gangue have strong complementarity and accuracy.

[0005] For example, Chinese Patent Publication No. CN119295953A discloses a multi-scale detection method and system for coal gangue based on the YOLOv8 network. This method increases the detection capability of small targets by embedding the CBAM attention mechanism into the core module C2f of the entire network. Secondly, a fast spatial pyramid pooling module is adopted in the backbone network to fuse more preceding and following features, enhance the interaction of feature information, and further improve the multi-scale detection capability. Then, a bidirectional feature fusion module with more skip connections is used to increase the multi-scale detection capability and prevent the loss of small target features due to network depth. Finally, the convolutional blocks used for feature fusion in the detection head are merged to reduce weight, and the regression loss is set to EIoU_Loss to avoid errors caused by the consistent aspect ratio of the predicted box and the ground truth box.

[0006] Existing technologies use appearance images, thermal imaging, and X-ray imaging as the subjects of detection and analysis, and use the features of coal gangue identified in three dimensions as distinguishing points to complete the classification of coal gangue; and pyramid pooling analysis is used to complete the multi-scale detection processing of coal gangue. However, these processing methods ignore the inaccurate classification caused by factors such as element overlap and occlusion and feature loss. They emphasize feature pooling and convolution processing under a single frame image, which makes it easy for the captured feature values ​​to be difficult to adapt to the spatial information corresponding to the transport belt, thus causing misidentification when coal and coal gangue are mixed. Summary of the Invention

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a coal gangue classification and identification method based on image vision, including: S1, using a camera array to acquire a mixed image of coal and coal gangue, and using the position coordinates of any element in the mixed image as a basis to divide multiple candidate regions.

[0008] S2, based on the overlapping positions between each candidate region, determines the mixing state of coal and gangue at the overlapping positions, quantifies the feature loss value of each candidate region under overlapping occlusion according to the mixing state, and uses the feature loss to distinguish between occluded and unoccluded regions under gangue mixing.

[0009] S3. Based on the continuous frame images of the occluded region, perform grid fitting on the occluded region and the adjacent unoccluded region in the current frame, and synchronize the feature loss of the occluded region to the adjacent unoccluded region to determine the spatial similarity threshold of the adjacent unoccluded region.

[0010] S4 uses the spatial location of the occluded area for multi-target collaborative processing, and combines the spatial similarity threshold between adjacent unoccluded areas to unify the classification label of coal gangue in each candidate area.

[0011] S5 defines the regional characteristics of each candidate region based on the range of values ​​for the classification labels, and uses the regional characteristics as synchronous output data.

[0012] The image vision-based coal gangue classification and recognition system includes: a region segmentation module, which uses a camera array to acquire a mixed image of coal and coal gangue, and uses the position coordinates of any element in the mixed image as a basis to divide multiple candidate regions.

[0013] The occlusion recognition module is used to determine the mixing state of coal and gangue at the overlapping positions based on the overlapping positions between each candidate region. It quantifies the feature loss value of each candidate region under overlapping occlusion according to the mixing state, and uses the feature loss to distinguish between occluded and unoccluded regions under the mixing of coal and gangue.

[0014] The feature synchronization module is used to perform grid fitting on the occluded region and the adjacent unoccluded region in the current frame based on the continuous frame images of the occluded region, and synchronize the feature loss of the occluded region to the adjacent unoccluded region to determine the spatial similarity threshold of the adjacent unoccluded region.

[0015] The classification and labeling module is used for multi-target collaborative processing based on the spatial location of occluded areas. It combines the spatial similarity threshold between adjacent unoccluded areas to unify the classification labels of coal gangue in each candidate area.

[0016] The data output module is used to define the regional characteristics of each candidate region based on the value range of the classification label, and to output the regional characteristics as data synchronously.

[0017] The beneficial effects of this invention are as follows: First, this invention acquires mixed images through a camera array, locks the position of the same element by aligning with timestamps, constructs an original feature pool by combining the size, height, and pixel value of coal gangue, generates a semantic segmentation label map through clustering, and then optimizes it by size constraints and overlap filtering to realize the location and filtering of candidate regions, thereby completing the basic definition of coal gangue data and providing a data foundation for the subsequent analysis of overlapping positions, occluded area status, and unoccluded area status.

[0018] Second, this invention determines the feature ratio by comparing overlapping location features with the cluster centers of coal gangue features, classifies mixed states, and statistically analyzes the feature losses in three dimensions: color, texture, and shape. It then verifies the rationality of the mixed states using conditional probability, and finally clarifies the boundaries between occluded and unoccluded areas using feature loss thresholds. By differentiating the occluded and unoccluded areas at overlapping locations, unreasonable mixed states can be detected in a timely manner, improving the recognition accuracy for occluded areas.

[0019] Third, this invention uses consecutive frame images to filter unobstructed reference frames, performs grid fitting on obstructed areas and adjacent unobstructed areas, allocates feature loss according to grid area ratio and distance weight, corrects the feature values ​​of unobstructed grids, and completes the feature completion of obstructed grids by combining with reference frame templates and calibrating spatial similarity thresholds. By utilizing temporal information and spatial correlation to achieve feature complementarity, it fills the feature gaps in obstructed areas. At the same time, it correlates features with adjacent unobstructed areas to achieve the sum processing of obstructed and adjacent areas, making the features of obstructed areas complete. Finally, according to the supplemented feature values, each obstructed and unobstructed area is classified and labeled to complete the integration processing of features in each area. Attached Figure Description

[0020] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0021] Figure 1 This is a flowchart illustrating a method for classifying and identifying coal gangue based on image vision.

[0022] Figure 2 This is a flowchart illustrating step S1 of the image vision-based coal gangue classification and identification method.

[0023] Figure 3 This is a flowchart illustrating step S2 of the image vision-based coal gangue classification and identification method.

[0024] Figure 4 This is a flowchart illustrating step S3 of the image vision-based coal gangue classification and identification method.

[0025] Figure 5 This is a flowchart illustrating step S4 of the image vision-based coal gangue classification and identification method.

[0026] Figure 6 This is a system framework diagram of a coal gangue classification and recognition system based on image vision. Detailed Implementation

[0027] The embodiments of the present invention are described in detail below. The embodiments described below are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. Where specific techniques or conditions are not specified in the embodiments, they shall be performed in accordance with the techniques or conditions described in the literature in the art or in accordance with the product manual.

[0028] See Figure 1 A coal gangue classification and recognition method based on image vision includes: S1, using a camera array to acquire a mixed image of coal and coal gangue, and using the position coordinates of any element in the mixed image as a basis to divide multiple candidate regions.

[0029] S2, based on the overlapping positions between each candidate region, determines the mixing state of coal and gangue at the overlapping positions, quantifies the feature loss value of each candidate region under overlapping occlusion by the mixing state, and uses the feature loss to distinguish between occluded and unoccluded regions under gangue mixing.

[0030] S3. Based on the continuous frame images of the occluded region, perform grid fitting on the occluded region and the adjacent unoccluded region in the current frame, and synchronize the feature loss of the occluded region to the adjacent unoccluded region to determine the spatial similarity threshold of the adjacent unoccluded region.

[0031] S4 uses the spatial location of the occluded area for multi-target collaborative processing, and combines the spatial similarity threshold between adjacent unoccluded areas to unify the classification label of coal gangue in each candidate area.

[0032] S5 defines the regional characteristics of each candidate region based on the value range of the classification label, and uses the regional characteristics as synchronous output data to improve the subsequent coal gangue classification process.

[0033] like Figure 2 As shown, the implementation of step S1 includes: S11, aligning the acquired mixed images with timestamps according to the acquired time sequence to determine the position of the same element in the output each time it is acquired.

[0034] S12, according to the position of the same element, divide multiple regions in a grid-like form, and determine that each region contains at least one element; at this time, the contained element represents the identified coal or coal gangue. These grid-like regions are only used to initially locate the distribution location of coal or coal gangue. Subsequent candidate regions will be dynamically adjusted based on the location result and feature clustering to determine the mixed state identified in each region.

[0035] S13, after mapping the position of each element to the pixel coordinates of the mixed image, sets candidate regions for the identified coal or coal gangue.

[0036] When implementing step S1, the mixture of coal and coal gangue is identified by images collected by multiple cameras. This process can be performed before processes such as heavy media separation, jigging, and flotation during coal washing to avoid wear and tear on the equipment caused by coal gangue, or it can be performed after separation to identify 10-20mm coal gangue through secondary separation and complete its classification.

[0037] An array of 2-4 infrared + visible light dual-spectrum industrial cameras is used, deployed in parallel along the width of the transmission belt. The spacing is adjusted according to the width of the transmission belt. For example, when the bandwidth is 1.2m, the camera spacing is 0.6m to ensure that the field of view is completely stitched together and there are no blind spots in the acquisition.

[0038] The camera and conveyor belt encoder work together to synchronously acquire the mixed image and the physical location of the conveyor belt corresponding to each frame. After mapping the physical location to the pixel coordinates in the image, candidate regions are set, and the parts contained in each candidate region are determined to identify the features represented by the coal and coal gangue. At the same time, the identified coal or coal gangue is output one by one according to the sliding time window to determine the main features and classification of coal gangue in the current scene.

[0039] In step S1, some coal gangue has essentially been identified. Next, the features exhibited by these identified coal gangue will be used to construct a feature pool. Subsequently, the feature pool will be clustered, and candidate regions that can cover the corresponding features will be selected from the clustered content. The size of the candidate regions will be set, thereby completing the definition and processing of the mixed state of coal and coal gangue in the image.

[0040] Preferably, when setting the candidate region in step S13, the implementation method further includes: S131, retrieving the size, height and pixel value of the coal gangue during identification, and constructing the original feature pool of the coal gangue.

[0041] S132, the pixels in the original feature pool are used as data points, and clustering is performed according to the pixel value corresponding to each data point to generate a semantically segmented label map.

[0042] At this point, the K-means algorithm can be used to cluster based on features such as pixel values. For example, for three target categories—coal (dark black), gangue (grayish white), and background (soil color)—they are converted into pixel values. The Euclidean distance from each pixel to each cluster center is calculated, and the pixel is assigned to the cluster corresponding to the nearest center. The mean feature value of all pixels within a cluster is recalculated as the new cluster center. The process stops when the center point moves less than a threshold (e.g., 0.01) or reaches the maximum number of iterations (e.g., 100). This generates a label map corresponding to each pixel, which is labeled as coal, gangue, or background, forming the initial semantic segmentation region. Then, the silhouette coefficient of the current cluster is calculated, and the K value is chosen to maximize the average silhouette coefficient, thus obtaining the semantic segmentation result with the optimal number of clusters.

[0043] S133, Select the content in the label map that corresponds to coal or coal gangue, and aggregate the label map using a similarity metric to form preliminary candidate regions.

[0044] After obtaining the labels, a selective search method will be used to find the content related to coal and coal gangue in the image. These data will be merged one by one according to the similarity measure to form the preliminary candidate regions for hierarchical output.

[0045] The similarity metric is based on color similarity, texture similarity, and shape similarity to merge the labeled parts in the label image. At this point, the label image is mapped to the input mixed image to determine the color, texture, and shape features of each region in the label image during annotation. After obtaining the feature values ​​of these three aspects using feature description, the features of these three dimensions are calculated using cosine similarity, and the labeled parts in the label image at the corresponding positions are merged. At this point, the weighted values ​​of color similarity, texture similarity, and shape similarity are selected as the basis for judgment, and the merging is performed with weights of 0.4, 0.4, and 0.2. Since the feature values ​​of color, texture, and shape are different in different scenarios, a correlation matrix can be set using data uploaded in multiple batches in a specific scenario, and the weight of each similarity can be obtained by normalizing its feature values.

[0046] Meanwhile, the regions selected for merging will be based on a similarity threshold of 0.7, merging the corresponding label image parts. This similarity threshold can also be based on the percentage value at each image merging, such as the similarity value at 90% during normal merging, to merge images with similar colors, textures, and shapes to obtain multi-scale candidate regions.

[0047] It should be noted that during the merging process, the similarity will be merged step by step from high to low to complete the aggregation of the initial candidate regions.

[0048] S134: By obtaining the size constraints and overlap between the preliminary candidate regions, the preliminary candidate regions are filtered to determine the output candidate regions.

[0049] As for the size constraints of the initial candidate regions, the minimum and maximum areas of the regions are emphasized, and regions that are too large or too small are filtered and merged. The overlap will remove low-quality candidate regions that overlap highly to prevent duplicate identification of single coal gangue or coal.

[0050] Therefore, the implementation of step S134 also includes: performing size filtering on all preliminary candidate regions to filter out preliminary candidate regions that do not conform to the description of the label map.

[0051] The filtering section here targets candidate regions whose descriptions do not match, such as large pieces of gangue, finely crushed gangue, or a mixture of gangue and coal. This ensures that the labeled information corresponds to the aggregated portion of the initial candidate regions, allowing for the selection of regions that match the description. When dividing candidate regions, the size can be set by selecting the range of individual normal coal or gangue after sorting. For example, if the current image recognition is of a fragmented image, large pieces of gangue need to be removed before processes such as heavy media separation, jigging, and flotation. In this case, the candidate region size can be selected as 50-60mm. If the current image recognition is after the sorting process, then a size between 10-20mm can be selected to ensure that the candidate region includes at least one piece of gangue, thus completing the image size filtering.

[0052] For the filtered preliminary candidate regions, the overlap between the preliminary candidate regions is calculated, and the preliminary candidate regions are filtered using the coordinate values ​​mapped by each preliminary candidate region to obtain the output candidate regions.

[0053] At this point, it is necessary to remove two highly overlapping preliminary candidate regions. First, sort the preliminary candidate regions in descending order according to their confidence level (the probability value of detecting coal gangue). The confidence level is obtained by outputting the probability value of detecting coal gangue in the corresponding region when inputting an image recognition model such as YOLO7. Then, select the initial candidate region with the highest confidence level as the benchmark and remove all initial candidate regions with an intersection-union ratio greater than 0.7. In this way, the screening of all initial candidate regions is completed one by one.

[0054] As for selecting the coordinates of the initial candidate region, it involves calculating the offset of the candidate region under the alignment of multiple consecutive images to prevent the position of the initial candidate region from shifting when identifying a single coal gangue or coal during multiple tracking operations.

[0055] Offset identification is mainly achieved by mapping the center of the initial candidate region to coordinate values ​​and then using the Euclidean distance between the coordinate values ​​to check the magnitude of positional changes of the coal gangue and other content identified in the current image under continuous image recognition. The average distance of the center point in multiple images can be used to check the features such as coal gangue in the current initial candidate region. If the average distance does not change after introducing the current coal gangue and other content, it indicates that the corresponding initial candidate region has a consistent position in the continuous frame images, and it is used as the output candidate region.

[0056] In one embodiment of the present invention, in step S2, overlapping identification is performed on coal gangue within the same candidate area to determine the characteristic proportion belonging to coal gangue, and the mixed state is divided according to the proportion, such as pure coal area, pure gangue area and mixed area, thereby identifying the image processing of coal and coal gangue during sorting.

[0057] It should be noted that the candidate region represents the anchor frame for marking coal gangue or coal during image recognition. The layout of the region represents the size of the anchor frame set when recognizing the object. When overlapping positions occur, it means that the anchor frames of coal gangue and coal overlap during the recognition process. This usually indicates that the coal gangue and coal are distributed too close together and are in a relatively mixed and occluded state. These contents are easily ignored and misprocessed during image processing. It is necessary to record the position and situation of the overlap to complete the image recognition and processing.

[0058] Step S2, which identifies overlapping locations, differs from the overlapping location processing in step S134. Here, it is used to identify overlapping locations, that is, to select candidate regions with an intersection-union ratio (IU) greater than 0.3, and to determine the spatial competition between these candidate regions in image recognition. At this point, the occurrence of overlap indicates the interleaved distribution of coal and gangue. For example, overlaps with low IU ratios may cause feature distortion due to partial occlusion, such as coal becoming darker in color after being occluded by gangue, and gangue also becoming darker in color after being occluded. It is necessary to determine whether these situations will lead to recognition errors. Based on the features at these locations, the mixed state of coal and gangue is determined, thereby obtaining the most accurate identification of gangue while avoiding semantic errors.

[0059] In the current step, when selecting candidate regions with an intersection-union ratio greater than 0.3, they are used to identify regions with feature distortion caused by partial occlusion. In the aforementioned step S134, regions with an intersection-union ratio greater than 0.3 are used to filter duplicate candidate regions. The two have different usage scenarios to find candidate regions with feature distortion or repeated occlusion as much as possible.

[0060] like Figure 3As shown, the implementation of step S2 includes: S21, for each overlapping position, using the feature value at each overlapping position and the feature clustering center of coal gangue in the entire mixed image, the feature proportion belonging to coal gangue in the overlapping position is determined; at this time, based on the local features corresponding to the pixels at the overlapping position, these features are used to determine the mixing state of coal gangue and coal at each overlapping position according to the semantic segmentation method of coal gangue and coal. At this time, the proportion of coal gangue is the main factor. When the feature proportion of coal gangue is greater than 0.9, it means that the overlapping position is almost all coal gangue and can be divided into a pure gangue area. When its feature proportion is less than 0.1, it means that the proportion of coal gangue is small and can be divided into a pure coal area. As for the feature proportion between 0.1 and 0.9, it is considered as a mixed area. It is necessary to further check the feature loss that appears at this position to determine whether the feature loss is caused by occlusion, and then identify the occluded area and the non-occluded area.

[0061] S22, based on the characteristic proportions attributed to coal gangue, divides the mixed state of each overlapping position.

[0062] Preferably, when outputting candidate regions, multidimensional features corresponding to color, texture, and shape are recorded, and these features are normalized using RGB mean, texture contrast, and IR (feature value of infrared imaging) to obtain the features of each candidate region.

[0063] Then, the feature values ​​obtained at each overlapping position are fused with the global feature values ​​to distinguish the feature proportions belonging to coal gangue or coal at the overlapping positions.

[0064] like ;in, This represents the feature value of the i-th dimension after fusion, where i represents the index of the corresponding dimension, such as the dimension corresponding to RGB mean, texture contrast and IR. At this time, RGB mean can be divided into 3 dimensions, texture contrast can be divided into multiple dimensions, and the multiple bands of IR infrared features also represent multiple dimensions. When these three are processed together, they usually form a 20-100 dimension feature vector. This represents the local feature of the i-th dimension at a single overlapping location; This represents the global feature of the i-th dimension in the entire image. When the feature ratio of coal gangue is taken as the target, the feature value after clustering will be selected from the region marked as coal gangue in the entire mixed image. This represents the fusion weight, which is typically set to 0.7. This prioritizes retaining feature values ​​at overlapping locations while fusing feature values ​​of coal gangue from the global image.

[0065] Then, the cosine similarity between this fused feature value and the feature cluster centers of coal gangue in the entire image will be calculated, and the feature proportion belonging to coal gangue will be obtained from the calculated value. If coal clusters are to be identified more accurately, the cosine similarity between the fused feature value and the feature cluster centers of coal can also be calculated to determine whether the overlapping position belongs to pure coal, pure gangue, or a mixed zone.

[0066] The cluster center here is the cluster corresponding to the label image segmentation, and its cluster center is used as the feature ratio at the current overlapping position.

[0067] S23, based on the blending state of the current overlapping position, calculate the loss values ​​in the corresponding dimensions of color, texture and shape to obtain the feature loss of the corresponding overlapping position.

[0068] The current loss value can be set as a baseline value by using the feature values ​​of the corresponding dimensions of color, texture and shape as a global average value, and the feature loss of a single dimension can be set by the deviation ratio between the feature value of the current overlapping position in the corresponding dimension and the baseline value.

[0069] Then, based on the normalized values ​​of each feature dimension, the weights corresponding to each dimension are set, and the feature losses of each single dimension are weighted and summed to obtain the feature loss at the current overlapping position.

[0070] Preferably, step S23 is further implemented by: calculating the conditional probability of the candidate region in each mixed state, and verifying the rationality of each mixed state configuration by using the probability distribution of each candidate region.

[0071] At this point, the feature values ​​at the overlapping locations are mapped to high-dimensional and low-dimensional spaces. The feature loss generated at this time is quantified by the probability distribution of the high-dimensional and low-dimensional spaces to determine the possible recognition defects caused by overlapping occlusion.

[0072] At this point, the t-SNE dimensionality reduction algorithm can be used to reduce the current high-dimensional feature values ​​to 2D / 3D space, as shown below.

[0073] The preferred approach is to obtain the conditional probability in high-dimensional space. ;in, The conditional probability that the j-th candidate region in a high-dimensional space is considered a neighbor of the b-th candidate region. The larger this value is, the more similar the two candidate regions are in the high-dimensional feature space. and Represents a high-dimensional feature vector, which describes the feature values ​​of the b-th and j-th candidate regions at overlapping locations, respectively; The bandwidth parameter represents the Gaussian kernel and controls the neighborhood size. It is set based on the description of the current mixing state. For example, if the features of a pure coal area are dense, the bandwidth parameter value is small. If it is a mixed area with dispersed features, a larger value is set. It is usually set based on 5%-10% of the current number of samples, that is, based on the number of overlapping positions identified under the same mixing state, and set according to its standardized value. This represents an exponential function with base e; Let represent the feature vector of the k-th candidate region in the high-dimensional feature space. The process of obtaining the denominator is used for normalization to ensure that the sum of all conditional probabilities is 1. The numerator maps the distance between different features to similarity, thereby clarifying the conditional probability of the current candidate region under adjacent overlap.

[0074] Then obtain the probability values ​​in the lower-dimensional space. ;in, Let represent the conditional probability that the j-th candidate region in the low-dimensional space is considered a neighbor of the b-th candidate region; and Represents a low-dimensional feature vector, which describes the feature values ​​of the b-th and j-th candidate regions at overlapping locations, respectively; This represents the eigenvector of the k-th candidate region in the low-dimensional feature space. In this case, the t-distribution is chosen instead of the Gaussian kernel in the numerator term of the low-dimensional space to ensure that the coal and gangue mixture is not over-compressed into pure coal or pure gangue regions. The denominator term is also used for normalization to complete the overall processing.

[0075] Finally, the KL divergence method was used to verify its value, such as ;in, The value of KL divergence directly reflects the fidelity of the high-dimensional feature space structure in the low-dimensional representation. For example, when the KL divergence value is less than 2, it means that more than 90% of the local proximity relationships are preserved, and the pure coal area, mixed area and gangue area are clearly represented regions, which can intuitively determine the mixing state of coal and coal gangue.

[0076] When the KL divergence is between 2 and 5, 70%-90% of the local structure is preserved, pure coal and pure gangue are basically separated, and some mixed areas overlap at the boundary. When the KL divergence is between 5 and 10, 50%-70% of the local structure is preserved. At this time, pure coal and pure gangue overlap in some locations, and the boundary between the mixed area and other areas is blurred, with obvious abnormal aggregation. As for when the KL divergence is greater than 10, it means that less than 50% of the local structure is preserved. In the current mixed state, the various categories are over-mixed, making it difficult to distinguish the coal and gangue accurately. At this time, the candidate areas in each mixed state will be checked based on the probability value recorded in this scenario to indicate whether the current candidate area has been identified, and the corresponding data will be used as a supplement to the data.

[0077] S24, in the overlapping positions, the part with feature loss greater than the threshold is regarded as the occluded area, and the remaining non-overlapping or overlapping positions with loss not exceeding the threshold are regarded as the unoccluded area, and the label of its mixed state is retained.

[0078] In the overlapping region, the feature loss caused by occlusion is calculated, such as the difference in color histogram and texture abrupt changes. Regions with losses exceeding a threshold are marked as occluded regions, thereby determining the inaccuracy of feature description at the overlapping location under occlusion.

[0079] In the overlapping region, based on the value of the feature loss, a value of 0.2 is selected as the current threshold, or the average value of the feature loss when coal gangue is mixed in historical data is selected as the current threshold. The part greater than the threshold is selected as the occlusion region, and the remaining region retains its mixing state and the label in the label map to explain the result of the current image segmentation.

[0080] In one embodiment of the present invention, the purpose of step S3 is to transfer the feature loss of the occluded area to the adjacent unoccluded area, complete the missing information of the occluded area by feature association of the unoccluded area, and quantify the spatial similarity between the unoccluded areas, thereby explaining the occlusion situation when coal gangue is mixed and distributed.

[0081] like Figure 4 As shown, the implementation of step S3 includes: S31, calling the continuous frame images corresponding to the occluded area, determining the position of the same coal gangue target in the continuous frame images based on the position coordinates of the occluded area and the movement trajectory of the coal gangue, and filtering out reference frames that contain the coal gangue target and are in an unoccluded area.

[0082] At this point, based on the speed of the conveyor belt, continuous frame images will be extracted from the camera array, and images of the unobstructed area corresponding to the current coal gangue target will be found, with the corresponding frame being regarded as a reference frame.

[0083] The reasons why the same coal gangue target can be captured in both obscured and unobscured states are roughly as follows: 1. During continuous frame image acquisition, some frames are blurred due to conveyor belt movement, making it impossible to accurately identify the boundary between coal and gangue; 2. Because the cameras are arranged in an array, their shooting angle changes with the conveyor belt movement, resulting in different visual depths for identifying the same coal gangue from different angles, thus causing two different recognition states in multiple consecutive frames; 3. Due to uneven lighting at the camera locations, some features are still missing after image processing following lighting calibration, leading to subtle differences between one frame and other frames, resulting in two states: obscured and unobscured areas. Besides the reasons described above, vibration or other movement of some coal during conveyor belt transport may also cause mixing and overlapping of crushed coal and gangue.

[0084] At this point, the current frame is an image with missing features due to image blur, shooting angle differences, uneven lighting, or coal breakage and mixing. The reference frame is a clearly acquired image. The values ​​in the reference frame are used to correct the values ​​in the current frame.

[0085] S32, perform distance calculation between the unobstructed area and the obstructed area of ​​the current frame to determine the adjacent unobstructed area of ​​the current frame.

[0086] At this point, the unobstructed and obstructed areas corresponding to the current coal gangue target are determined using pixel space or physical coordinates. Adjacent unobstructed areas are identified, for example, if the pixel distance between two areas is 20 pixels. Then, the obstructed and unobstructed areas are divided into multiple grids. For example, the obstructed area is divided into an 8×8 pixel grid, and the unobstructed area is divided into grids of the same size as the obstructed area grid, based on their adjacent positions. Each grid corresponds to a unique ID, and the calculated feature loss is allocated to each grid. The distance value for determining adjacent unobstructed areas will be adaptively adjusted according to different conveyor belt configurations. For example, when there is mixed coverage of crushed coal, an unobstructed area with a distance of 30 pixels can be selected as the current processing part.

[0087] S33, divide the occluded region and the adjacent unoccluded region according to the grid, and obtain multiple occluded grids and adjacent unoccluded grids in sequence. Based on the grid area ratio and distance weight, the feature loss of the occluded region is assigned to each occluded grid, and the feature loss of the occluded grid is used to correct the feature value of the adjacent unoccluded grid.

[0088] The feature loss assigned to each occluded grid at this point can be expressed as follows.

[0089] ;in, This represents the feature loss of the occluded area. This represents the feature loss for each occluded grid. This represents the area of ​​each occlusion grid. Indicates the area of ​​the obstructed region. This represents the distance weight, which is set by the distance between the adjacent unoccluded region and the current occluded region. The reciprocal of this distance value is used as the weight to determine how the feature loss of the current occluded region is diffused to the corresponding adjacent unoccluded region.

[0090] The feature loss of the occluded grid is then synchronized to the adjacent unoccluded grids. The feature value of the adjacent unoccluded grids is then reduced by the product of the feature loss and the occlusion coefficient to obtain the updated feature value of each adjacent unoccluded grid. At this time, the occlusion coefficient is set to a constant value according to the cause of the occlusion area, such as 0.3 for normal occlusion and 0.2 for coal crushing and mixed coverage, to indicate the reason for the current unclear identification.

[0091] S34, call the reference frame as the completion template, combine the feature values ​​corrected by the adjacent unoccluded grids, perform feature completion for each occluded grid, and calibrate the spatial similarity threshold of all adjacent unoccluded grids after completion.

[0092] When performing feature completion, the occluded grid position that needs to be corrected will be (m, n), where m represents its row index and n represents its column index. The four grids corresponding to the occluded grid (up, down, left, right) will be found from the adjacent unoccluded grids. If there are more than four grids corresponding to the occluded grid (up, down, left, right), the feature value of the reference frame at the corresponding position will be selected to complete the feature completion of each grid.

[0093] Next, bilinear interpolation is used to calculate the distance between the center points of the current occluded grid and its corresponding adjacent unoccluded grid. The distance weight is set using the reciprocal of the distance value, and the feature loss weight is set according to the feature loss of the corresponding occluded grid. The weights of the four grids adjacent to the current occluded grid are then obtained by multiplying the distance weights and the feature loss weights. After normalizing the four grid weights, the row median value is extracted by multiplying the corrected feature values ​​of the upper and lower grids by their respective grid weights. The same method is used to obtain the column median value. Finally, the feature value at the same position in the reference frame is used for correction to obtain the corrected value for the occluded grid, thus completing the processing for overlapping positions.

[0094] For example ;in, This represents the feature value after the occluded mesh is completed; This represents the feature value of the reference frame at the corresponding position. This represents the middle value of the row corresponding to the occluded grid. This represents the middle value of the column corresponding to the occluded grid. This represents the feature weight, used to set weight values ​​for different forms of occlusion. For example, 0.7 is used for regular occlusion, 0.8 for mixed coal crushing and covering, and 0.75 for shooting angle difference. These values ​​emphasize feature repair at the corresponding locations to obtain more accurate coal gangue features for classification. The calculation process here iteratively optimizes the feature values ​​of the occluded grid without changing the feature values ​​of adjacent unoccluded areas. Finally, the corrected feature values ​​are assigned to the original feature positions of the occluded grid, completing the feature completion for a single grid.

[0095] As for the spatial similarity threshold, it is determined by judging all adjacent unoccluded grids in the current frame, converting these grids into vectors containing coordinates, and using the average cosine similarity between all adjacent unoccluded grids as the output spatial similarity threshold. The spatial similarity threshold provides a calculation basis for feature completion of occluded grids, such as the spatial similarity of unoccluded areas under conditions of normal occlusion, coal crushing and mixed coverage, to illustrate the similarity under different occlusion conditions and provide a criterion for subsequent unified classification labels.

[0096] If the coal gangue target in the current occluded area does not have a corresponding unoccluded area in consecutive frame images, it means that the mixing state of this part of the coal gangue and coal is unclear throughout the entire conveyor belt transmission process. It is necessary to use a robotic arm or other devices to process the coal gangue at this location. As for the method of using feature transfer between occluded and unoccluded areas, it is to supplement the images and features for identifying the mixing state of coal gangue and coal, and improve the accuracy of subsequent processing for the mixing of fine coal gangue and coal.

[0097] In one embodiment of the present invention, the multi-target collaborative processing in step S4 emphasizes multi-target path tracking. After tracking, the positions of each target are displayed according to the overall path to highlight the classification of the overall tracking process, thereby reducing jitter in data analysis and improving the accuracy of coal gangue classification label setting.

[0098] like Figure 5 As shown, the implementation of step S4 includes: S41, extracting the spatial location of the occluded area and dividing it into multiple sorting paths according to the location of the conveyor belt.

[0099] S42, combining the feature values ​​of the occluded areas on each sorting path with the spatial similarity threshold between adjacent unoccluded areas, set classification labels for each sorting path; such as the label combination of pure coal / coal gangue mixture / pure gangue + large / medium / fine fragments, to define the identification status of coal gangue on each sorting path.

[0100] Since the processing of the unclear parts in the occluded area is completed in step S3, the currently set classification label will be based on its corrected feature value and the spatial similarity threshold between its feature value and the adjacent unoccluded area to set a label for the location of each occluded area; the label corresponding to the spatial similarity threshold will represent the feature mean of the unoccluded area in the adjacent state, and its value will represent the supplementary description in the case of pure coal / coal gangue mixture / pure gangue, and will be added to the classification label to describe the corresponding area.

[0101] The method of setting labels is the same as the aforementioned method of identifying pure coal / coal gangue mixture / pure gangue, and according to the size of the identified material, it is labeled with a description of large / medium / fine fragments to complete the data unification on each path.

[0102] As for the sorting path, this path represents the main part that the robotic arm will process in the subsequent work. Based on the position of each robotic arm, its conveyor belt will be divided into multiple long strip paths. If classification labels are set for the sorting path, the Hungarian algorithm can be used to unify the occlusion area under the same classification label description and obtain the specific location of the coal gangue distribution to complete the integration processing of coal gangue.

[0103] When used, the Hungarian algorithm solves the problem of constructing sorting paths under the same classification label. That is, starting from the coal gangue recorded as the same classification label, it finds the shortest connecting path under the same classification label, with the spatial distance between each labeled coal gangue as the cost. This path is the final output sorting path, which describes the distribution of coal gangue on the conveyor belt when the robotic arm is working.

[0104] Step S5 is only used to summarize the classification labels and related data, and to define the regional features identified under each image, so as to improve the accuracy of subsequent coal gangue identification and classification process.

[0105] Therefore, the implementation of step S5 also includes: obtaining the classification labels of each candidate region under multiple iterations, and labeling the candidate regions with the corresponding occluded and unoccluded regions, and using the combined data as the output region features.

[0106] The output data obtained at this time will be labeled in the form of large pieces of gangue, medium pieces of gangue, and gangue powder, etc., and after labeling each candidate region in the form of occluded and unoccluded areas, the region features are used as part of the synchronous output.

[0107] like Figure 6As shown, in one embodiment of the present invention, the present invention also provides a coal gangue classification and identification system based on image vision, including: a region division module, an occlusion recognition module, a feature synchronization module, a classification labeling module, and a data output module; wherein, the output end of the region division module is connected to the occlusion recognition module, the output end of the occlusion recognition module is connected to the feature synchronization module, the output end of the feature synchronization module is connected to the classification labeling module, and the output end of the classification labeling module is connected to the data output module.

[0108] The region segmentation module is used to acquire a mixed image of coal and coal gangue using a camera array, and to divide multiple candidate regions based on the position coordinates of any element in the mixed image.

[0109] The occlusion recognition module is used to determine the mixing state of coal and gangue at the overlapping positions based on the overlapping positions between each candidate region. It quantifies the feature loss value of each candidate region under overlapping occlusion according to the mixing state, and uses the feature loss to distinguish between occluded and unoccluded regions under the mixing of coal and gangue.

[0110] The feature synchronization module is used to perform grid fitting on the occluded region and the adjacent unoccluded region in the current frame based on the continuous frame images of the occluded region, and synchronize the feature loss of the occluded region to the adjacent unoccluded region to determine the spatial similarity threshold of the adjacent unoccluded region.

[0111] The classification and labeling module is used for multi-target collaborative processing based on the spatial location of occluded areas. It combines the spatial similarity threshold between adjacent unoccluded areas to unify the classification labels of coal gangue in each candidate area.

[0112] The data output module is used to define the regional characteristics of each candidate region based on the value range of the classification label, and to output the regional characteristics as data in a synchronous manner to improve the subsequent coal gangue classification process.

[0113] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention, which are still covered within the protection scope of the present invention.

Claims

1. A method for classifying and identifying coal gangue based on image vision, characterized in that, include: S1. A camera array is used to acquire a mixed image of coal and coal gangue. Based on the position coordinates of any element in the mixed image, multiple candidate regions are divided. S2, based on the overlapping positions between each candidate region, determine the mixing state of coal and gangue at the overlapping positions, quantify the feature loss value of each candidate region under overlapping occlusion according to the mixing state, and use the feature loss to distinguish between occluded and unoccluded regions under gangue mixing. S3, based on the continuous frame images of the occluded area, perform grid fitting on the occluded area and the adjacent unoccluded area in the current frame, and synchronize the feature loss of the occluded area to the adjacent unoccluded area to determine the spatial similarity threshold of the adjacent unoccluded area. S4 uses the spatial location of the occluded area to perform multi-target collaborative processing, and combines the spatial similarity threshold between adjacent unoccluded areas to unify the classification label of coal gangue in each candidate area. S5 defines the regional characteristics of each candidate region based on the range of values ​​for the classification labels, and uses the regional characteristics as synchronous output data.

2. The image-based visual classification and identification method for coal gangue according to claim 1, characterized in that, The implementation methods for step S1 include: S11, The acquired mixed images are aligned with timestamps according to the acquired time sequence to determine the position of the same element in the output each time it is acquired; S12, divide the area into multiple regions in a grid-like manner according to the position of the same element, and determine that each region contains at least one element; S13, after mapping the position of each element to the pixel coordinates of the mixed image, sets candidate regions for the identified coal or coal gangue.

3. The image-based visual classification and identification method for coal gangue according to claim 2, characterized in that, When setting the candidate region in step S13, the implementation method also includes: S131, retrieve the size, height, and pixel values ​​of the coal gangue during identification, and construct the original feature pool of the coal gangue; S132, the pixels in the original feature pool are used as data points, and clustering is performed according to the pixel value corresponding to each data point to generate a semantically segmented label map; S133, Select the content in the label map that corresponds to coal or coal gangue, and aggregate the label map using a similarity metric to form preliminary candidate regions; S134: By obtaining the size constraints and overlap between the preliminary candidate regions, the preliminary candidate regions are filtered to determine the output candidate regions.

4. The image vision-based coal gangue classification and identification method according to claim 3, characterized in that, The implementation of step S134 also includes: All preliminary candidate regions are size-filtered to filter out those that do not conform to the description of the label map. For the filtered preliminary candidate regions, the overlap between the preliminary candidate regions is calculated, and the preliminary candidate regions are filtered using the coordinate values ​​mapped by each preliminary candidate region to obtain the output candidate regions.

5. The image-based visual classification and identification method for coal gangue according to claim 1, characterized in that, Step S2 can be implemented in the following ways: S21, For each overlapping location, the feature proportion belonging to coal gangue in the overlapping location is determined by the feature value at each overlapping location and the feature cluster center of coal gangue in the entire mixed image. S22, based on the characteristic proportions attributed to coal gangue, divide the mixed state of each overlapping position; S23, Based on the mixing state of the current overlapping position, calculate the loss values ​​in the corresponding dimensions of color, texture and shape to obtain the feature loss of the corresponding overlapping position; S24, in the overlapping positions, the part with feature loss greater than the threshold is regarded as the occluded area, and the remaining non-overlapping or overlapping positions with loss not exceeding the threshold are regarded as the unoccluded area, and the label of its mixed state is retained.

6. The image vision-based coal gangue classification and identification method according to claim 5, characterized in that, The implementation of step S23 also includes: Calculate the conditional probability of the candidate region under each mixed state, and verify the rationality of each mixed state configuration using the probability distribution of each candidate region.

7. The image vision-based coal gangue classification and identification method according to claim 1, characterized in that, Step S3 can be implemented in the following ways: S31, call the continuous frame images corresponding to the occluded area, determine the position of the same coal gangue target in the continuous frame images based on the position coordinates of the occluded area and the movement trajectory of the coal gangue, and filter out the reference frames that contain the coal gangue target and are in the unoccluded area state. S32, perform distance calculation between the unoccluded area and the occluded area of ​​the current frame to determine the adjacent unoccluded area of ​​the current frame; S33, divide the occluded region and the adjacent unoccluded region according to the grid, and obtain multiple occluded grids and adjacent unoccluded grids in turn. Based on the grid area ratio and distance weight, the feature loss of the occluded region is allocated to each occluded grid, and the feature loss of the occluded grid is used to correct the feature value of the adjacent unoccluded grid. S34, call the reference frame as the completion template, combine the feature values ​​corrected by the adjacent unoccluded grids, perform feature completion for each occluded grid, and calibrate the spatial similarity threshold of all adjacent unoccluded grids after completion.

8. The image-based vision-based coal gangue classification and identification method according to claim 1, characterized in that, Step S4 can be implemented in the following ways: S41, extract the spatial location of the obstructed area and divide it into multiple sorting paths according to the location of the conveyor belt; S42, combine the feature values ​​of the occluded areas on each sorting path with the spatial similarity threshold between adjacent unoccluded areas, and set a classification label for each sorting path.

9. The image-based visual classification and identification method for coal gangue according to claim 1, characterized in that, The implementation of step S5 also includes: Obtain the classification labels of each candidate region under multiple iterations, and label the candidate regions with the corresponding occluded and unoccluded regions. Use the combined data as the output region features.

10. A coal gangue classification and identification system based on image vision, characterized in that, include: The region segmentation module is used to acquire a mixed image of coal and coal gangue using a camera array, and to divide multiple candidate regions based on the position coordinates of any element in the mixed image. The occlusion recognition module is used to determine the mixing state of coal and gangue at the overlapping position based on the overlapping position between each candidate region, quantify the feature loss value of each candidate region under overlapping occlusion according to the mixing state, and use the feature loss to distinguish between occluded and unoccluded regions under the mixing of gangue. The feature synchronization module is used to perform grid fitting on the occluded region and the adjacent unoccluded region in the current frame based on the continuous frame image of the occluded region, and synchronize the feature loss of the occluded region to the adjacent unoccluded region to determine the spatial similarity threshold of the adjacent unoccluded region. The classification and labeling module is used to perform multi-target collaborative processing based on the spatial location of the occluded area, and to unify the classification label of coal gangue in each candidate area by combining the spatial similarity threshold between adjacent unoccluded areas. The data output module is used to define the regional characteristics of each candidate region based on the value range of the classification label, and to output the regional characteristics as data synchronously.

Images