A visible light image-based coal belt foreign matter detection method and system

Abstract

The application relates to the technical field of image recognition, and discloses a visible light image-based coal conveying belt foreign matter detection method and system, the method comprising the following steps: smoothing and denoising a visible light image of a coal conveying belt operation area to obtain a standard image, obtaining a contour enhancement and texture suppression image through multi-stage enhancement, fusing the contour enhancement and texture suppression image to obtain a fused feature, performing semantic segmentation on the standard image to obtain a segmentation mask, obtaining a three-dimensional point cloud through binocular vision and counting projection to obtain a region surface normal distribution image, coupling and evaluating the fused feature, the mask shape feature and the normal distribution image geometric feature, and generating a comprehensive foreign matter detection report of the coal conveying belt operation area; and the application can improve the efficiency of the visible light image-based coal conveying belt foreign matter detection.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method and system for detecting foreign objects on coal conveyor belts based on visible light images. Background Technology

[0002] In the field of foreign object detection on coal conveyor belts, traditional detection methods based on visible light images have significant shortcomings in the image preprocessing stage. Traditional smoothing and denoising techniques struggle to retain key information about belt edges and potential foreign objects while suppressing noise, easily leading to the loss of effective features. Meanwhile, single-mode image enhancement methods cannot specifically optimize the contrast between the foreign object outline and the belt texture, making it difficult for the preprocessed image to provide a clear and reliable detection basis, directly affecting the effectiveness of subsequent detection steps.

[0003] Existing detection technologies suffer from shortcomings in the synergy between feature extraction and foreign object identification, lacking deep fusion and comprehensive evaluation of multi-dimensional features. Traditional methods often rely on single features for foreign object detection, failing to fully integrate image contour features, texture features, and spatial geometric features, resulting in limited accuracy in semantic segmentation and insufficient utilization of three-dimensional spatial information. This makes it difficult to distinguish foreign objects from conveyor belts and background areas, ultimately leading to frequent missed and false detections during the detection process. The detection efficiency and accuracy cannot meet the actual needs of continuous and efficient operation of coal conveying systems, making it difficult to effectively avoid equipment failures and safety risks caused by foreign objects. Summary of the Invention

[0004] This invention provides a method and system for detecting foreign objects on coal conveyor belts based on visible light images, in order to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides a method for detecting foreign objects in a coal conveyor belt based on visible light images, comprising:

[0006] S1. Smooth and denoise the visible light image of the coal conveyor belt operating area to obtain a standard belt area image of the coal conveyor belt operating area;

[0007] S2. Perform multi-level image enhancement on the standard belt area image to obtain the contour enhancement image and texture suppression image of the coal conveyor belt operating area;

[0008] S3. The contour enhancement image and the texture suppression image are synergistically enhanced and fused to obtain the fusion features of the coal conveyor belt operating area. Based on the fusion features, the standard belt area image is semantically segmented and quantized to obtain the segmentation mask of the standard belt area image.

[0009] S4. Based on the segmentation mask, perform binocular visual perception on the standard belt area image to obtain the three-dimensional point cloud data of the coal conveyor belt operating area;

[0010] S5. Perform statistical projection on the spatial distribution characteristics of surface normals in the three-dimensional point cloud data to obtain the regional surface normal distribution map of the coal conveyor belt operating area;

[0011] S6. Perform a foreign object coupling evaluation on the fusion feature, the shape feature of the segmentation mask and the geometric feature of the surface normal distribution map of the region to obtain a comprehensive foreign object detection report for the coal conveyor belt operating area.

[0012] In a preferred embodiment, smoothing and denoising the visible light image of the coal conveyor belt operating area to obtain a standard belt area image of the coal conveyor belt operating area includes:

[0013] The visible light images of the coal conveyor belt operating area are used as the raw image data.

[0014] Gaussian filtering is applied to the original image data to obtain an intermediate filtered image of the coal conveyor belt operating area;

[0015] Edge detection is performed on the intermediate filtered image to obtain the edge retention mask of the coal conveyor belt operating area;

[0016] Based on the edge-preserving mask, the parameters of the Gaussian filtering operation are adaptively adjusted to perform secondary filtering processing, thereby obtaining a standard belt area image of the coal conveyor belt operating area.

[0017] In a preferred embodiment, the step of performing multi-level image enhancement on the standard belt area image to obtain an enhanced contour image and a texture-suppressed image of the coal conveyor belt operating area includes:

[0018] Brightness contrast adjustment is performed on the standard belt area image to obtain a brightness-balanced image of the coal conveyor belt operating area;

[0019] Applying the Laplacian edge enhancement operator to the brightness equalization image yields the contour enhancement image of the coal conveyor belt operating area;

[0020] Multi-directional texture analysis is performed on the brightness equalization image to generate a texture suppression image of the coal conveyor belt operating area.

[0021] In a preferred embodiment, the step of co-enhancing and fusing the contour-enhanced image and the texture-suppressed image to obtain the fused features of the coal conveyor belt operating area, and then performing semantic segmentation and quantization on the standard belt area image based on the fused features to obtain a segmentation mask for the standard belt area image, includes:

[0022] Based on historical foreign object image detection records of the coal conveyor belt operating area, and combined with an attention mechanism, the contour enhancement image and the texture suppression image are learned and mapped to construct a fusion weight generation network for the coal conveyor belt operating area.

[0023] The contour enhancement image and the texture suppression image are input into the fusion weight generation network to obtain a pixel-level contour enhancement weight map of the contour enhancement image and a pixel-level texture suppression weight map of the texture suppression image.

[0024] Based on the pixel-level contour enhancement weight map and the pixel-level texture suppression weight map, the contour enhancement image and the texture suppression image are weighted and quantized respectively to obtain the contour contribution component of the contour enhancement image and the texture contribution component of the texture suppression image.

[0025] The contour contribution component and the texture contribution component are superimposed pixel by pixel to obtain the fused feature map of the coal conveyor belt operating area.

[0026] Based on the fused feature map, a semantic labeling system is constructed for the belt area, background area and the area to be identified in the coal conveyor belt operating area, and pixel-level semantic classification is performed on the standard belt area image to obtain the initial semantic segmentation result of the standard belt area image;

[0027] Spatial context optimization is performed on the initial semantic segmentation result to obtain the segmentation mask of the standard belt region image.

[0028] In a preferred embodiment, the step of constructing a semantic labeling system for the belt region, background region, and region to be identified in the coal conveyor belt operating area based on the fused feature map, and performing pixel-level semantic classification on the standard belt region image to obtain the initial semantic segmentation result of the standard belt region image, includes:

[0029] Perform multi-scale convolution operations on the fused feature map to obtain the context feature map of the coal conveyor belt operating area;

[0030] Based on the context feature map, a semantic label category set is defined for the coal conveyor belt operating area. The semantic label category set includes categories corresponding to the belt area, categories corresponding to the background area, and categories corresponding to the area of ​​foreign objects to be identified.

[0031] The context feature map is input into the pixel-level classification network in the fusion weight generation network to obtain the probability vector of the semantic label category set;

[0032] Based on the pixel positions of the fused feature map, the probability vector is numerically filtered to generate an initial semantic label map of the coal conveyor belt operating area;

[0033] Connectivity analysis is performed on the initial semantic label map, and based on the analysis results, spatially adjacent pixel regions with the same label in the coal conveyor belt operating area are merged to obtain the initial semantic segmentation result of the standard belt area image.

[0034] In a preferred embodiment, the step of performing binocular visual perception on the standard belt area image based on the segmentation mask to obtain three-dimensional point cloud data of the coal conveyor belt operating area includes:

[0035] Based on the preset binocular camera parameters, stereo correction is performed on the standard belt area image and the corresponding synchronous image from another perspective to obtain a stereo image pair of the coal conveyor belt operating area;

[0036] Based on the segmentation mask, the matching region image pair corresponding to the coal conveyor belt operating area is extracted from the stereo image pair;

[0037] Dense stereo matching is performed on the image pairs of the regions to be matched to obtain the initial disparity map of the coal conveyor belt operating area;

[0038] Based on the binocular camera parameters and the initial disparity map, triangulation calculations are performed on the coal conveyor belt operating area to obtain the initial three-dimensional spatial point set of the coal conveyor belt operating area.

[0039] Outlier filtering and noise smoothing are performed on the initial three-dimensional spatial point set to obtain the three-dimensional point cloud data of the coal conveyor belt operating area.

[0040] In a preferred embodiment, the calculation formula for the triangulation is as follows:

[0041] ;

[0042] In the formula, The pixel coordinates of the initial disparity map. The target three-dimensional spatial coordinates of the coal conveyor belt operating area are: pixel coordinates The weighted projection matrix, The principal point coordinates of the binocular camera parameters are... For calculating the disparity in the dense stereo matching, The binocular camera baseline distance is the parameter of the binocular camera. The equivalent focal length of the binocular camera is the parameter of the camera.

[0043] In a preferred embodiment, the step of statistically projecting the spatial distribution characteristics of surface normals in the three-dimensional point cloud data to obtain a regional surface normal distribution map of the coal conveyor belt operating area includes:

[0044] Surface normals are estimated from the three-dimensional point cloud data to obtain the three-dimensional normal field of the three-dimensional point cloud data;

[0045] The normal vectors in the three-dimensional normal field are oriented and normalized to obtain the normal direction feature map of the coal conveyor belt operating area.

[0046] The three-dimensional point cloud data of the coal conveyor belt operating area is projected onto a two-dimensional plane, and the mapping relationship between the normal direction feature map and the two-dimensional projection grid in the two-dimensional plane is established according to the projection relationship.

[0047] Based on the mapping relationship, the normal direction features of the normal direction feature map are mapped to the independent grids of the two-dimensional projection grid to perform statistical histograms, thereby obtaining the normal distribution descriptor of the normal direction feature map;

[0048] The normal distribution descriptor is visualized and fused according to the spatial order of the two-dimensional projection grid to obtain the regional surface normal distribution map of the coal conveyor belt operating area.

[0049] In a preferred embodiment, the foreign object coupling evaluation of the fusion feature, the shape feature of the segmentation mask, and the geometric feature of the regional surface normal distribution map to obtain a comprehensive foreign object detection report for the coal conveyor belt operating area includes:

[0050] Spatial feature analysis is performed on the connected components of the region to be identified in the segmentation mask to obtain the shape features of the connected components;

[0051] Based on the spatial location of the connected domain, the attribute features of the surface normal distribution map of the region are analyzed to obtain the geometric features of the coal conveyor belt operating area.

[0052] The fused features, shape features, and geometric features are concatenated to construct a comprehensive description vector of the region to be identified.

[0053] Based on the comprehensive description vector, foreign object detection is performed on the area to be identified to obtain a comprehensive foreign object detection report for the coal conveyor belt operating area.

[0054] To address the aforementioned problems, the present invention also provides a foreign object detection system for coal conveyor belts based on visible light images, the system comprising:

[0055] The image smoothing and denoising module is used to smooth and denoise the visible light image of the coal conveyor belt operating area to obtain a standard belt area image of the coal conveyor belt operating area.

[0056] A multi-level image enhancement module is used to perform multi-level image enhancement on the standard belt area image to obtain an enhanced contour image and a texture-suppressed image of the coal conveyor belt operating area.

[0057] The collaborative enhancement fusion and semantic segmentation module is used to perform collaborative enhancement fusion of the contour enhancement image and the texture suppression image to obtain the fusion features of the coal conveyor belt operating area, and based on the fusion features, to perform semantic segmentation quantization on the standard belt area image to obtain the segmentation mask of the standard belt area image;

[0058] A binocular vision perception module is used to perform binocular vision perception on the standard belt area image based on the segmentation mask, so as to obtain the three-dimensional point cloud data of the coal conveyor belt operating area.

[0059] The surface normal distribution statistics module is used to statistically project the spatial distribution characteristics of surface normals in the three-dimensional point cloud data to obtain the regional surface normal distribution map of the coal conveyor belt operating area.

[0060] The foreign object coupling assessment module is used to perform foreign object coupling assessment on the fusion feature, the shape feature of the segmentation mask and the geometric feature of the surface normal distribution map of the region, so as to obtain a comprehensive foreign object detection report of the coal conveyor belt operating area.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] 1. This invention uses adaptive smoothing denoising and multi-level image enhancement technology to suppress image noise while fully preserving the key edge information of coal conveyor belts and potential foreign objects. It then combines an attention mechanism to construct a fusion weight generation network, achieving pixel-level accurate fusion of contour enhancement features and texture suppression features. This significantly improves the extraction quality and completeness of foreign object-related features, laying a highly reliable foundation for subsequent detection processes.

[0063] 2. This invention relies on the construction of a semantic tagging system and spatial context optimization to achieve accurate semantic segmentation of the belt area, background area, and area to be identified. Through binocular visual perception and surface normal distribution statistics, it fully explores the three-dimensional spatial geometric features. Then, through the coupled evaluation of fusion features, shape features, and geometric features, it significantly improves the accuracy and comprehensiveness of foreign object detection. It can efficiently generate detailed comprehensive foreign object detection reports, providing strong support for the safe and stable operation of coal conveyor belts. Attached Figure Description

[0064] Figure 1 This is a flowchart illustrating a foreign object detection method for coal conveyor belts based on visible light images, provided in an embodiment of the present invention.

[0065] Figure 2 This is a functional block diagram of a foreign object detection system for coal conveyor belts based on visible light images, provided in an embodiment of the present invention.

[0066] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0067] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0068] This application provides a method for detecting foreign objects on a coal conveyor belt based on visible light images. The executing entity of this method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the method for detecting foreign objects on a coal conveyor belt based on visible light images can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0069] Reference Figure 1 The diagram shown is a flowchart illustrating a foreign object detection method for coal conveyor belts based on visible light images, according to an embodiment of the present invention. In this embodiment, the foreign object detection method for coal conveyor belts based on visible light images includes:

[0070] S1. Smooth and denoise the visible light image of the coal conveyor belt operating area to obtain a standard belt area image of the coal conveyor belt operating area;

[0071] In this embodiment of the invention, the step of smoothing and denoising the visible light image of the coal conveyor belt operating area to obtain a standard belt area image of the coal conveyor belt operating area includes:

[0072] The visible light images of the coal conveyor belt operating area are used as the raw image data.

[0073] Gaussian filtering is applied to the original image data to obtain an intermediate filtered image of the coal conveyor belt operating area;

[0074] Edge detection is performed on the intermediate filtered image to obtain the edge retention mask of the coal conveyor belt operating area;

[0075] Based on the edge-preserving mask, the parameters of the Gaussian filtering operation are adaptively adjusted to perform secondary filtering processing, thereby obtaining a standard belt area image of the coal conveyor belt operating area.

[0076] The visible light image collected in real time within the operating area of ​​the coal conveyor belt is received. This image is a two-dimensional pixel array containing the coal conveyor belt itself, the surrounding environment, and various noises. This raw visible light image without any image processing is directly defined as the raw image data. This raw image data serves as the basic data source for all subsequent image processing operations. All subsequent filtering and detection-related processing is based on this initial unprocessed image data.

[0077] When performing Gaussian filtering on the original image data, a neighborhood window with a preset fixed pixel range is selected, centered on each pixel in the original image data. The gray values ​​of all pixels within the neighborhood window are then weighted and averaged. The weighting is based on the distance between the pixel and the center pixel; pixels closer to the center pixel are assigned a higher weight, and pixels farther from the center pixel are assigned a lower weight. A new gray value is calculated using this weighted averaging method, and this new gray value replaces the original gray value of the corresponding center pixel in the original image data. The operations of neighborhood window selection, weighted averaging of gray values, and replacement of the center pixel's gray value are performed sequentially for each pixel in the original image data to complete the Gaussian filtering of the entire original image data. The complete image obtained after this processing is the intermediate filtered image of the coal conveyor belt operating area.

[0078] When performing edge detection processing on the intermediate filtered image, the grayscale difference between each pixel and its four neighboring pixels is calculated pixel by pixel. The four neighboring pixels are the four pixels above, below, to the left, and to the right of the current pixel. The absolute difference of grayscale value between the current pixel and its four neighboring pixels is calculated for each direction. The calculated absolute difference of grayscale value in each direction is compared with a preset grayscale difference threshold. If the absolute difference of grayscale value in a certain direction is greater than the preset grayscale difference threshold, the pixel is determined to have edge features in that direction. If the pixel has edge features in any direction, it is directly determined to be an edge pixel. If the absolute difference in grayscale values ​​between pixels is not greater than the preset grayscale difference threshold, then the pixel is determined to be a non-edge pixel. After performing the following operations on all pixels in the intermediate filtered image in sequence: calculating the absolute difference in grayscale values ​​of the four neighboring pixels, comparing with the preset grayscale difference threshold, and determining edge and non-edge pixels, a binary mask image with the same pixel size as the intermediate filtered image is generated. In this mask image, the pixel value of the edge pixel is set to 255 to mark the edge area that needs to be retained, and the pixel value of the non-edge pixel is set to 0 to mark the non-edge area that can be further filtered. The generated binary mask image is the edge retention mask of the coal conveyor belt operating area.

[0079] When performing adaptive parameter adjustment and secondary filtering on Gaussian filtering based on an edge-preserving mask, the pixel value corresponding to each pixel in the edge-preserving mask is first identified pixel by pixel. If the pixel value is 255, meaning the pixel is an edge pixel marked by the edge-preserving mask, the neighborhood window of the Gaussian filter for that pixel is adjusted to a preset minimum neighborhood window, which is a single-pixel window containing only that pixel. If the pixel value is 0, meaning the pixel is a non-edge pixel marked by the edge-preserving mask, the initial neighborhood window of the Gaussian filter for that pixel remains unchanged. This completes the step-by-step adjustment of the Gaussian filter neighborhood window parameters. After pixel adaptive adjustment, based on the adjusted neighborhood window, Gaussian filtering is performed again on the intermediate filtered image. That is, for each pixel, pixels within the corresponding range are selected according to the adjusted neighborhood window, and a weighted average of gray values ​​based on the distance between the pixel and the center pixel is calculated. The calculated new gray value replaces the original gray value of the corresponding center pixel in the intermediate filtered image. The above Gaussian filtering operation based on the adjusted neighborhood window is performed on all pixels in the intermediate filtered image in sequence to complete the secondary filtering process of the entire intermediate filtered image. The complete image obtained after this processing is the standard belt area image of the coal conveyor belt operating area.

[0080] S2. Perform multi-level image enhancement on the standard belt area image to obtain the contour enhancement image and texture suppression image of the coal conveyor belt operating area;

[0081] In this embodiment of the invention, the step of performing multi-level image enhancement on the standard belt area image to obtain an enhanced contour image and a texture-suppressed image of the coal conveyor belt operating area includes:

[0082] Brightness contrast adjustment is performed on the standard belt area image to obtain a brightness-balanced image of the coal conveyor belt operating area;

[0083] Applying the Laplacian edge enhancement operator to the brightness equalization image yields the contour enhancement image of the coal conveyor belt operating area;

[0084] Multi-directional texture analysis is performed on the brightness equalization image to generate a texture suppression image of the coal conveyor belt operating area.

[0085] When performing brightness contrast adjustment on a standard conveyor belt area image, the grayscale values ​​of all pixels in the image are extracted pixel by pixel. First, the grayscale value distribution of the entire standard conveyor belt area image is statistically analyzed, and the minimum and maximum grayscale values ​​are determined. Grayscale values ​​outside the range of the minimum to the maximum value are mapped to the minimum and maximum values ​​respectively. Then, all mapped grayscale values ​​are redistributed to the grayscale value range of 0 to 255 using a linear transformation. For each pixel, the corresponding transformed grayscale value is calculated based on its original grayscale value, and the original grayscale value of the pixel is replaced with the transformed grayscale value. After completing the above grayscale value adjustment operation on all pixels of the entire standard conveyor belt area image, the resulting complete image is the brightness equalization image of the coal conveyor belt operating area.

[0086] When performing Laplacian edge enhancement operator processing on a brightness-balanced image, a fixed 3×3 neighborhood window is selected centered on each pixel in the image. The gray values ​​of the eight surrounding pixels within the neighborhood window are sequentially compared with the gray value of the center pixel. The results of all the difference calculations are superimposed to obtain the gray value correction amount. The original gray value of the center pixel is superimposed with the gray value correction amount to obtain the corrected gray value. The original gray value of the corresponding center pixel in the brightness-balanced image is replaced with the corrected gray value. After completing the above neighborhood selection, difference calculation, gray value correction and replacement operations for all pixels in the entire brightness-balanced image, the complete image obtained is the contour enhancement image of the coal conveyor belt operating area.

[0087] When performing multi-directional texture analysis on a brightness-equalized image and generating a texture-suppressed image, four fixed texture analysis directions are selected: horizontal, vertical, 45-degree diagonal, and 135-degree diagonal. Starting from each pixel in the image, a pixel sequence containing nine consecutive pixels is selected in each analysis direction. The absolute difference in grayscale values ​​between adjacent pixels in each pixel sequence is calculated and accumulated to obtain the total grayscale value change amplitude of the sequence. Pixel sequences with a total grayscale value change amplitude of less than 50 are identified as texture regions, and pixel sequences with a total grayscale value change amplitude of greater than or equal to 50 are identified as non-texture regions. For each pixel in a texture region, the average grayscale value of all pixels in the pixel sequence in the four analysis directions is taken, and the original grayscale value of the pixel is replaced with this average value. For pixels in non-texture regions, their original grayscale values ​​remain unchanged. After completing the above multi-directional sequence selection, amplitude calculation, region determination, and grayscale value processing and replacement operations for all pixels in the entire brightness-equalized image, the complete image obtained is the texture-suppressed image of the coal conveyor belt operating area.

[0088] S3. The contour enhancement image and the texture suppression image are synergistically enhanced and fused to obtain the fusion features of the coal conveyor belt operating area. Based on the fusion features, the standard belt area image is semantically segmented and quantized to obtain the segmentation mask of the standard belt area image.

[0089] In this embodiment of the invention, the step of co-enhancing and fusing the contour-enhanced image and the texture-suppressed image to obtain the fused features of the coal conveyor belt operating area, and then performing semantic segmentation and quantization on the standard belt area image based on the fused features to obtain the segmentation mask of the standard belt area image, includes:

[0090] Based on historical foreign object image detection records of the coal conveyor belt operating area, and combined with an attention mechanism, the contour enhancement image and the texture suppression image are learned and mapped to construct a fusion weight generation network for the coal conveyor belt operating area.

[0091] The contour enhancement image and the texture suppression image are input into the fusion weight generation network to obtain a pixel-level contour enhancement weight map of the contour enhancement image and a pixel-level texture suppression weight map of the texture suppression image.

[0092] Based on the pixel-level contour enhancement weight map and the pixel-level texture suppression weight map, the contour enhancement image and the texture suppression image are weighted and quantized respectively to obtain the contour contribution component of the contour enhancement image and the texture contribution component of the texture suppression image.

[0093] The contour contribution component and the texture contribution component are superimposed pixel by pixel to obtain the fused feature map of the coal conveyor belt operating area.

[0094] Based on the fused feature map, a semantic labeling system is constructed for the belt area, background area and the area to be identified in the coal conveyor belt operating area, and pixel-level semantic classification is performed on the standard belt area image to obtain the initial semantic segmentation result of the standard belt area image;

[0095] Spatial context optimization is performed on the initial semantic segmentation result to obtain the segmentation mask of the standard belt region image.

[0096] Based on the fused feature map, a semantic labeling system is constructed for the belt area, background area, and unidentified area of ​​the coal conveyor belt operating area, and pixel-level semantic classification is performed on the standard belt area image to obtain the initial semantic segmentation result of the standard belt area image, including:

[0097] Perform multi-scale convolution operations on the fused feature map to obtain the context feature map of the coal conveyor belt operating area;

[0098] Based on the context feature map, a semantic label category set is defined for the coal conveyor belt operating area. The semantic label category set includes categories corresponding to the belt area, categories corresponding to the background area, and categories corresponding to the area of ​​foreign objects to be identified.

[0099] The context feature map is input into the pixel-level classification network in the fusion weight generation network to obtain the probability vector of the semantic label category set;

[0100] Based on the pixel positions of the fused feature map, the probability vector is numerically filtered to generate an initial semantic label map of the coal conveyor belt operating area;

[0101] Connectivity analysis is performed on the initial semantic label map, and based on the analysis results, spatially adjacent pixel regions with the same label in the coal conveyor belt operating area are merged to obtain the initial semantic segmentation result of the standard belt area image.

[0102] All image data contained in the historical foreign object image detection records of the coal conveyor belt operating area are extracted. Feature information related to the coal conveyor belt contour and feature information related to the coal conveyor belt texture are separated from them. An attention mechanism is applied to the two types of feature information, so that the mechanism focuses on the pixel area where foreign objects appear in the historical foreign object image detection records. Pixel-level association learning and feature mapping are performed on the contour enhancement image features and texture suppression image features of the region. Through continuous sample learning and mapping relationship optimization, a fusion weight generation network that can adapt to the image features of the coal conveyor belt operating area is built.

[0103] The contour enhancement image and texture suppression image are input into the fusion weight generation network in the form of a pixel array. Based on the feature mapping relationship learned in the previous stage, the network performs feature matching and weight calculation on each pixel of the contour enhancement image to generate a pixel-level contour enhancement weight map with the same pixel size as the contour enhancement image. At the same time, the same feature matching and weight calculation operation is performed on each pixel of the texture suppression image to generate a pixel-level texture suppression weight map with the same pixel size as the texture suppression image.

[0104] The weight value of each pixel in the pixel-level contour enhancement weight image is extracted. This weight value is then combined with the gray value of the corresponding pixel in the contour enhancement image. A full-pixel weighted quantization operation is performed on the contour enhancement image to obtain the contour contribution component of the contour enhancement image, which can reflect the contribution of contour features. At the same time, the weight value of each pixel in the pixel-level texture suppression weight image is extracted. This weight value is then combined with the gray value of the corresponding pixel in the texture suppression image. A full-pixel weighted quantization operation is performed on the texture suppression image to obtain the texture contribution component of the texture suppression image, which can reflect the contribution of texture features.

[0105] The gray values ​​of the contour contribution component and the texture contribution component are extracted pixel by pixel. The gray values ​​of the contour contribution component and the texture contribution component at the same pixel position are superimposed. If the superimposed gray value exceeds the preset gray value range of 0 to 255, the gray value exceeding the upper limit is corrected to 255, and the gray value below the lower limit is corrected to 0. After the above superposition and gray value correction operations are performed on all pixels of the entire image, the complete image obtained is the fusion feature map of the coal conveyor belt operating area.

[0106] Based on the pixel features of the coal conveyor belt area, background area, and area to be identified extracted from the fused feature map, a unique semantic label is assigned to each of the three types of areas. The belt area corresponds to semantic label 1, the background area corresponds to semantic label 2, and the area to be identified corresponds to semantic label 3. A semantic label system for the coal conveyor belt operation area, including the three types of areas and their corresponding semantic labels, is constructed. Then, based on this semantic label system, pixel features of the standard belt area image are extracted pixel by pixel and matched with the features of the corresponding pixels in the fused feature map. According to the feature matching results, each pixel of the standard belt area image is assigned a corresponding semantic label. After completing the pixel-level semantic classification of the entire standard belt area image, the resulting image with semantic labels is the initial semantic segmentation result of the standard belt area image.

[0107] When performing spatial context optimization on the initial semantic segmentation result, a 3×3 neighborhood window is selected centered on each pixel in the result. The distribution of semantic labels corresponding to the nine pixels in the neighborhood window is statistically analyzed. If the semantic label of the center pixel is inconsistent with the semantic labels of more than seven pixels in the neighborhood window, the semantic label of the center pixel is replaced with the semantic label that appears most frequently in the neighborhood window. If the semantic label of the center pixel is inconsistent with the semantic labels of seven or fewer pixels in the neighborhood window, the original semantic label of the center pixel is kept unchanged. After completing the above neighborhood statistics and label correction operations on all pixels of the entire initial semantic segmentation result, the resulting binarized image with accurate semantic labels is the segmentation mask of the standard belt region image.

[0108] When performing multi-scale convolution operations on the fused feature map, three fixed-size neighborhood windows of 3×3, 5×5, and 7×7 are selected as convolution windows of different scales. Taking each pixel in the fused feature map as the center, the grayscale values ​​of the center pixel and its surrounding pixels are integrated using the three scale convolution windows respectively. Each scale convolution window obtains the feature value of the center pixel at that scale by traversing the pixels around the center pixel and integrating their grayscale features. All pixels in the fused feature map are subjected to convolution processing at the three scales in sequence. Then, the feature values ​​obtained by each pixel at the three scales are integrated to form a complete image containing feature information at different scales. This image is the context feature map of the coal conveyor belt operating area.

[0109] When constructing a semantic label category set based on the context feature map, the feature information of all pixels in the context feature map is first extracted pixel by pixel. Feature pixel groups with continuous gray value distribution and pixels clustered to form long strip-shaped areas are selected. The area corresponding to this type of pixel group is defined as the conveyor belt area and assigned a unique semantic category. Pixel groups with random gray value distribution and no continuous association with the long strip-shaped pixel groups are selected. The area corresponding to this type of pixel group is defined as the background area and assigned a unique semantic category. Pixel groups with gray value different from the conveyor belt area and fewer than 50 pixels that are isolated are selected. The area corresponding to this type of pixel group is defined as the foreign object area to be identified and assigned a unique semantic category. The three unique semantic categories together form the semantic label category set of the coal conveyor belt operation area, and each category has a unique and distinguishable identifier.

[0110] The context feature map is input as a complete pixel array into the pixel-level classification network in the fusion weight generation network. Based on the feature mapping relationship learned from previous foreign object image detection records, the classification network performs feature matching on each pixel in the context feature map, calculates the feature matching degree between the pixel and each category in the semantic label category set, and converts the matching degree of the three categories into corresponding values. The three values ​​together form a vector, which is the probability vector of the semantic label category set. Each value corresponds to the matching probability of a semantic category, and the sum of the three values ​​is a fixed value.

[0111] When performing numerical filtering of probability vectors based on pixel positions in the fused feature map, the probability filtering threshold is first determined to be 0.6. Each pixel position in the fused feature map is located pixel by pixel, and three values ​​in the probability vector corresponding to that position are extracted. The magnitudes of the three values ​​are compared and the maximum value is selected. If the maximum value is greater than or equal to 0.6, the semantic category label corresponding to the maximum value is assigned to that pixel position; if the maximum value is less than 0.6, the semantic category label corresponding to the foreign object region to be identified is assigned to that pixel position. The above filtering and label assignment operations are performed on all pixel positions in the fused feature map to generate an image in which each pixel has a semantic category label. This image is the initial semantic label map of the coal conveyor belt operating area.

[0112] When performing connected component analysis on the initial semantic label map, all pixels in the initial semantic label map are traversed row by row and column by column. When an unlabeled pixel is scanned, its semantic label is recorded. At the same time, the pixels adjacent to the pixel in the four directions of up, down, left, and right are searched. It is determined whether the semantic labels of the adjacent pixels are consistent with those of the pixel. If they are consistent, they are determined to be in the same connected component and are marked. The search continues until there are no adjacent pixels with the same label, thus completing the marking of a connected component. The above operation is repeated for all unlabeled pixels. After the analysis of all connected components is completed, spatially adjacent connected components with the same semantic label are merged. The merged connected component is taken as a complete region, and the region label in the initial semantic label map is updated. The image obtained after completing all merging operations is the initial semantic segmentation result of the standard belt region image.

[0113] S4. Based on the segmentation mask, perform binocular visual perception on the standard belt area image to obtain the three-dimensional point cloud data of the coal conveyor belt operating area;

[0114] In this embodiment of the invention, the step of performing binocular visual perception on the standard belt area image based on the segmentation mask to obtain three-dimensional point cloud data of the coal conveyor belt operating area includes:

[0115] Based on the preset binocular camera parameters, stereo correction is performed on the standard belt area image and the corresponding synchronous image from another perspective to obtain a stereo image pair of the coal conveyor belt operating area;

[0116] Based on the segmentation mask, the matching region image pair corresponding to the coal conveyor belt operating area is extracted from the stereo image pair;

[0117] Dense stereo matching is performed on the image pairs of the regions to be matched to obtain the initial disparity map of the coal conveyor belt operating area;

[0118] Based on the binocular camera parameters and the initial disparity map, triangulation calculations are performed on the coal conveyor belt operating area to obtain the initial three-dimensional spatial point set of the coal conveyor belt operating area.

[0119] Outlier filtering and noise smoothing are performed on the initial three-dimensional spatial point set to obtain the three-dimensional point cloud data of the coal conveyor belt operating area.

[0120] The calculation formula for triangulation is as follows:

[0121] ;

[0122] In the formula, The pixel coordinates of the initial disparity map. The target three-dimensional spatial coordinates of the coal conveyor belt operating area are: pixel coordinates The weighted projection matrix, The principal point coordinates of the binocular camera parameters are... For calculating the disparity in the dense stereo matching, The binocular camera baseline distance is the parameter of the binocular camera. The equivalent focal length of the binocular camera is the parameter of the camera.

[0123] The pixel coordinates are derived from the initial disparity map of the coal conveyor belt operating area. The initial disparity map is the product obtained after performing dense stereo matching on the image pairs of the regions to be matched. The position information of each pixel in the initial disparity map is extracted pixel by pixel. This position information is the pixel coordinate used for calculation in the formula. Its coordinate value corresponds completely to the pixel arrangement order of the initial disparity map. Each pixel coordinate uniquely corresponds to a pixel in the initial disparity map, and thus corresponds to a pixel in the image pair of the regions to be matched.

[0124] The target three-dimensional spatial coordinates are the result calculated by this formula. They are used to characterize the real spatial position of each pixel point within the operating area of ​​the coal conveyor belt. After the calculation is completed, the set of all target three-dimensional spatial coordinates is the initial three-dimensional spatial point set of the coal conveyor belt operating area. These coordinates can accurately reflect the three-dimensional spatial distribution of the coal conveyor belt, the foreign object to be identified, and the surrounding related areas.

[0125] The weighted projection matrix is ​​obtained by learning the mapping based on historical foreign object image detection records of the coal conveyor belt operation area and combining them with an attention mechanism. In the early stage, the contour features and texture features of the historical foreign object image detection records are extracted. The attention mechanism is used to focus on the pixel area where the foreign object appears. The pixel mapping relationship of the region is continuously learned and optimized, and finally the weighted projection matrix corresponding to each pixel coordinate point is obtained. Its function is to correct the deviation in the pixel projection process and ensure the accuracy of the three-dimensional coordinate calculation.

[0126] Principal point coordinates are an important component of the binocular camera parameters. They are obtained by calibrating the binocular camera in advance. During the calibration process, the center position of the camera's imaging plane is located by photographing a standard calibration board. The coordinates of this center position are the principal point coordinates. After calibration, the principal point coordinates are stored as fixed parameters, and subsequent triangulation calculations directly call the principal point coordinates in the preset binocular camera parameters.

[0127] The disparity calculation originates from the dense stereo matching process. Using the first-view image after centering and correction of the image of the region to be matched as a reference, the matching pixels in the second-view image are searched pixel by pixel for gray values ​​and differences in gray distribution characteristics of the neighborhood that meet the preset requirements. The pixel distance between the reference pixel and the matching pixel in the horizontal direction is calculated. This distance is the calculated disparity of the corresponding pixel coordinate point. All calculated disparities are arranged according to pixel coordinates to form an initial disparity map. The calculated disparity is directly extracted and used from the initial disparity map.

[0128] The baseline distance of the binocular camera is a fixed component of the binocular camera parameters. It is derived from the prior installation and calibration of the binocular camera. The straight-line distance between the centers of the two camera lenses is accurately measured using professional measuring tools. After the distance measurement is completed, it is stored as a fixed parameter and used as one of the preset binocular camera parameters for direct use in triangulation calculations. Its value remains unchanged.

[0129] The equivalent focal length of a camera is a core component of the parameters of a stereo camera. It is derived from the imaging calibration of the stereo camera in advance. By shooting a standard target of known actual size and combining the target's imaging size in the image, the equivalent focal length of the camera lens is determined. After calibration, this focal length is stored as a fixed parameter in the stereo camera parameters, and subsequent triangulation calculations directly call this preset parameter.

[0130] The core significance of this formula is to realize the transformation from pixel coordinates to three-dimensional spatial coordinates in the coal conveyor belt operating area. By combining the pixel coordinates in the initial disparity map obtained from dense stereo matching with preset binocular camera parameters and weighted projection matrix, the true three-dimensional spatial coordinates corresponding to each pixel are calculated through a fixed correlation relationship, thereby obtaining an initial three-dimensional spatial point set. This provides the core three-dimensional coordinate foundation for subsequent outlier filtering and noise smoothing, and finally obtains the three-dimensional point cloud data of the coal conveyor belt operating area. It accurately correlates image pixel information with real spatial distribution information, ensuring that the three-dimensional point cloud data can accurately reflect the actual spatial situation of the coal conveyor belt operating area, which is in line with the core purpose of triangulation calculation.

[0131] The preset binocular camera parameters are fixed parameters obtained in advance through camera calibration, including fixed information such as the relative position of the two cameras and the imaging focal length. The standard belt area image is used as the first-view image, and the visible light image of the coal conveyor belt running area acquired simultaneously from another viewpoint is used as the second-view image. The pixel positions of the first-view and second-view images are adjusted pixel by pixel to eliminate pixel offset caused by camera installation angle and lens distortion, so that the pixels of corresponding scenes in the two images are on the same horizontal alignment line, ensuring that the images of the two views can correspond accurately. After all pixel adjustment operations are completed, the image pair composed of the corrected first-view image and the corrected second-view image is the stereo image pair of the coal conveyor belt running area.

[0132] The segmentation mask contains pixel markers for the belt area, background area, and area to be identified within the coal conveyor belt operating area. The pixel markers of the segmentation mask are read pixel by pixel, and all pixel positions marked as belt area and area to be identified are filtered out. The coordinate information of these pixel positions is recorded, and pixel positions marked as background area are removed. Based on the filtered pixel coordinate information, all pixels at the corresponding coordinate positions are extracted from the two images of the stereo image pair to form two images containing only belt area and area to be identified. These two images together form the image pair of the area to be matched corresponding to the coal conveyor belt operating area.

[0133] Using the first-view image after centering and correction of the image of the region to be matched as the reference image, the gray value of each pixel in the reference image and the gray value distribution features of the surrounding 3×3 neighborhood are extracted pixel by pixel. In the second-view image after centering and correction of the image of the region to be matched, along the horizontal alignment line corresponding to the pixel, pixels with a difference of less than 10 in gray value and gray value distribution features of the surrounding 3×3 neighborhood are found as matching pixels. The pixel distance between the reference image pixel and the matching pixel in the horizontal direction is calculated. This distance is the disparity value of the corresponding pixel in the reference image. The above matching and disparity value calculation operations are performed on all pixels in the reference image. The disparity values ​​of all pixels are arranged according to the corresponding pixel positions. The complete image formed is the initial disparity map of the coal conveyor belt operating area.

[0134] The preset binocular camera parameters are invoked. These parameters include pre-calibrated fixed information such as the fixed distance between the two cameras and the camera imaging focal length. The disparity value of each pixel in the initial disparity map is extracted pixel by pixel. Combined with the coordinate position of the corresponding pixel in the reference image, the specific position of the real spatial point corresponding to the pixel in the three-dimensional coordinate system is determined. The disparity value is fixedly related to the distance from the spatial point to the camera. The larger the disparity value, the closer the spatial point is to the camera. The smaller the disparity value, the farther the spatial point is from the camera. Combined with the relative position of the two cameras, the coordinates of the three-dimensional spatial point corresponding to each pixel are determined. The set of all three-dimensional spatial point coordinates is the initial three-dimensional spatial point set of the coal conveyor belt operating area.

[0135] When filtering out outliers from the initial 3D spatial point set, each spatial point in the initial 3D spatial point set is selected one by one, and the spatial distance between the spatial point and its 10 neighboring spatial points is calculated. The average of these 10 spatial distances is then calculated. If the difference between the spatial point and the average value is greater than 5 spatial units, the spatial point is determined to be an outlier and is removed. After removing outliers, noise smoothing is performed on the remaining spatial points. The average 3D coordinates of each spatial point and its 5 neighboring spatial points are calculated one by one, and the average value is used to replace the original 3D coordinates of the spatial point, making the distribution of spatial points more uniform. After completing the outlier filtering and noise smoothing operations, the remaining 3D spatial point set is the 3D point cloud data of the coal conveyor belt operation area.

[0136] S5. Perform statistical projection on the spatial distribution characteristics of surface normals in the three-dimensional point cloud data to obtain the regional surface normal distribution map of the coal conveyor belt operating area;

[0137] In this embodiment of the invention, the step of statistically projecting the spatial distribution characteristics of surface normals in the three-dimensional point cloud data to obtain a regional surface normal distribution map of the coal conveyor belt operating area includes:

[0138] Surface normals are estimated from the three-dimensional point cloud data to obtain the three-dimensional normal field of the three-dimensional point cloud data;

[0139] The normal vectors in the three-dimensional normal field are oriented and normalized to obtain the normal direction feature map of the coal conveyor belt operating area.

[0140] The three-dimensional point cloud data of the coal conveyor belt operating area is projected onto a two-dimensional plane, and the mapping relationship between the normal direction feature map and the two-dimensional projection grid in the two-dimensional plane is established according to the projection relationship.

[0141] Based on the mapping relationship, the normal direction features of the normal direction feature map are mapped to the independent grids of the two-dimensional projection grid to perform statistical histograms, thereby obtaining the normal distribution descriptor of the normal direction feature map;

[0142] The normal distribution descriptor is visualized and fused according to the spatial order of the two-dimensional projection grid to obtain the regional surface normal distribution map of the coal conveyor belt operating area.

[0143] When estimating the surface normal of the 3D point cloud data of the coal conveyor belt operating area, each spatial point in the 3D point cloud data is selected point by point. Taking this spatial point as the center, 20 adjacent spatial points are selected as the neighborhood point set. By integrating the spatial coordinate distribution of the neighborhood point set, a plane that fits the spatial point and the neighborhood point set is fitted. The vertical direction of this plane is the surface normal direction of the spatial point. At the same time, it is determined that the orientation of the normal always points to the outside of the coal conveyor belt operating area. The above neighborhood point selection, plane fitting and normal direction determination operations are performed on all spatial points in the 3D point cloud data. The set formed by arranging the surface normals of all spatial points according to the corresponding spatial point positions is the 3D normal field of the 3D point cloud data.

[0144] When performing direction normalization encoding on the normal vectors in the 3D normal field, the direction information of each normal vector in the 3D normal field is extracted, and the directions of all normal vectors are uniformly mapped to the gray value range of 0 to 255. Each specific normal direction corresponds to a unique gray value, where the horizontal normal corresponds to a gray value of 128, the vertical normal corresponds to a gray value of 0, and the inclined normal corresponds to a gray value between 0 and 255 according to the inclination angle. This ensures that normals in different directions can be clearly distinguished by gray values. The gray values ​​corresponding to all normal vectors are arranged according to the coordinate position of their corresponding spatial points, and the complete image formed is the normal direction feature map of the coal conveyor belt operating area.

[0145] When projecting the 3D point cloud data of the coal conveyor belt operating area onto a 2D plane, a vertical projection method is used to convert the 3D coordinates of all spatial points in the 3D point cloud data into 2D coordinates. Only the horizontal and vertical coordinates of each spatial point are retained, while the height coordinates are discarded. At the same time, the 2D projection area is divided into a fixed-size 100×100 2D projection grid. Each grid corresponds to a fixed range of 2D coordinates. Based on the 2D projection coordinates of each spatial point, its corresponding 2D projection grid is determined. Then, a one-to-one correspondence is established between each pixel in the normal direction feature map and the corresponding grid in the 2D projection grid, ensuring that each gray value in the normal direction feature map can be accurately mapped to the corresponding position in the 2D projection grid.

[0146] Based on the established mapping relationship, the normal direction features of the normal direction feature map are mapped to independent grids of the two-dimensional projection mesh. The gray values ​​corresponding to all mapped normal direction features are extracted grid by grid. The gray values ​​in each independent grid are processed by statistical histogram, and the number of times each gray value appears in the grid is counted. Each grid corresponds to a statistical histogram containing 256 gray levels. The set of statistical histograms of all independent grids is the normal distribution descriptor of the normal direction feature map. Each histogram can clearly reflect the distribution of normal directions in the corresponding grid.

[0147] When visualizing and rendering the normal distribution descriptors according to the spatial order of the two-dimensional projected grid, the normal distribution descriptors corresponding to each two-dimensional projected grid are extracted sequentially from left to right and from top to bottom. The statistical results of each histogram are converted into corresponding colors. The color corresponding to the gray value that appears most frequently is used as the main color of the grid. The colors of other gray values ​​in the grid are superimposed on the main color according to their frequency of occurrence. The above color conversion and superposition operations are performed on all two-dimensional projected grids. The colors of all grids are stitched together and blended in spatial order to form a complete visualization image, which is the regional surface normal distribution map of the coal conveyor belt operating area.

[0148] S6. Perform a foreign object coupling evaluation on the fusion feature, the shape feature of the segmentation mask and the geometric feature of the surface normal distribution map of the region to obtain a comprehensive foreign object detection report for the coal conveyor belt operating area.

[0149] In this embodiment of the invention, the foreign object coupling evaluation of the fusion feature, the shape feature of the segmentation mask, and the geometric feature of the surface normal distribution map of the region to obtain a comprehensive foreign object detection report for the coal conveyor belt operating area includes:

[0150] Spatial feature analysis is performed on the connected components of the region to be identified in the segmentation mask to obtain the shape features of the connected components;

[0151] Based on the spatial location of the connected domain, the attribute features of the surface normal distribution map of the region are analyzed to obtain the geometric features of the coal conveyor belt operating area.

[0152] The fused features, shape features, and geometric features are concatenated to construct a comprehensive description vector of the region to be identified.

[0153] Based on the comprehensive description vector, foreign object detection is performed on the area to be identified to obtain a comprehensive foreign object detection report for the coal conveyor belt operating area.

[0154] The segmentation mask is marked with connected component information indicating the region to be identified. All connected components marked as regions to be identified are traversed sequentially. Each connected component corresponds to an independent target. The total number of pixels contained within each connected component is counted; this number represents the area of ​​the connected component. The edges of the connected components are scanned pixel by pixel, and the total number of edge pixels is counted; this number represents the perimeter of the connected component. The minimum bounding rectangle of the connected component is calculated, and its length and width are counted. The shape regularity of the connected component is obtained by dividing the width by the length. When the regularity is between 0.8 and 1.2, it is considered approximately circular; less than 0.8 is considered elongated; and greater than 1.2 is considered wide and flat. Simultaneously, the center pixel coordinates of the connected component are counted; these coordinates represent the spatial center position of the connected component. The area, perimeter, shape regularity, and spatial center position of each connected component are integrated to form the shape feature set of the connected component. The shape features of all connected components of the region to be identified together constitute the shape feature set of the region to be identified in the segmentation mask.

[0155] Extract the spatial center coordinates and pixel range coordinates of the connected domain of each region to be identified. Based on this coordinate information, locate the corresponding region in the surface normal distribution map. This region completely corresponds to the spatial position of the connected domain of the region to be identified in the segmentation mask. Extract the gray value of the normal direction in the corresponding region pixel by pixel, and statistically analyze the distribution of gray values. Record the normal direction corresponding to the gray value that appears most frequently. This direction is the main normal orientation of the region to be identified. Calculate the percentage of pixels with the same gray value in the region. When the percentage is greater than 80%, the normal orientation of the region is considered to be consistent. When it is less than or equal to 80%, the normal orientation is considered to be diverse. At the same time, extract the average normal gray value of the region and the corresponding average normal direction. Integrate the main normal orientation, the consistency of normal orientation, and the average normal direction to obtain the geometric features of the region to be identified corresponding to the coal conveyor belt operation area. The geometric features of all regions to be identified together form the overall geometric feature set.

[0156] For each region to be identified, a feature concatenation operation is performed. First, the pixel grayscale value distribution features of the corresponding region to be identified are extracted from the fused feature map. This feature is a set of grayscale values ​​of all pixels in the region to be identified arranged in coordinate order. Then, the shape features corresponding to the region to be identified are extracted, including area, perimeter, shape regularity, and spatial center position. Finally, the geometric features corresponding to the region to be identified are extracted, including the main normal orientation, normal orientation consistency, and average normal direction. These three types of features are concatenated in a fixed order of "fused features - shape features - geometric features" to ensure that the information of each feature is not omitted or repeated. Each region to be identified corresponds to a concatenated feature set, which is the comprehensive description vector of the region to be identified. The comprehensive description vectors of all regions to be identified together form the comprehensive description vector set.

[0157] Historical foreign object detection records for the coal conveyor belt operating area are extracted, and comprehensive feature templates corresponding to various types of foreign objects are compiled. Each foreign object template includes typical fusion features, shape features, and geometric features of the foreign object. The comprehensive description vector of each area to be identified is compared with the historical foreign object comprehensive feature templates one by one, and the feature matching degree is calculated. A matching degree threshold of 60% is set. When the matching degree is greater than or equal to 60%, the area to be identified is determined to be a foreign object, and the spatial location, shape features, geometric features, and foreign object type corresponding to the matching degree are recorded. When the matching degree is less than 60%, the area to be identified is determined to be a non-foreign object, and its spatial location and the basis for determination are recorded. The determination results of all areas to be identified are statistically analyzed, and the total number of foreign objects, detailed feature information of each foreign object, and distribution of non-foreign objects are compiled to form a complete text report, which is the comprehensive foreign object detection report for the coal conveyor belt operating area.

[0158] like Figure 2 The diagram shown is a functional block diagram of a foreign object detection system for coal conveyor belts based on visible light images, provided in an embodiment of the present invention.

[0159] The foreign object detection system 100 for coal conveyor belts based on visible light images described in this invention can be installed in an electronic device. Depending on the functions implemented, the foreign object detection system 100 may include an image smoothing and denoising module 101, a multi-level image enhancement module 102, a collaborative enhancement fusion and semantic segmentation module 103, a binocular vision perception module 104, a surface normal distribution statistics module 105, and a foreign object coupling evaluation module 106. The modules described in this invention can also be referred to as units, which are a series of computer program segments that can be executed by the processor of an electronic device and perform a fixed function, stored in the memory of the electronic device.

[0160] In this embodiment, the functions of each module / unit are as follows:

[0161] The image smoothing and denoising module 101 is used to smooth and denoise the visible light image of the coal conveyor belt operating area to obtain a standard belt area image of the coal conveyor belt operating area.

[0162] The multi-level image enhancement module 102 is used to perform multi-level image enhancement on the standard belt area image to obtain the contour enhancement image and texture suppression image of the coal conveyor belt operating area.

[0163] The collaborative enhancement fusion and semantic segmentation module 103 is used to perform collaborative enhancement fusion of the contour enhancement image and the texture suppression image to obtain the fusion features of the coal conveyor belt operating area, and based on the fusion features, to perform semantic segmentation quantization on the standard belt area image to obtain the segmentation mask of the standard belt area image.

[0164] The binocular vision perception module 104 is used to perform binocular vision perception on the standard belt area image based on the segmentation mask, so as to obtain the three-dimensional point cloud data of the coal conveyor belt operating area.

[0165] The surface normal distribution statistics module 105 is used to statistically project the spatial distribution characteristics of surface normals in the three-dimensional point cloud data to obtain the regional surface normal distribution map of the coal conveyor belt operating area.

[0166] The foreign object coupling evaluation module 106 is used to perform foreign object coupling evaluation on the fusion feature, the shape feature of the segmentation mask and the geometric feature of the surface normal distribution map of the region, so as to obtain a comprehensive foreign object detection report of the coal conveyor belt operating area.

[0167] In the several embodiments provided by this invention, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0168] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0169] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0170] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0171] This application embodiment can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for detecting foreign objects on a coal conveyor belt based on visible light images, characterized in that, The method includes: S1. Smooth and denoise the visible light image of the coal conveyor belt operating area to obtain a standard belt area image of the coal conveyor belt operating area; S2. Perform multi-level image enhancement on the standard belt area image to obtain the contour enhancement image and texture suppression image of the coal conveyor belt operating area; S3. The contour enhancement image and the texture suppression image are synergistically enhanced and fused to obtain the fusion features of the coal conveyor belt operating area. Based on the fusion features, the standard belt area image is semantically segmented and quantized to obtain the segmentation mask of the standard belt area image. S4. Based on the segmentation mask, perform binocular visual perception on the standard belt area image to obtain the three-dimensional point cloud data of the coal conveyor belt operating area; S5. Perform statistical projection on the spatial distribution characteristics of surface normals in the three-dimensional point cloud data to obtain the regional surface normal distribution map of the coal conveyor belt operating area; S6. A foreign object coupling evaluation is performed on the fusion feature, the shape feature of the segmentation mask, and the geometric feature of the surface normal distribution map of the region to obtain a comprehensive foreign object detection report for the coal conveyor belt operating area, including: Spatial feature analysis is performed on the connected components of the region to be identified in the segmentation mask to obtain the shape features of the connected components; Based on the spatial location of the connected domain, the attribute features of the surface normal distribution map of the region are analyzed to obtain the geometric features of the coal conveyor belt operating area. The fused features, shape features, and geometric features are concatenated to construct a comprehensive description vector of the region to be identified. Based on the comprehensive description vector, foreign object detection is performed on the area to be identified to obtain a comprehensive foreign object detection report for the coal conveyor belt operating area.

2. The method for detecting foreign objects in a coal conveyor belt based on visible light images as described in claim 1, characterized in that, The process of smoothing and denoising the visible light image of the coal conveyor belt operating area to obtain a standard belt area image of the coal conveyor belt operating area includes: The visible light images of the coal conveyor belt operating area are used as the raw image data. Gaussian filtering is applied to the original image data to obtain an intermediate filtered image of the coal conveyor belt operating area; Edge detection is performed on the intermediate filtered image to obtain the edge retention mask of the coal conveyor belt operating area; Based on the edge-preserving mask, the parameters of the Gaussian filtering operation are adaptively adjusted to perform secondary filtering processing, thereby obtaining a standard belt area image of the coal conveyor belt operating area.

3. The method for detecting foreign objects in a coal conveyor belt based on visible light images as described in claim 1, characterized in that, The step of performing multi-level image enhancement on the standard belt area image to obtain the contour enhancement image and texture suppression image of the coal conveyor belt operating area includes: Brightness contrast adjustment is performed on the standard belt area image to obtain a brightness-balanced image of the coal conveyor belt operating area; Applying the Laplacian edge enhancement operator to the brightness equalization image yields the contour enhancement image of the coal conveyor belt operating area; Multi-directional texture analysis is performed on the brightness equalization image to generate a texture suppression image of the coal conveyor belt operating area.

4. The method for detecting foreign objects in a coal conveyor belt based on visible light images as described in claim 1, characterized in that, The step of co-enhancing and fusing the contour-enhanced image and the texture-suppressed image to obtain the fused features of the coal conveyor belt operating area, and then performing semantic segmentation and quantization on the standard belt area image based on the fused features to obtain the segmentation mask of the standard belt area image, includes: Based on historical foreign object image detection records of the coal conveyor belt operating area, and combined with an attention mechanism, the contour enhancement image and the texture suppression image are learned and mapped to construct a fusion weight generation network for the coal conveyor belt operating area. The contour enhancement image and the texture suppression image are input into the fusion weight generation network to obtain a pixel-level contour enhancement weight map of the contour enhancement image and a pixel-level texture suppression weight map of the texture suppression image. Based on the pixel-level contour enhancement weight map and the pixel-level texture suppression weight map, the contour enhancement image and the texture suppression image are weighted and quantized respectively to obtain the contour contribution component of the contour enhancement image and the texture contribution component of the texture suppression image. The contour contribution component and the texture contribution component are superimposed pixel by pixel to obtain the fused feature map of the coal conveyor belt operating area. Based on the fused feature map, a semantic labeling system is constructed for the belt area, background area and the area to be identified in the coal conveyor belt operating area, and pixel-level semantic classification is performed on the standard belt area image to obtain the initial semantic segmentation result of the standard belt area image; Spatial context optimization is performed on the initial semantic segmentation result to obtain the segmentation mask of the standard belt region image.

5. The method for detecting foreign objects in a coal conveyor belt based on visible light images as described in claim 4, characterized in that, Based on the fused feature map, a semantic labeling system is constructed for the belt area, background area, and unidentified area of ​​the coal conveyor belt operating area, and pixel-level semantic classification is performed on the standard belt area image to obtain the initial semantic segmentation result of the standard belt area image, including: Perform multi-scale convolution operations on the fused feature map to obtain the context feature map of the coal conveyor belt operating area; Based on the context feature map, a semantic label category set is defined for the coal conveyor belt operating area. The semantic label category set includes categories corresponding to the belt area, categories corresponding to the background area, and categories corresponding to the area of ​​foreign objects to be identified. The context feature map is input into the pixel-level classification network in the fusion weight generation network to obtain the probability vector of the semantic label category set; Based on the pixel positions of the fused feature map, the probability vector is numerically filtered to generate an initial semantic label map of the coal conveyor belt operating area; Connectivity analysis is performed on the initial semantic label map, and based on the analysis results, spatially adjacent pixel regions with the same label in the coal conveyor belt operating area are merged to obtain the initial semantic segmentation result of the standard belt area image.

6. The method for detecting foreign objects in a coal conveyor belt based on visible light images as described in claim 1, characterized in that, The step of performing binocular visual perception on the standard belt area image based on the segmentation mask to obtain the three-dimensional point cloud data of the coal conveyor belt operating area includes: Based on the preset binocular camera parameters, stereo correction is performed on the standard belt area image and the corresponding synchronous image from another perspective to obtain a stereo image pair of the coal conveyor belt operating area; Based on the segmentation mask, the matching region image pair corresponding to the coal conveyor belt operating area is extracted from the stereo image pair; Dense stereo matching is performed on the image pairs of the regions to be matched to obtain the initial disparity map of the coal conveyor belt operating area; Based on the binocular camera parameters and the initial disparity map, triangulation calculations are performed on the coal conveyor belt operating area to obtain the initial three-dimensional spatial point set of the coal conveyor belt operating area. Outlier filtering and noise smoothing are performed on the initial three-dimensional spatial point set to obtain the three-dimensional point cloud data of the coal conveyor belt operating area.

7. The method for detecting foreign objects in a coal conveyor belt based on visible light images as described in claim 6, characterized in that, The calculation formula for triangulation is as follows: ； In the formula, The pixel coordinates of the initial disparity map. The target three-dimensional spatial coordinates of the coal conveyor belt operating area are: pixel coordinates The weighted projection matrix, The principal point coordinates of the binocular camera parameters are... For calculating the disparity in the dense stereo matching, The binocular camera baseline distance is the parameter of the binocular camera. The equivalent focal length of the binocular camera is the parameter of the camera.

8. The method for detecting foreign objects in a coal conveyor belt based on visible light images as described in claim 1, characterized in that, The step of statistically projecting the spatial distribution characteristics of surface normals in the three-dimensional point cloud data to obtain a regional surface normal distribution map of the coal conveyor belt operating area includes: Surface normals are estimated from the three-dimensional point cloud data to obtain the three-dimensional normal field of the three-dimensional point cloud data; The normal vectors in the three-dimensional normal field are oriented and normalized to obtain the normal direction feature map of the coal conveyor belt operating area. The three-dimensional point cloud data of the coal conveyor belt operating area is projected onto a two-dimensional plane, and the mapping relationship between the normal direction feature map and the two-dimensional projection grid in the two-dimensional plane is established according to the projection relationship. Based on the mapping relationship, the normal direction features of the normal direction feature map are mapped to the independent grids of the two-dimensional projection grid to perform statistical histograms, thereby obtaining the normal distribution descriptor of the normal direction feature map; The normal distribution descriptor is visualized and fused according to the spatial order of the two-dimensional projection grid to obtain the regional surface normal distribution map of the coal conveyor belt operating area.

9. A foreign object detection system for coal conveyor belts based on visible light images, characterized in that, The system for implementing the foreign object detection method for coal conveyor belts based on visible light images as described in claim 1 includes: The image smoothing and denoising module is used to smooth and denoise the visible light image of the coal conveyor belt operating area to obtain a standard belt area image of the coal conveyor belt operating area. A multi-level image enhancement module is used to perform multi-level image enhancement on the standard belt area image to obtain an enhanced contour image and a texture-suppressed image of the coal conveyor belt operating area. The collaborative enhancement fusion and semantic segmentation module is used to perform collaborative enhancement fusion of the contour enhancement image and the texture suppression image to obtain the fusion features of the coal conveyor belt operating area, and based on the fusion features, to perform semantic segmentation quantization on the standard belt area image to obtain the segmentation mask of the standard belt area image; A binocular vision perception module is used to perform binocular vision perception on the standard belt area image based on the segmentation mask, so as to obtain the three-dimensional point cloud data of the coal conveyor belt operating area. The surface normal distribution statistics module is used to statistically project the spatial distribution characteristics of surface normals in the three-dimensional point cloud data to obtain the regional surface normal distribution map of the coal conveyor belt operating area. The foreign object coupling assessment module is used to perform foreign object coupling assessment on the fusion feature, the shape feature of the segmentation mask and the geometric feature of the surface normal distribution map of the region, so as to obtain a comprehensive foreign object detection report of the coal conveyor belt operating area.

Images