A rock mass centroid positioning method based on instance segmentation and key point detection

Abstract

This invention discloses a method for locating the centroid of a rock block based on instance segmentation and keypoint detection, belonging to the field of visual perception and automated operation of mining robots. First, this invention constructs a dataset for mining rock pile scenarios and annotates and enhances the images. Then, an improved YOLOv8 instance segmentation network is designed to identify multi-target regions in the rock pile images, outputting an instance mask for each rock block. Next, the geometric centroid and effective impact area center of the rock block are calculated based on the instance mask, and centroid keypoint supervision information is generated. Finally, a posture keypoint detection network is trained based on centroid keypoint supervision to achieve end-to-end rapid localization of the rock block's centroid, used for the generation of impact points by the hydraulic breaker arm and operational control. Experimental results show that compared with localization methods based on detection box centers or traditional image processing, as well as the original YOLOv8 segmentation-keypoint baseline method, this invention can significantly improve the accuracy and robustness of centroid localization under complex occlusion and dense stacking conditions.

Description

Technical Field

[0001] This invention relates to the field of visual perception and automated operation of mining robots, specifically to a method for locating the centroid of a rock block based on instance segmentation and key point detection. Background Technology

[0002] In mining crushing, rock pile clearing, and loading operations, robotic crushing arms need to generate reliable impact points or grasping target points based on visual results. The rock mass center can be directly used for impact point generation, path planning, and end effector control; therefore, the accuracy and stability of the mass center positioning are crucial to operational efficiency and safety.

[0003] In existing technologies, common practices include: Using the center of the detection frame as the center point of the rock block, this method is prone to the center point deviating from the actual rock block area when the rock block shape is irregular, densely stacked, or severely obscured, resulting in unreliable impact points.

[0004] Traditional image processing is used to obtain the target region and then find its center; however, this method is sensitive to changes in lighting, dust, and background texture, and is not robust enough under field conditions.

[0005] Directly using general instance segmentation or keypoint networks; In the mining rock pile scene, the target scale spans a large range, there are many occlusions and the background is complex. The basic network has limited ability to separate boundaries and identify small targets, and is prone to instance adhesion or mask breakage, which causes centroid calculation errors; At the same time, relying solely on keypoint networks is also prone to keypoints falling on the background or adjacent rock blocks, resulting in unstable output.

[0006] Therefore, a method is needed for mining rock pile scenarios that can stably output the centroid coordinates of rock blocks under complex occlusion and dense stacking conditions. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention provides a method for locating the centroid of a rock block based on instance segmentation and key point detection.

[0008] To achieve the above technology, the specific steps include: S1. Acquire images of the mine rock pile scene using a camera, and perform pixel-level instance segmentation on each rock block in the image to obtain the instance mask and target number corresponding to each image; S2. Construct a rock block instance segmentation network based on deep learning. Input the image data and instance segmentation annotation data obtained in S1 into the network for training to obtain the rock block instance segmentation model. S3. Input the image of the rock pile to be located into the rock block instance segmentation model to obtain the instance mask of each rock block, and perform filtering and noise reduction on the instance mask to obtain the effective rock block region. S4. For each valid rock block region obtained in S3, calculate the geometric centroid coordinates of the rock block according to the mask pixel coordinates, and write the centroid coordinates into the centroid key point annotation file to form centroid key point training data. S5. Construct a key point detection network. Input the image data from S1 and the centroid key point training data from S4 into the key point detection network for training to obtain the rock block centroid key point detection model. S6. Match and verify the centroid key point results output by S5 with the instance mask output by S3 to obtain the final stable centroid coordinates of the rock block, thus completing the positioning.

[0009] Specifically, the method of constructing a deep learning-based rock block instance segmentation network includes: using YOLOv8-seg as the basic framework, adding the MANet module in the feature fusion stage, and introducing the SEBlock module between C2f and the segmentation detection head.

[0010] Specifically, step S2 includes: S2.1 Construct the basic YOLOv8-seg network skeleton; The input image is denoted as ,in, and These represent the height and width of the input image, respectively. The backbone network extracts three layers of output features, which are represented as follows: , and ; in, , and These represent the number of channels for each layer of features; The feature fusion network Neck will , and The features are fused to obtain a multi-scale fused feature set for segmentation prediction. S2.2 Introduce the MANet module for feature enhancement on multi-scale features; Features fused at any scale Performing the MANet enhancement operation is represented as: The MANet module consists of multi-branch feature extraction and attention weighting mechanisms, represented as follows: , , In the formula, Indicates the first Branches pair features The transformed output; Indicates the first The feature transformation operator with each branch is used to extract rock block structure information under different receptive fields; Indicates the number of branches; Indicates the first Attention weight coefficients for each branch; Represents the normalization function; This represents the attention weight generation function; The output feature space scale of the MANet module is consistent with the input feature space scale, expressed as: Right now and downsampling scale , and No change; S2.3. Introduce SEBlock between C2f and the segmentation detection head for channel recalibration; Inputting MANet augmented features of any scale into SEBlock yields: The calculation process for SEBlock is as follows: In the formula, Indicates the first Global average pooling results for each channel; and They represent the first The height and width of the feature map at each scale; and These represent the spatial location indices in the feature map; Represents the channel weight vector; For the Sigmoid function; and These represent the first learnable parameter and the second learnable parameter, respectively. For activation functions; This is the channel description vector obtained from global average pooling; Let be the weight of the c-th channel; Features after SE recalibration; Received As input to SegHead, it is used to output the rock block instance mask; S2.4 Output instance masks and perform end-to-end training. Will Input the segmentation prediction head SegHead, and output the category information, location regression information, and instance mask of the rock block target; The training phase employs a multi-task joint loss function: In the formula, Represents the joint loss function for multiple tasks; , , and Let represent the classification loss, bounding box regression loss, distributed regression loss, and mask loss, respectively. These are the weighting coefficients for the corresponding items.

[0011] Specifically, the filtering and noise reduction process in S3 includes: Remove targets with a confidence level below the confidence threshold; Convert the mask into a binary image; Only retain connected components with an area greater than the area threshold; Morphological opening, closing, and hole filling operations are performed on the preserved binary mask to obtain the effective rock block region corresponding to each target.

[0012] Specifically, the method by which S4 forms the centroid keypoint training data is as follows: Based on the effective rock block regions output by S3, calculate the geometric centroid for each effective region: In the formula, These are the pixel coordinates within the mask area; This represents the number of pixels in the effective area. Write the centroid coordinates into the centroid key point annotation file, in the following format: ,in, The coordinates of the centroid key point; As a marker of validity; All centroid keypoint annotation files in each image are mapped to the corresponding rock blocks.

[0013] Specifically, in S5, the method for obtaining the rock block centroid key point detection model is as follows: The image data of S1 and the centroid keypoint training data of S4 are used as inputs; Based on YOLOv8-pose as the basic framework for keypoint detection network, with the number of keypoints set to 1, corresponding to the centroid of the rock block, the predicted output is represented as follows: In the formula, To predict the centroid coordinates; To predict confidence levels; During training, the total loss is obtained by combining the keypoint regression loss with the detection correlation loss term, and is expressed as: In the formula, To detect branch loss; For keypoint regression loss; These are the weighting coefficients for each item.

[0014] Specifically, the method of S6 is as follows: Based on the effective mask output by S3 and the centroid key point output by S5; If the centroid keypoint falls within the effective mask, the match is successful; If the centroid keypoint does not fall within the effective mask, calculate the shortest distance from the centroid keypoint to the mask: In the formula, These are the pixel coordinates within the mask area; The coordinates of the centroid key point; Indicates the first One effective rock block area; If satisfied Then it is assumed that the key point lies within the neighborhood, where, The neighborhood threshold; If it falls neither within the mask nor within the domain, then the critical point is discarded; At the same time, key points with confidence levels below the threshold η are removed; Finally, the output centroid is stabilized using exponential smoothing: In the formula, The output centroid is α; α is the smoothing coefficient. The centroid of the current frame; The centroid coordinates output from the previous frame; Output the coordinates of the rock block's centroid.

[0015] The beneficial effects of this invention are: Instance segmentation is used to output the real target region, avoiding center point offset caused by the detection box center.

[0016] In YOLOv8-seg, MANet and SEBlock are introduced to improve mask quality under conditions of occlusion, dense stacking, and scale variation, thereby reducing centroid error from the source.

[0017] By using a mask to calculate the centroid and generate keypoint supervision, the keypoint model learns the "centroid definition," enabling end-to-end rapid localization.

[0018] By using "mask-key point consistency constraint" to remove abnormal key points, the error output of key points falling on the background or adjacent rock blocks is reduced.

[0019] Combined with confidence threshold and stability strategy, the output centroid is more stable and more suitable for real-time control of the breaker arm. Attached Figure Description

[0020] Figure 1 This is a step diagram of the method for locating the centroid of a mine rock block based on instance segmentation and key point detection according to the present invention; Figure 2 This is a framework diagram of the rock block instance segmentation and centroid localization technology based on the improved YOLOv8-seg of this invention; Figure 3 This is a schematic diagram of the improved YOLOv8-seg network structure of the present invention; Figure 4 This is a structural diagram of the MANet module used in this invention; Figure 5 This is a structural diagram of the SEBlock module used in this invention; Figure 6 This is a comparison chart of the segmentation results between the original model and the improved model. Detailed Implementation

[0021] The present invention will be further described in detail below with reference to specific embodiments.

[0022] like Figure 1 As shown, a method for locating the centroid of a rock block based on instance segmentation and key point detection includes the following steps: S1. Data collection and annotation; Images of rock pile scenes in mines are captured by industrial cameras installed on mobile crushers or fixed work platforms. The captured images contain different degrees of occlusion and different density of stacking. Images include: monocular RGB images or RGB-D images; Each instance of the rock block target in each image is labeled to obtain instance segmentation labeling data, which is labeled in the form of pixel-level instance masks; To enhance the model's generalization ability, image data is augmented, including but not limited to random scaling, cropping, flipping, and brightness perturbation, thereby constructing a dataset for training the instance segmentation network.

[0023] S2. Construct a deep learning-based rock block instance segmentation network. Input the image data and instance segmentation annotation data obtained in S1 into the network for training to obtain a rock block instance segmentation model. This includes the following steps: S2.1 Construct the basic YOLOv8-seg network skeleton; like Figure 2 and Figure 3 As shown, YOLOv8-seg is used as the basic framework, which includes: Backbone network, Neck feature fusion network, and SegHead segmentation prediction head; The input image is denoted as ,in, and These represent the height and width of the input image, respectively. The backbone network consists of convolutional layers (Conv), a feature extraction module (C2f), and a spatial pyramid pooling module (SPPF), which is used to extract multi-scale semantic features of rock block targets. The backbone network extracts three layers of output features, which are represented as follows: , and ; in, , and These represent the number of channels for each layer of features; The feature fusion network Neck employs an FPN / PAN-style feature fusion structure, which integrates... , and Perform top-down and bottom-up fusion to obtain a multi-scale fusion feature set for segmentation prediction; S2.2 Introduce the MANet module for feature enhancement on multi-scale features; To improve target separation capabilities under occlusion and dense stacking conditions, a MANet module is introduced into the multi-scale fusion features output by Neck, such as... Figure 4 As shown; Features fused at any scale Performing the MANet enhancement operation is represented as: Specifically, the MANet module consists of multi-branch feature extraction and attention weighting mechanisms, represented as follows: , , In the formula, Indicates the first Branches pair features The transformed output; Indicates the first The feature transformation operator with each branch is used to extract rock block structure information under different receptive fields; Indicates the number of branches; Indicates the first Attention weight coefficients for each branch; Represents the normalization function; This represents the attention weight generation function; The output feature space scale of the MANet module is consistent with the input feature space scale, expressed as: Right now and downsampling scale ( , and No change; S2.3. Introduce SEBlock between C2f and the segmentation detection head for channel recalibration; like Figure 5 As shown, the SEBlock module is introduced between the C2f of the Neck and the segmentation prediction head SegHead to adaptively calibrate the channel importance; Inputting MANet augmented features of any scale into SEBlock yields: Specifically, the calculation process for SEBlock is as follows: In the formula, Indicates the first Global average pooling results for each channel; and They represent the first The height and width of the feature map at each scale; and These represent the spatial location indices in the feature map; Represents the channel weight vector; For the Sigmoid function; and These represent the first learnable parameter and the second learnable parameter, respectively. For activation functions; This is the channel description vector obtained from global average pooling; Let be the weight of the c-th channel; Features after SE recalibration; Received As input to SegHead, it is used to output the rock block instance mask; S2.4 Output instance masks and perform end-to-end training. Will Input the segmentation prediction head SegHead, and output the category information, location regression information, and instance mask of the rock block target; The training phase employs a multi-task joint loss function: In the formula, Represents the joint loss function for multiple tasks; , , and Let represent the classification loss, bounding box regression loss, distributed regression loss, and mask loss, respectively. These are the weighting coefficients for the corresponding items.

[0024] S3. Input the image of the rock pile to be located into the rock block instance segmentation model to obtain the instance mask of each rock block, and perform filtering and noise reduction on the instance mask to obtain the effective rock block region. The screening and noise reduction steps include: S3.1 Confidence level screening; For each target output by the instance segmentation model confidence level Perform threshold filtering if: If the target is , then the target is removed; where, The confidence threshold; S3.2, Mask binarization; target instance mask Binarization is performed to obtain a binary mask. : In the formula, This is the binarization threshold; S3.3 Connectivity Component Filtering and Area Constraints; right Perform connected component analysis to obtain the set of connected components. The area of ​​the connected region is: ; Only retain those that meet the requirements The connected components or only the largest connected components are retained, where, Area threshold; S3.4 Morphological denoising and hole filling; Perform morphological opening, closing, and hole filling operations on the preserved binary mask, denoted as: In the formula, For opening operation, For closing operations, Fill the holes; S3.5, Output effective rock block area ; After processing As a goal The effective rock block region is used for subsequent centroid calculation and consistency constraints.

[0025] S4. For each valid rock block region obtained in S3, calculate the geometric centroid coordinates of the rock block according to the mask pixel coordinates, and write the centroid coordinates into the centroid keypoint annotation file to form centroid keypoint training data. The steps include: S4.1 Calculation of geometric centroid; For the target Effective rock block area Computational geometric centroid : In the formula, These are the pixel coordinates within the mask area; This represents the number of pixels in the effective area. S4.2 Generate single key point monitoring information; target Write the centroid coordinates into the centroid keypoint annotation file, and denote a single keypoint as: In the formula, The coordinates of the centroid key point; As a marker of validity; S4.3, Generate training data for centroid key points; All targets in each image The corresponding images are associated to form a centroid keypoint training dataset for training the keypoint detection network.

[0026] S5. Construct a keypoint detection network. Input the image data from S1 and the centroid keypoint training data from S4 into the keypoint detection network for training to obtain the rock block centroid keypoint detection model. This includes the following steps: S5.1 Construct the YOLOv8-pose single keypoint detection network; Using YOLOv8-pose as the basic framework for keypoint detection, with the number of keypoints set to 1, corresponding to the centroid keypoints of the rock block, the predicted output of input image I is represented as: In the formula, To predict the centroid coordinates; To predict confidence levels; S5.2 Definition of Keypoint Regression Loss; Keypoint regression loss is used during training. , is represented as: The total loss is obtained by combining the detection-related loss items: In the formula, To detect branch loss; These are the weighting coefficients.

[0027] S5.3 Output the centroid key point detection model; A rock mass centroid key point detection model is obtained through training, which can directly output the coordinates and confidence scores of the centroid key points during the inference stage.

[0028] S6. Match and verify the centroid key point results output by S5 with the effective mask output by S3, remove results that do not meet the condition that "key points fall within the corresponding mask or neighborhood", and obtain the final rock block centroid coordinates through confidence threshold and stability strategy. The output rock block centroid coordinates are used for the generation of the breaker arm impact point, path planning or operation control. The steps include: S6.1 Matching of Key Points and Masks For each predicted key point of the S5 output The output of S3 One effective rock block area Perform a match if: Then it is determined that the key point is related to the first Target matching; when multiple matching targets exist, the target with the highest confidence or the closest distance is selected as the matching result; S6.2 Consistency constraint verification; When the keypoint does not fall within the mask, calculate the shortest distance from the keypoint to the mask area: If satisfied Then it is assumed that the key point lies within the neighborhood, where, The neighborhood threshold; S6.3, Removal of Abnormal Results If the key point does not meet the condition of "falling within the corresponding mask or neighborhood", that is: The result for that key point will be discarded and will not be included in the final output. S6.4 Confidence threshold and stability strategy output the final centroid; The retained key point results are filtered using a confidence threshold η. Then it is discarded; and the output centroid is stabilized using exponential smoothing: In the formula, The output centroid is α; α is the smoothing coefficient. The centroid of the current frame; The centroid coordinates output from the previous frame; The final output of the rock block centroid coordinates is used for generating the impact point of the rockbreaker arm, path planning, or operation control.

[0029] To verify the present invention, the segmentation performance of the model of the present invention was compared with that of the original YOLOv8-seg model. The results are as follows: Figure 6 As shown; from Figure 6 It can be seen that, compared with the original model, the improved model can obtain a more complete rock mask area and a clearer target boundary under conditions of dense stacking, severe occlusion and blurred boundaries, effectively reducing instance adhesion and missegmentation, and providing a more stable input for subsequent centroid calculation and key point localization.

Claims

1. A method for locating the centroid of a rock block based on instance segmentation and key point detection, characterized in that, Includes the following steps: S1. Acquire images of the mine rock pile scene using a camera, and perform pixel-level instance segmentation on each rock block in the image to obtain the instance mask and target number corresponding to each image; S2. Construct a rock block instance segmentation network based on deep learning. Input the image data and instance segmentation annotation data obtained in S1 into the network for training to obtain the rock block instance segmentation model. S3. Input the image of the rock pile to be located into the rock block instance segmentation model to obtain the instance mask of each rock block, and perform filtering and noise reduction on the instance mask to obtain the effective rock block region. S4. For each valid rock block region obtained in S3, calculate the geometric centroid coordinates of the rock block according to the mask pixel coordinates, and write the centroid coordinates into the centroid key point annotation file to form centroid key point training data. S5. Construct a key point detection network. Input the image data from S1 and the centroid key point training data from S4 into the key point detection network for training to obtain the rock block centroid key point detection model. S6. Match and verify the centroid key point results output by S5 with the instance mask output by S3 to obtain the final stable centroid coordinates of the rock block, thus completing the positioning.

2. The method for locating the centroid of a rock block based on instance segmentation and key point detection according to claim 1, characterized in that: The method for constructing a deep learning-based rock block instance segmentation network includes: using YOLOv8-seg as the basic framework, adding the MANet module in the feature fusion stage, and introducing the SEBlock module between C2f and the segmentation detection head.

3. The method for locating the centroid of a rock block based on instance segmentation and key point detection according to claim 1, characterized in that: Step S2 includes: S2.1 Construct the basic YOLOv8-seg network skeleton; The input image is denoted as ,in, and These represent the height and width of the input image, respectively. The backbone network extracts three layers of output features, which are represented as follows: , and ; in, , and These represent the number of channels for each layer of features; The feature fusion network Neck will , and The features are fused to obtain a multi-scale fused feature set for segmentation prediction. S2.2 Introduce the MANet module for feature enhancement on multi-scale features; Features fused at any scale Performing the MANet enhancement operation is represented as: The MANet module consists of multi-branch feature extraction and attention weighting mechanisms, represented as follows: ， ， In the formula, Indicates the first Branches pair features The transformed output; Indicates the first The feature transformation operator with each branch is used to extract rock block structure information under different receptive fields; Indicates the number of branches; Indicates the first Attention weight coefficients for each branch; Represents the normalization function; This represents the attention weight generation function; The output feature space scale of the MANet module is consistent with the input feature space scale, expressed as: Right now and downsampling scale , and No change; S2.

3. Introduce SEBlock between C2f and the segmentation detection head for channel recalibration; Inputting MANet augmented features of any scale into SEBlock yields: The calculation process for SEBlock is as follows: In the formula, Indicates the first Global average pooling results for each channel; and They represent the first The height and width of the feature map at each scale; and These represent the spatial location indices in the feature map; Represents the channel weight vector; For the Sigmoid function; and These represent the first learnable parameter and the second learnable parameter, respectively. For activation functions; This is the channel description vector obtained from global average pooling; Let be the weight of the c-th channel; Features after SE recalibration; Received As input to SegHead, it is used to output the rock block instance mask; S2.4 Output instance masks and perform end-to-end training. Will Input the segmentation prediction head SegHead, and output the category information, location regression information, and instance mask of the rock block target; The training phase employs a multi-task joint loss function: In the formula, Represents the joint loss function for multiple tasks; , , and Let represent the classification loss, bounding box regression loss, distributed regression loss, and mask loss, respectively. These are the weighting coefficients for the corresponding items.

4. The method for locating the centroid of a rock block based on instance segmentation and key point detection according to claim 1, characterized in that: The filtering and noise reduction process in S3 includes: Remove targets with a confidence level below the confidence threshold; Convert the mask into a binary image; Only retain connected components with an area greater than the area threshold; Morphological opening, closing, and hole filling operations are performed on the preserved binary mask to obtain the effective rock block region corresponding to each target.

5. The method for locating the centroid of a rock block based on instance segmentation and key point detection according to claim 1, characterized in that: The method by which S4 generates centroid keypoint training data is as follows: Based on the effective rock block regions output by S3, calculate the geometric centroid for each effective region: In the formula, These are the pixel coordinates within the mask area; This represents the number of pixels in the effective area. Write the centroid coordinates into the centroid key point annotation file, in the following format: ,in, The coordinates of the centroid key point; As a marker of validity; All centroid keypoint annotation files in each image are mapped to the corresponding rock blocks.

6. The method for locating the centroid of a rock block based on instance segmentation and key point detection according to claim 1, characterized in that: In S5, the method for obtaining the rock block centroid key point detection model is as follows: The image data of S1 and the centroid keypoint training data of S4 are used as inputs; Based on YOLOv8-pose as the basic framework for keypoint detection network, with the number of keypoints set to 1, corresponding to the centroid of the rock block, the predicted output is represented as follows: In the formula, To predict the centroid coordinates; To predict confidence levels; During training, the total loss is obtained by combining the keypoint regression loss with the detection correlation loss term, and is expressed as: In the formula, To detect branch loss; For keypoint regression loss; These are the weighting coefficients for each item.

7. The method for locating the centroid of a rock block based on instance segmentation and key point detection according to claim 1, characterized in that: The method of S6 is as follows: Based on the effective mask output by S3 and the centroid key point output by S5; If the centroid keypoint falls within the effective mask, the match is successful; If the centroid keypoint does not fall within the effective mask, calculate the shortest distance from the centroid keypoint to the mask: In the formula, These are the pixel coordinates within the mask area; The coordinates of the centroid key point; Indicates the first One effective rock block area; If satisfied Then it is assumed that the key point lies within the neighborhood, where, The neighborhood threshold; If it falls neither within the mask nor within the domain, then the critical point is discarded; At the same time, key points with confidence levels below the threshold η are removed; Finally, the output centroid is stabilized using exponential smoothing: In the formula, The output centroid is α; α is the smoothing coefficient. The centroid of the current frame; The centroid coordinates output from the previous frame; Output the coordinates of the rock block's centroid.

Images