Industrial pellet raw material instance segmentation method based on YO-SAM2

Abstract

The invention discloses a YO-SAM2-based industrial pellet raw material instance segmentation method, which belongs to the field of image detection, and carries out target detection (an efficient up-sampling module EUCB, a multi-scale heterogeneous convolution module MSHCC and fast normalized weighted fusion FFusion) by means of a YOLOv10 framework and fusion of a brand new EMBSFPN (an efficient up-sampling module EUCB, a multi-scale heterogeneous convolution module MSHCC and a fast normalized weighted fusion FFusion); and then the output bounding box is used as prompt information to be embedded into a prompt encoder of the customized SAM2 to output a high-quality segmentation result. Experiments show that the model constructed by the method well overcomes the defects of traditional instance segmentation in an industrial pellet segmentation task, an excellent mAPmask score of 92.8% is shown on a self-constructed PelletORE pellet data set, the best accuracy is realized, and the method has high practical application and popularization values.

Description

Technical Field

[0001] The invention belongs to the field of image detection, and in particular relates to an industrial pellet raw material instance segmentation method based on YO-SAM2. Background Art

[0002] In industrial production, pellets are spherical particles formed by sintering under high temperature conditions. They are often used as an important raw material for blast furnace ironmaking in industry. During the smelting process, the particle size of the pellets has an important influence on the compressive strength and ring cooling time of the pellets. Particle size detection of pellet raw materials is an important means to ensure green production, improve efficiency and reduce costs.

[0003] At present, factories mainly rely on manual sampling and screening to control the particle size of green balls within 8-18mm, but manual sampling and screening is time-consuming and labor-intensive, and has disadvantages such as low efficiency, delayed screening results, and poor real-time performance. This has become a bottleneck problem restricting the intelligent development of the entire process of pellet production. With the development of machine vision, the intelligent detection of pellet particle size distribution has gradually entered people's field of vision. For example, M. Harayama et al. proposed a method to predict the average particle size of green balls using multi-order power spectrum ratios; Thurley et al. proposed a 3D visual detection system, using laser triangulation to measure the 3D data information of green balls to obtain particle size information; Mao Chuangang used an improved threshold segmentation method to segment green balls and shadow areas from images, and used K-means clustering algorithm and watershed algorithm based on region expansion and marker control to segment green balls for particle size distribution detection.

[0004] Although the above methods have overcome the drawbacks of manual sampling and screening to a certain extent, the methods using instruments and equipment for measurement generally have the disadvantages of high cost, cumbersome measurement process, and long time consumption; the method of segmentation first and then measurement has the disadvantages of complicated operation and poor applicability. Moreover, due to the small size and dense distribution of pellets, traditional instance segmentation methods generally have defects such as varying degrees of missed calls and low edge segmentation accuracy, high data annotation costs, high training difficulty, and poor generalization. Therefore, in order to promote the low-cost and full-process intelligent development of pellet production, designing a good instance segmentation algorithm is the key. Summary of the invention

[0005] In order to solve the defects of traditional instance segmentation methods such as low segmentation accuracy, high cost and great training difficulty, a YO-SAM2-based instance segmentation method for industrial pellet raw materials was proposed. The bounding box output by the EMBS-YOLO detection module was embedded as prompt information into the prompt encoder of the customized SAM2 segmentation module to output high-quality segmentation results.

[0006] The present invention is implemented by adopting the following technical scheme: a method for segmenting industrial pellet raw materials based on YO-SAM2, comprising the following steps:

[0007] Step A, constructing a YO-SAM2 model and training it; the YO-SAM2 model includes an EMBS-YOLO detection module and a customized SAM2 segmentation module;

[0008] The EMBS-YOLO detection module is used to perform target detection to output target frame information. Its neck network adopts the EMBSFPN module design. The EMBSFPN module includes an efficient upsampling module EUCB, a multi-scale heterogeneous kernel convolution module MS_HCB and a fast normalized weighted fusion module FFusion.

[0009] The sampling process of the efficient upsampling module EUCB is as follows:

[0010] EUCB(x)=C 1×1 (SilU(BN(DWC(Up(x)))))

[0011] Among them, x represents the input feature map, C 1×1 represents 1×1 convolution, SiLU is the activation function, BN represents normalization, Up represents upsampling, and DWC represents depthwise convolution.

[0012] The multi-scale heterogeneous kernel convolution module MS_HCB first uses point-by-point convolution Conv1×1 to expand the number of feature map channels, followed by BN batch normalization and SiLU activation function, and uses heterogeneous kernel depth convolution HKDC to capture multi-scale and multi-resolution context, and finally uses another point-by-point convolution Conv1×1 and BN normalization to transform back to the original number of channels, and finally performs element-by-element addition to obtain the enhanced feature map, which is specifically expressed as follows:

[0013] F out =BN(Conv(HKDC(SiLU(BN(Conv)))))

[0014] Among them, BN means normalization, Conv means convolution, and SiLU is the activation function;

[0015] The principle of heterogeneous kernel deep convolution HKDC is as follows:

[0016] HKDC(x)=∑ ks SiLU(BN(DWC ks ))

[0017] Among them, x represents the input feature map, DWC ks It is the first layer of depthwise convolution with kernel size ks.

[0018] The fast normalized weighted fusion module FFusion adds an additional weight to each input, allowing the network to learn the importance of each input feature, and adaptively selects weighted fusion based on the importance of features at different scales. The weighted normalization principle is as follows:

[0019]

[0020] Among them, P1, P2, ..., P n For input feature maps from different scales, each feature map is assigned a learnable weight: w1, w2, ..., w n .

[0021] The customized SAM2 segmentation module integrates local + global features and adapters into the original SAM2. The customized SAM2 segmentation module includes an image encoder, a mask decoder, and a prompt encoder. The prompt encoder receives the target box information from the EMBS-YOLO detection module and encodes it to generate a prompt embedding input to the mask decoder. The mask decoder combines the image embedding from the image encoder and the prompt embedding encoded by the prompt encoder to generate a mask feature to obtain the final segmentation result;

[0022] Step B, preprocess the pellet image to be segmented, and input the preprocessed image into the EMBS-YOLO detection module and the customized SAM2 segmentation module respectively. The EMBS-YOLO detection module outputs the target box of each raw ball as prompt information embedded in the prompt encoder of the customized SAM2 segmentation module, and outputs the final segmentation result by fusing the shallow and deep features of the image encoder with the mask features of the mask decoder.

[0023] Compared with the prior art, the advantages and positive effects of the present invention are:

[0024] This scheme builds the YO-SAM2 model to detect ball mines. The YO-SAM2 model includes an EMBS-YOLO detection module and a customized SAM2 segmentation module. The EMBS-YOLO detection module is constructed by integrating the new EMBSFPN network into the YOLOv10 network, so as to improve the accuracy of dense ball positioning information while helping to make the model lightweight. The customized SAM2 segmentation module introduces the local + global feature fusion method in SAM2 to refine the output ball mask edge, and integrates the adapter to fine-tune the parameters to make it more suitable for specific downstream tasks. By converting complex instance segmentation tasks into target detection, the shortcomings of the classic instance model such as missed calls and low edge segmentation accuracy are compensated. Moreover, the computational requirements during training are low, which greatly reduces the difficulty of training. In addition, the data is changed from polygonal annotation to box annotation, and the annotation cost is greatly reduced, which can obtain accurate segmentation results while having excellent generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the flow chart of the instance segmentation method according to an embodiment of the present invention;

[0026] Figure 2 It is a structural framework diagram of the YO-SAM2 model of an embodiment of the present invention;

[0027] Figure 3 This is a framework diagram of the EMBS-YOLO detection module according to an embodiment of the present invention;

[0028] Figure 4 This is a structural diagram of the MS_HCB module in an embodiment of the present invention;

[0029] Figure 5 Customize the SAM2 segmentation module structure diagram for the embodiment of the present invention;

[0030] Figure 6 A schematic diagram showing the comparison of the segmentation effects of the method of the present invention and other traditional methods;

[0031] Figure 7 Schematic diagram of performance comparison between the method of the present invention and the YOLOv10+original SAM2 model. DETAILED DESCRIPTION

[0032] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described below in conjunction with the accompanying drawings and embodiments. In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention can also be implemented in other ways different from those described herein, and therefore, the present invention is not limited to the specific embodiments disclosed below.

[0033] Pellets have the characteristics of a large number of raw balls, similar shapes, dense distribution and adhesion. The traditional SAM2 relies heavily on manual points, boxes, and masks. Without manual prompts, it will indiscriminately segment all categories (including background). In addition, SAM2 is also insufficient in the edge details of raw balls, which will lead to a large number of irregular manual prompts during the segmentation process. The actual application is difficult and inefficient, which means that there are great limitations in directly transferring SAM2 to industrial specialized fields. In addition, the traditional YOLOv10 neck feature pyramid network is composed of a single bottom-up path and two blocks, which is not conducive to model learning and representing small target features to a certain extent, which leads to missed detection in industrial pellet detection.

[0034] Based on the shortcomings of traditional instance segmentation methods and the fact that the potential of SAM2's powerful image segmentation capabilities has not been explored in the industrial field, this embodiment creatively proposes an industrial pellet raw material instance segmentation method based on YO-SAM2, integrates the new EMBSFPN for target detection, and jointly constructs the YO-SAM2 model by combining the EMBS-YOLO detection module with the customized SAM2 module, aiming to improve the instance segmentation performance of industrial pellet images.

[0035] like Figure 1 The above is a flowchart of the industrial pellet raw material segmentation method based on YO-SAM2 according to an embodiment of the present invention, comprising the following steps:

[0036] Step A: Build and train the YO-SAM2 model, which includes the EMBS-YOLO detection module and the customized SAM2 segmentation module.

[0037] Step B: preprocessing the pellet image to be segmented; and inputting the preprocessed pellet image to be segmented into the trained YO-SAM2 model for detection and segmentation, so as to output the final segmentation result.

[0038] 1. Build and train the YO-SAM2 model

[0039] In this embodiment, the overall framework of the YO-SAM2 model is as follows Figure 2 As shown, the YO-SAM2 synthesis process is as follows:

[0040]

[0041] Among them, τ o represents the original image, {xy i ,b i} represents the bounding box and its corresponding semantic label, which is generated by the EMBS-YOLO module detector Training and reasoning results; F img , The image encoder is a custom SAM2 module. and prompt encoder Encoding produces intermediate image features and sparse cue markers; Represents the output image features, which are composed of four different mask filters T mfilter and its corresponding IoU plus The decoder of the customized SAM2 module is finally spliced ​​according to the intermediate image feature F img And the output image features Generate the final prediction mask M i .

[0042] (1) EMBS-YOLO detection module

[0043] Combination Figure 3 As shown in the figure, the EMBS-YOLO detection module includes the Backbone network, the Neck network and the Head network. The Neck part replaces the original YOLOv10 neck feature pyramid with a new Efficient Multi-Branch&Scale feature pyramid network (EMBSFPN module) to optimize the detection performance of the model. The EMBSFPN module includes an efficient upsampling module EUCB, a multi-scale heterogeneous kernel convolution module MS_HCB and a fast normalized weighted fusion module FFusion. Specifically:

[0044] This scheme uses the upsampling module EUCB to improve the upsampling process of the network, aiming to improve the resolution of the feature map while maintaining a low computational cost, thereby promoting better feature fusion and enhancing the overall detection performance. Its structure is as follows Figure 3 The efficient upsampling module EUCB first upsamples the feature map input from the backbone network by a factor of 2 (Up) to enlarge the feature map, and then enhances the enlarged feature map through 3×3 deep convolution (DWC), normalization (BN) and activation function (SiLU) (the SiLU activation function with smooth activation characteristics and more expressive feature map is selected in this embodiment), and finally uses 1×1 convolution (C 1×1 ) to reduce the number of channels and ensure that they are consistent with the high-resolution features from the skip connection. Because it uses 3×3 depth convolution instead of ordinary convolution, it can minimize the computational overhead. The process formula of the efficient upsampling module EUCB can be summarized as Equation 2, where x represents the input feature map:

[0045] EUCB(x)=C 1×1 (SilU(BN(DWC(Up(x))))) (2)

[0046] Then, we further adopt an efficient multi-scale heterogeneous kernel convolution module MS_HCB ( Figure 4 ) to enhance the features generated by the cascade expansion path, the multi-scale heterogeneous kernel convolution module MS_HCB performs deep convolution at multiple scales, and uses channel_shuffle to shuffle channels across groups. The specific principles are as follows:

[0047] The multi-scale heterogeneous kernel convolution module MS_HCB first uses point-by-point convolution (Con1×1) to expand the number of feature map channels, followed by BN batch normalization and SiLU activation function, and uses heterogeneous kernel depth convolution HKDC ( Figure 4) to capture multi-scale and multi-resolution context, and finally use another point-by-point convolution (Con1×1) and BN to transform back to the original number of channels, and finally perform element-by-element addition to obtain the enhanced feature map, which is specifically expressed as follows:

[0048] F out =BN(Conv(HKDC(SiLU(BN(Conv)))))

[0049] Among them, HKDC module ( Figure 4 ) consists of three parallel deep convolutions as the first layer, followed by BN and SiLU. The final Channel_shuffle alternately connects channels of different groups, integrates the relationship between channels and outputs high-quality feature maps. The heterogeneous kernel selection mechanism (HKSM) is integrated in the parallel deep convolution, so that the feature map uses custom convolution kernels of 1, 3, 5, 3, 5, 7, 5, 7, and 9 respectively when passing through the first layer of parallel 3×3 depth-separable convolution to adapt to the requirements of different resolutions, thereby gradually obtaining multi-scale receptive field information. The process can be expressed by formula (3). Where x represents the input feature map, DWC ks It is the first layer of depthwise convolution with kernel size ks.

[0050] HKDC(x)=∑ ks SiLU(BN(DWC ks )) (3)

[0051] Considering that the YOLOv10 feature fusion is just a simple feature addition without distinguishing the features added at the same time, and different input features have different resolutions, their contributions to the fused input features are also different, so simple addition or splicing processing is not the best operation. Therefore, in this embodiment, the EMBSFPN module uses the Fastnormalized fusion (FFusion) fusion method, that is, adding an additional weight to each input, allowing the network to learn the importance of each input feature, and adaptively selecting weighted fusion through the importance of features of different scales. At the same time, in order to avoid these weighted values ​​causing gradient instability or unbalanced feature contribution, by directly normalizing the sum of the weights, the contribution of the feature map of each scale is within a reasonable range. This method allows the network to freely adjust the importance of each feature map as needed, thereby greatly improving the efficiency and accuracy of the model. Weighted normalization can refer to formula (4):

[0052]

[0053] Among them, P1, P2, ..., P n For input feature maps from different scales, each feature map is assigned a learnable weight: w1, w2, ..., wn .

[0054] In summary, we can see that the efficient upsampling module EUCB is used to upsample the feature map (the input image is passed through the Backbone network to obtain the original image features of different scales) to improve the resolution and enhance the feature details; the multi-scale heterogeneous kernel convolution module MS_HCB is used to enhance the features generated by the cascade expansion path. The HKDC module in MS_HCB applies the heterogeneous kernel selection mechanism (HKSM), which can gradually obtain multi-scale receptive field information according to different resolutions; the feature fusion adopts fast weighted normalization fusion (FFusion) to fuse the features of different scales to generate a higher quality feature map. It can greatly reduce the number of parameters and calculations, and enhance the ability to detect small targets. The innovative EMBS-YOLO module has higher detection accuracy and speed for densely adhered small targets.

[0055] Finally, the high-quality fused feature map output by the EMBSFPN module is input into the Head network to output the final detection result (target box information), including the calculation of bounding box and category prediction. The fused features of the neck are input, and One-to-Many Head is used in the training process to generate multiple prediction results for each target object. This can provide more supervision signals and improve the learning accuracy of the model. One-to-One Head is used in the reasoning process: a single best prediction result is generated for each target object. This can save the NMS (non-maximum suppression) step, reduce processing time, and improve overall efficiency.

[0056] (2) Customized SAM2 segmentation module

[0057] Combination Figure 5 As shown in the figure, it is a structural diagram of the customized SAM2 segmentation module. The customized SAM2 segmentation module includes an image encoder, a mask decoder, and a prompt encoder. The original image is input into the image encoder, and is first scaled geometrically to 1024 on the long side. If the short side is not enough, it is padded to obtain an image of the same dimension (1024×1024). The customized SAM2 segmentation module can finely segment the target object according to the box prompt embedded in the prompt encoder. The prompt encoder receives the target box information (upper left corner coordinates, lower right corner coordinates) from the EMBS-YOLO detection module and encodes it to generate a prompt embedding (size 2×256) and input it into the mask decoder. The mask decoder combines the image features from the image encoder and the prompt information encoded by the prompt encoder to generate a segmentation mask.

[0058] Since the edges of the raw balls are tortuous and not smooth and have serious occlusion, the original SAM2 is insufficient in processing edge details. In this embodiment, the original SAM2 is specially customized (local + global feature fusion and adapter integration into the original SAM2) to improve the segmentation accuracy. In addition, memory modules such as memory encoders that do not contribute much to the results are discarded. Specifically:

[0059] The Hiera used by the image encoder is a layered visual Transformer architecture. Each layer block is used to process features of different scales. The shallow layer contains local features such as edge details of small objects, and the deep layer contains global context information of the image. In order to obtain accurate segmentation results, this embodiment extracts local edge features after the first global attention block of the image encoder, then fuses them with the last layer of global features, and then adds them to the mask features of the mask decoder to obtain new, high-quality features ( Figure 5 (a) Output of the segmentation result. The early layer and final layer encoder features are upsampled to a spatial size of 256×256 by transposed convolution. Then, after a simple convolution process, the features of the three stages are summed up in an additive manner to obtain the final high-quality features.

[0060] In order to make SAM2 more suitable for specialized downstream tasks, this embodiment also integrates the adapter into the image encoder to achieve efficient fine-tuning of parameters and retrain the entire model. As a ViT architecture, Hiera has a large number of parameters. If it is fully fine-tuned, the memory requirement is extremely high. Therefore, this embodiment first freezes all the backbone network parameters and adds an adapter before each multi-scale block, such as Figure 5 As shown in (b), the adapter parameters are fine-tuned while maintaining the original structure. The adapter adopts a classic structure, such as Figure 5 (c) Contains a linear layer for downsampling, a GeLU activation function, and another linear layer for upsampling and a final GeLU activation function.

[0061] In this embodiment, considering that there is currently no public data set available for pellets, the application of deep learning algorithms to industrial pellets does not seem to be so smooth, so this solution collects data at the production site to create a pellet dataset---PelletORE. Specifically, in the pelletizing machine equipment, when the particle size of the raw balls reaches a certain size, they will automatically be discharged over the edge of the disc pelletizing machine to form a discharging area. The raw balls in the discharging area fall quickly, and the discharge process of the raw ball particles is random, the falling paths are inconsistent, and there is an uneven distribution phenomenon caused by the adhesion of pellet particles. Therefore, an industrial camera is installed above the on-site discharging area, and in order to reduce the interference of the external environment on the image, an industrial light source is used for lighting to improve the image quality. A total of 533 images (about 53,300 green balls) of green balls in the discharge area were collected under different lighting conditions, including no fog, light fog, dense fog, and with an interval of 25 seconds. The collected images were preprocessed (image resizing and data enhancement, etc.), and the ball clusters in the images were annotated with polygonal masks using the labelme annotation tool. The dataset was divided into training set, validation set, and test set in a ratio of 8:1:1, which reduced overfitting to a certain extent and enhanced the robustness and generalization of the model, so as to train and optimize the model.

[0062] In addition, during EMBS-YOLO training and optimization, the labels need to be converted into box form. This transformation involves the minimum and maximum x and y coordinates of the polygon mask.

[0063] 2. Apply the trained YO-SAM2 model to pellet segmentation

[0064] For the pellets to be segmented, they are first preprocessed, including image resizing and image enhancement. This is a relatively mature technical means and will not be elaborated on here. The preprocessed images to be segmented are respectively input into the trained EMBS-YOLO detection module and the customized SAM2 segmentation module. The EMBS-YOLO detection module outputs the target box of each raw ball as prompt information embedded in the prompt encoder of the customized SAM2 segmentation module; the customized SAM2 segmentation module outputs the final segmentation result by fusing image features at different stages (the shallow and deep features of the image encoder and the mask features of the mask decoder). Specifically:

[0065] (1) Detection stage

[0066] Due to the small volume and dense distribution of pelletized ore raw materials, the architectural strategy of YOLOv10 neck for small target detection is composed of a single top-down path and two blocks, which tends to merge feature maps of the same scale and lacks comprehensive processing and fusion of multi-scale information of different resolutions, which is not conducive to model learning and representation of small target features, which will lead to missed calls and poor boundary details during segmentation. Based on the above problems, this embodiment uses the proposed new efficient, multi-scale, multi-branch feature pyramid network (EMBSFPN) on the basis of the weighted bidirectional feature pyramid network to achieve the purpose of improving the small target detection performance while improving the computational efficiency.

[0067] Detection process: The image is input into the backbone network Backbone of the improved EMBS-YOLO module. The Backbone part uses EfficientRepV3 to extract features from the image and convert the image into a feature representation with rich semantic information. Then, the feature maps of different levels output by the backbone network Backbone are input into the EMBSFPN module (Neck) to enhance and fuse the features of different scales extracted by the backbone network to generate high-quality feature maps, and pass them to the output part of the network (Head). First, the number of channels of the feature map is unified, and then the MS_HCB extraction enhancement, EUCB upsampling, and FFusion feature fusion of different scale features are repeatedly performed from bottom to top and from top to bottom to generate high-quality feature maps, which are input into the Head detection head and the detection results are output. The specific combination Figure 3 As shown:

[0068] ① The pellet image is input into the Backbone network, which performs preliminary feature extraction on the image through the C2f module, downsampling module, compact inversion block (CIB), local self-attention module (PSA), etc. (i.e., a series of convolutional layers, normalization layers, and activation function processing). The output not only contains the local details of the pellet (edges, textures, shapes, etc.), but also contains multi-scale original image feature maps (P2, P3, P4, P5) of large-scale contextual information. Among them, the backbone network uses the original network structure of YOLOv10 to extract image features. For example, the compact inversion block CIB module extracts spatial and channel features through deep separable convolution, reduces computational costs and maintains feature expression. Large kernel convolution expands the receptive field and improves the global perception of larger targets. The local self-attention module PSA module enhances global feature modeling and optimizes the detection of complex scenes and large targets.

[0069] ② The multi-scale original image feature maps (P2, P3, P4, P5) output by the backbone network are used as the input of the Neck network (EMBSFPN module). First, the number of channels is adjusted to 256 through the conv convolution operation, and then the bidirectional path (bottom-up and top-down) is used to enhance and fuse the features of different scales. Finally, high-quality feature information from multiple information (such as position, edge information, category, context information, etc.) at different levels and scales is output.

[0070] Continue to refer Figure 3 , bottom-up path: p5 is first fused with the downsampled p4 by FFusion feature, and then enhanced by MS_HCB feature to obtain feature S1; then S1 is upsampled by EUCB, fused with p4 and the downsampled p3 by FFusion feature, and enhanced by MS_HCB feature to obtain feature S2; S2 is upsampled by EUCB, fused with p3 and the downsampled p2 by FFusion, and then enhanced by MS_HCB to obtain S3. Switch to the top-down path: S3 is fused with the upsampled S2 by EUCB by FFusion feature, and enhanced by MS_HCB feature to obtain feature S4; then S3 and S4 are downsampled, and S1 is upsampled by EUCB and S2 is fused with FFusion feature, and enhanced by MS_HCB feature to obtain feature S5; S2 and S5 are downsampled, and then fused with S1 by FFusion feature, and enhanced by MS_HCB feature to obtain feature S6; S4, 5, and 6 (multiple information at different levels and scales) are finally used as the input of the Head.

[0071] ③ The high-quality feature map output by the neck is used as the input of the head. The high-level features are extracted through the convolution layer, and then through category prediction, bounding box regression, etc., the box bounding box and semantic label information of each raw ball in the pellet image are finally output.

[0072] It should be noted here that the shape of the bounding box output by EMBS-YOLO is usually (batch_size, num_boxes, 4): batch_size is the batch size of the input, num_boxes is the number of boxes in each input, and 4 means that each box consists of four coordinates (x1, y1, x2, y2), where (x1, y1) is the coordinate of the upper left corner of the box, and (x2, y2) is the coordinate of the lower right corner of the box. These boxes are input to the prompt encoder of the custom SAM2, and the coordinates of the box are converted to the shape of (batch_size*num_boxes, 2, 2) by reshape(-1, 2, 2), and then the forward_with_coords method is used to positionally encode these corner points to obtain the position information of each corner point. Finally, the prompt embedding is output according to the position of the corner point:

[0073] P embedding =forward(reshape(B))

[0074] Among them, B represents the bounding box of the input, forward represents the encoding method, encoding the input prompt, and reshape represents the function that changes the shape of the tensor.

[0075] (2) Segmentation stage:

[0076] First, the original image is input into the image encoder of the pre-trained custom SAM2 segmentation module, and the image (c, h, w) is input. The image is scaled to 1024 according to the long side, and the short side is padded if it is not enough, and the image (c, 1024, 1024) is obtained. The features are encoded and extracted through each layer of the hierarchical Transformer architecture (Hiera). The adapter (Adapter) will perform UP upsampling, activation function (GeLU), downsampling (Down), and activation function (GeLU) on these features, mapping the original general image features to the specific representation required for the pellet segmentation task, and finally outputting the image embedding (image embedding) containing rich semantic information and detailed features of the image:

[0077] I embedding =∑GeLU(Down(GeLU(UP(f out )))

[0078] Among them, f out Represents the features extracted by each layer of Hiera.

[0079] Then, the image embedding and the hint embedding are used as the input of the mask decoder to generate the mask feature f through self-attention and cross-attention. mask (Self-attention mainly helps the model understand the spatial relationship in the image, and cross-attention generates mask features through the interaction between the target area information and the image features.) At the same time, the shallow features E_Features and the deepest features F_Features in the image encoder are upsampled to 256×256 through transposed convolution TransConv, and they are convolved with the mask features MS_Features of the mask decoder. The three types of features are fused in an additive manner to obtain a 256×256 high-quality feature map for generating accurate segmentation results:

[0080] Hfeatures=Add(UP TransConv (E_Features, F_Features), Conv(E_Features))

[0081] Among them, Hfeatures represents a high-quality feature map, and Add represents an element-by-element addition operation.

[0082] 3. Experimental verification

[0083] This embodiment selects the Ubuntu 20.04 system, and the hardware configuration is equipped with an NVIDIA_GeForce_RTX3090 graphics card and a CPU processor (Intel(R)Xeon(R)Gold 6330 CPU@2.00GHz) for experimental verification. The experimental model is based on the PyTorch deep learning framework, and the experimental environment selects python 3.8, torch and CUDA versions are 2.0.0 and 11.8 respectively. For EMBSYOLO and SAM2, yolov10m and sam2_hiera_tiny pre-trained models were used for model training.

[0084] In this scheme, the average precision mAP of the predicted box and the segmentation mask is used box , mAP mask , The model's inference speed (FPS) is used as the main evaluation indicator. AP represents the area enclosed by the Precision and Recall curves, which can be calculated from the TP, FP, TN, and FN of the confusion matrix. mAP50 is calculated by the average precision (AP) of different categories at a 50% IoU threshold. For details, see formulas (5)(6)(7). Where N is the number of categories in the dataset. The larger the mAP value, the more accurate the predicted instance mask, and therefore the better the instance segmentation performance.

[0085]

[0086] In order to evaluate the effectiveness and robustness of the YO-SAM2 model, this example comprehensively compares YO-SAM2 with traditional instance segmentation methods on the PelletORE dataset, including the two-stage method Mask R-CNN, the single-stage methods SOLOv2, YOLACT, and YOLO v8-seg. Table 1 shows the comparison of this method with several other methods under different indicators. Figure 6 The following is a visual comparison of the effects of the corresponding methods. It can be seen that although Mask R-CNN has complete segmentation edges, its missed recall rate is high, and the corresponding mAP mask SOLOv2 has a large number of missed calls and low edge segmentation accuracy, which is reflected in its mAP index. mask In contrast, the YO-SAM2 proposed in this scheme produced excellent results in pellet segmentation, with segmentation accuracy mAPmask and The results are 92.8% and 92.6% respectively, with an inference speed of 41.2fps / s. The instance segmentation effect has no missed calls, and the edges and contours of the target mask are clear and complete. It should be noted that the mAP values ​​of YOLACT and YOLOv8-seg are artificially high. Figure 6 In the visualization of the segmentation effect, their target edge segmentation is extremely inconsistent with the actual raw ball contour, but its mAP mask The values ​​are as high as 98.8% and 99.5% respectively, which do not reflect the true segmentation accuracy and are not completely applicable when actually evaluating their segmentation accuracy. During the raw material mixing stage, a certain amount of water is usually added to the pelletizing equipment to achieve the optimal humidity level to help the binder to fully function, so fog often forms at the pelletizing site. This also places certain requirements on the robustness of the model. Figure 6 (a, b, c) are the spherical segmentation effects of each model under different light intensities. Figure 6 (d, e) are the pellet segmentation effects of each model under light fog and dense fog, respectively. It can be seen that although the model of this scheme has slight missed calls in dense fog, its segmentation effect is far better than other models and can meet the pellet segmentation needs.

[0087] Table 1 Performance comparison between YO-SAM2 and classic strength segmentation methods

[0088]

[0089] In addition, this solution deleted each optimization module and conducted a series of experiments on the pelletized ore dataset to evaluate the impact of each module in EMBSFPN on the model performance through mAP, mAP50, and FPS performance indicators. mask , The FPS performance index is used to evaluate the impact of customized SAM2 strategies on model performance. The performance index results are shown in Tables 2 and 3, and the corresponding experimental visualization results are shown in Tables 2 and 3. Figure 7 shown.

[0090] As can be seen from Table 2, each improved module has significantly improved the algorithm performance: using EUCB upsampling improves the model mAP by 0.6% compared to the baseline model; the application of the MS_HCB module improves the model mAP by 1.1%; selecting FFusion as the feature fusion method increases the model speed to 45fps / s while also improving the average accuracy. When the four parts are fused into EMBSFPN and applied to YOLOv10, the model achieves the best overall performance and can meet the model's expected detection requirements. As can be seen from Table 3: The introduction of the local + global feature fusion method increases the model's mAP maskImproved by 0.5%; integrating the adapter into the image encoder, the model’s mAP mask There is also a 0.3% improvement; finally, the two customized strategies are integrated into SAM2, and the YO-SAM2 model achieves 92.8% mAP mask The model achieves the best comprehensive performance and can meet the segmentation requirements of industrial pellet raw material particle size statistics.

[0091] Table 2 Model performance of different modules of EMBS-YOLO

[0092]

[0093] Table 3 Model performance of different modules of YO-SAM2

[0094]

[0095] This solution explores the potential of the visual large model SAM2 in the task of pellet image segmentation, and uses the new EMBS-YOLO target detector and the customized SAM2 for joint segmentation, which facilitates the next step of particle size analysis. Combined with experimental verification, it can also be seen that the method proposed in this solution makes up for the shortcomings of the classic instance model such as missed calls and low edge segmentation accuracy; and converts the complex instance segmentation task into target detection, with low computational requirements during training, greatly reducing the difficulty of training. In addition, the data is changed from polygonal annotation to box annotation, and the annotation cost is greatly reduced.

[0096] The above description is only a preferred embodiment of the present invention and does not limit the present invention in other forms. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. The industrial pellet raw material instance segmentation method based on YO-SAM2 is characterized by: The following steps are involved: Step A, constructing a YO-SAM2 model and training it; the YO-SAM2 model includes an EMBS-YOLO detection module and a customized SAM2 segmentation module; The EMBS-YOLO detection module is used to perform target detection to output target frame information. Its neck network adopts the EMBSFPN module design. The EMBSFPN module includes an efficient upsampling module EUCB, a multi-scale heterogeneous kernel convolution module MS_HCB and a fast normalized weighted fusion module FFusion. The customized SAM2 segmentation module integrates local + global features and adapters into the original SAM2. The customized SAM2 segmentation module includes an image encoder, a mask decoder, and a prompt encoder. The prompt encoder receives the target box information from the EMBS-YOLO detection module and encodes it to generate a prompt embedding input to the mask decoder. The mask decoder combines the image embedding from the image encoder and the prompt embedding encoded by the prompt encoder to generate a mask feature to obtain the final segmentation result; Step B, preprocess the pellet image to be segmented, and input the preprocessed image into the EMBS-YOLO detection module and the customized SAM2 segmentation module respectively. The EMBS-YOLO detection module outputs the target box of each raw ball as prompt information embedded in the prompt encoder of the customized SAM2 segmentation module, and outputs the final segmentation result by fusing the shallow and deep features of the image encoder with the mask features of the mask decoder.

2. The industrial pellet raw material instance segmentation method based on YO-SAM2 according to claim 1 is characterized in that: In step A, the sampling process of the efficient upsampling module EUCB is as follows: EUCB(x)=C 1×1 (SilU(BN(DWC(Up(x))))) Among them, x represents the input feature map, C 1×1 represents 1×1 convolution, SiLU is the activation function, BN represents normalization, Up represents upsampling, and DWC represents depthwise convolution.

3. The industrial pellet raw material instance segmentation method based on YO-SAM2 according to claim 1, characterized in that: In step A, the multi-scale heterogeneous kernel convolution module MS_HCB first uses point-by-point convolution Conv1×1 to expand the number of feature map channels, followed by BN batch normalization and SiLU activation function, and uses heterogeneous kernel depth convolution HKDC to capture multi-scale and multi-resolution context, and finally uses another point-by-point convolution Conv1×1 and BN normalization to transform back to the original number of channels, and finally performs element-by-element addition to obtain an enhanced feature map, which is specifically expressed as follows: F out =BN(Conv(HKDC(SiLU(BN(Conv))))) Among them, BN means normalization, Conv means convolution, and SiLU is the activation function; The principle of heterogeneous kernel deep convolution HKDC is as follows: HKDC(x)=∑ ks SiLU(BN(DWC ks )) Among them, x represents the input feature map, DWC ks It is the first layer of depthwise convolution with kernel size ks.

4. The industrial pellet raw material instance segmentation method based on YO-SAM2 according to claim 1, characterized in that: In step A, the fast normalized weighted fusion module FFusion adds an additional weight to each input, allowing the network to learn the importance of each input feature, and adaptively selects weighted fusion based on the importance of features at different scales. The weighted normalization principle is as follows: Among them, P1, P2, ..., P n For input feature maps from different scales, each feature map is assigned a learnable weight: w1, w2, ..., w n .

5. The industrial pellet raw material instance segmentation method based on YO-SAM2 according to claim 1, characterized in that: In step A, the prompt embedding generated by the prompt encoder is represented as follows: P embedding =forward(reshape(B)) Among them, B represents the bounding box of the input, forward represents the encoding method for encoding the input prompt, and reshape represents the function that changes the shape of the tensor.

6. The industrial pellet raw material instance segmentation method based on YO-SAM2 according to claim 5 is characterized in that: In step A, the image embedding generated by the image encoder is represented as follows: I embedding =∑GeLU(Down(GeLU(UP(f out ))) Among them, f out Represents the features extracted from each layer of the hierarchical Transformer architecture Hiera, GeLU represents the activation function, UP represents upsampling, and Down represents downsampling.

7. The industrial pellet raw material instance segmentation method based on YO-SAM2 according to claim 6 is characterized in that: In step A, the final segmentation result is expressed as follows: Hfeatures=Add(UP TransConv (E_Features,F_Features),Conv(E_Features)) Among them, Hfeatures represents high-quality feature maps, Add represents element-by-element addition operations, and UP TransConv It represents upsampling by transposed volume, E_Features represents shallow features, and F_Features represents the deepest features.

8. The industrial pellet raw material instance segmentation method based on YO-SAM2 according to claim 1, characterized in that: In step A, the process of the EMBS-YOLO detection module outputting target frame information is as follows: The preprocessed image is input into the backbone network Backbone of the EMBS-YOLO detection module to convert the image into a feature representation with rich semantic information. Then, the feature maps of different levels output by the backbone network Backbone are input into the EMBSFPN module to enhance and fuse the features of different scales extracted by the backbone network. The EMBSFPN module first unifies the number of channels of the feature map, and then repeatedly performs MS_HCB feature enhancement, EUCB upsampling, and FFusion feature fusion on the features of different scales from bottom to top and top to bottom to generate high-quality feature maps, which are input into the Head detection head of the EMBS-YOLO detection module and output the detection results.

Images