Scrap steel instance segmentation method based on large model and related equipment

Abstract

The invention discloses a waste steel instance segmentation method based on a large model and related equipment, and the method comprises the steps: carrying out the multi-angle photographing of a waste steel pile, and obtaining a high-quality sample photo; performing instance segmentation labeling on the scrap steel material type in the obtained sample photo to obtain labeling information, converting the labeling information into a Json file in a COCO data format, and dividing the Json file into a data set; inputting the data set into a waste steel instance segmentation model, and carrying out training and parameter optimization based on a comprehensive loss function to obtain a trained waste steel instance segmentation model; performing object mask segmentation by using the trained waste steel instance segmentation model to output a regional mask; according to the method, an area mask output by a waste steel instance segmentation model is converted into a Json file in a COCO data format, waste steel monomer image visualization, material type information storage and waste steel monomer instance segmentation are carried out in a visualization module by taking an image as a reference, and the zero sample segmentation capability and generalization performance of the model are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence computer vision technology, and in particular to a large model-based scrap steel instance segmentation method and related equipment. Background Art

[0002] One of the core challenges facing steel companies in scrap steel recycling and reuse is scrap steel identification, that is, distinguishing individual scrap steel parts from densely stacked scrap steel piles and storing the corresponding material type information. At present, scrap steel identification mainly relies on quality inspectors to roughly identify and judge the instances in the scrap steel pile through visual inspection methods and work experience, which has problems such as high labor intensity, low safety factor, strong subjectivity and inaccurate identification. With the continuous development of artificial intelligence, intelligent scrap steel identification systems that use machines to replace traditional manual labor have emerged. However, in actual scrap steel recycling scenarios, different scrap steel parts have different shapes and sizes, and it is difficult to divide clear category information. How to combine artificial intelligence algorithms to process scrap steel images to achieve scrap steel instance segmentation and provide clear and complete single-level scrap steel part information is the key to subsequent intelligent grading of scrap steel.

[0003] At present, most intelligent scrap steel identification solutions are based on target detection models. However, target detection models can usually only provide bounding boxes and object category information indicating the target location, and cannot accurately depict the boundaries of each scrap steel unit, affecting the accuracy of subsequent scrap steel grading. Based on the image segmentation method, some scholars achieve the distinction of target individuals by pixel-by-pixel classification, which can provide clear and complete contour information for each steel part. It mainly includes semantic segmentation and instance segmentation. The former groups objects according to the classification results of each pixel in the image to obtain the mask of a certain category of objects, but does not distinguish between different instances of the same category. It cannot obtain the single-level material type information required for scrap steel detection. Therefore, the method based on semantic segmentation is difficult to apply. The method based on instance segmentation can distinguish different instances belonging to the same category, but considering that the size range of some scrap steel instances is relatively wide, the segmentation model directly trained is prone to incomplete object masks and fragmented segmentation, and the segmentation effect is not good. Summary of the invention

[0004] In order to overcome the defects of the above-mentioned prior art, the purpose of the present invention is to provide a scrap steel instance segmentation method and related equipment based on a large model, so as to solve the technical problems in the prior art of unclear object edge segmentation in the instance segmentation of scrap steel images with severe stacking and lack of clear category information, fragmented segmentation of large-size scrap steel parts, and poor generalization of trained models.

[0005] The present invention is achieved through the following technical solutions:

[0006] In a first aspect, the present invention provides a method for instance segmentation of scrap steel based on a large model, comprising:

[0007] Take photos of scrap steel piles from multiple angles to obtain high-quality sample photos;

[0008] Perform instance segmentation and annotation on the scrap steel types in the obtained sample photos to obtain annotation information, convert the annotation information into a Json file in the COCO data format, and divide the Json file into data sets;

[0009] The data set is input into the scrap steel instance segmentation model for training and parameter optimization based on the comprehensive loss function to obtain the trained scrap steel instance segmentation model; the trained scrap steel instance segmentation model is used to perform object mask segmentation and output the region mask;

[0010] The regional mask output by the scrap steel instance segmentation model is converted into a Json file in the COCO data format. Based on the image, the scrap steel monomer image is visualized in the visualization module, the material type information is stored, and the scrap steel monomer instance segmentation is completed.

[0011] Preferably, CVAT annotation software is used to perform instance segmentation annotation on the collected scrap steel images, and the annotation information is saved as a Json file, wherein the Json file divides the data set into a training set, a validation set and a test set in proportion.

[0012] Preferably, the specific process of inputting the data set into the scrap steel instance segmentation model for training and parameter optimization based on the comprehensive loss function is as follows:

[0013] S1, scale the scrap steel image to 1024×1024 size and enhance the training samples by random horizontal flipping;

[0014] S2, using the image encoder pre-trained on big data as the backbone feature extraction network Γ img-enc , process the scrap steel image I into a feature map F img And output the intermediate layer feature F of the model i , the formula is as follows:

[0015] F img , F i =Γ img-enc (I)

[0016] in, represents the input image, F img Represents the final output image features, F i , i∈{1,…,k} represents multiple intermediate layer features;

[0017] S3, in the feature aggregation module Γ img-agg In the example, the intermediate layer features F extracted by the image encoder are iTo perform feature fusion, the spatial information of the low-level feature map is used while combining the semantic information of the high-level feature map. The formula is as follows:

[0018] F agg =Γ img-agg ({F i})

[0019] Among them, {F i} represents the selected intermediate layer features; F agg Represents the multi-scale features obtained after fusion;

[0020] S4, input the multi-scale feature map into the region proposal network Γ RPN In the above example, a target candidate box is generated on each feature layer to improve the detection capability of scrap steel monomers of different sizes. The formula is as follows:

[0021] p j =Γ RPN (F agg )

[0022] Among them, F agg Represents feature maps of different sizes, p j Represents the generated candidate region;

[0023] S5, for the feature map F agg Perform position encoding, and then use the precise region of interest pooling layer instead of the traditional RoIAlign operation to convert the non-uniformly sized target candidate boxes p output by the RPN network into j Pooling to a fixed-size feature map, the formula is as follows:

[0024] o j =Γ PrRoI (E pos (F agg ), p j )

[0025] Among them, E pos Representative calculation feature map F agg Sinusoidal position encoding in x and y directions to increase position information; o j Represents Γ PrRoI The output of the network;

[0026] S6, Γ PrRoI The output of the network is input into the classification branch Γ cls and locate branch Γ reg , perform target classification and bounding box regression, and encode prompt information;

[0027] S7, the target bounding box b j After encoding, it is used as prompt information and image features F aggInput together with the frozen SAM model mask decoder Γ m-dec Perform scrap steel instance segmentation as follows:

[0028] m j =Γ m-dec (F agg , b j )

[0029] Among them, m j A binary mask of size H×W representing each scrap steel instance generated; the mask prediction branch is trained based on the binary cross entropy loss function.

[0030] Furthermore, in S2, the model adopts the ViT-B model, and the specific process is as follows:

[0031] The input image first passes through a convolution layer with a convolution kernel size of 16×16, a step size of 16, and 768 convolution kernels to obtain a feature map that is 16 times downsampled from the image.

[0032] A Class Token is added to the feature map for classification prediction, and the position code is added before it is sent to 12 stacked Transformer Blocks. Eight of these blocks are self-attention modules based on Window Partition. The feature maps are divided into 14×14 blocks within these blocks, and the other four blocks ([2, 5, 8, 11]) perform global attention.

[0033] The final image features and the intermediate features of each layer are output through two layers of convolution.

[0034] Furthermore, in S3, in the feature aggregation module Γ img-agg In the example, the intermediate layer features F extracted by the image encoder are i The specific process of feature fusion, which utilizes the spatial information of the low-level feature map and combines the semantic information of the high-level feature map, is as follows:

[0035] Select some intermediate features output by the image encoder according to the preset information, and reduce the number of channels by 16 times through a 1×1 convolution layer;

[0036] Use a convolutional layer with a kernel of 3×3 and a stride of 2 for downsampling;

[0037] Finally, two 3×3 convolutional layers and one 1×1 convolutional layer are used as fusion convolutional layers to output three feature maps F of different sizes. agg .

[0038] Furthermore, in S4, the multi-scale feature map is input into the region candidate network Γ RPNIn , the specific process of generating target candidate boxes on each feature layer is as follows:

[0039] L1, based on the three preset aspect ratios and pixel change scales, 9 anchor boxes of different sizes are generated with each pixel on each feature map as the center, so that the network can adapt to targets of different sizes;

[0040] L2, the classification layer is trained based on the cross entropy loss function to determine whether each anchor box contains a scrap steel instance; at the same time, the regression layer is trained based on the SmoothL1Loss loss function to fine-tune the center point coordinates and length and width of the anchor box so that the candidate box and the real detection box of the corresponding object are closer;

[0041] L3, according to the classification results, non-maximum suppression (NMS) is used to filter the candidate frames, and the top M candidate frames that are most likely to contain scrap steel monomers at each feature level are retained as the candidate regions of interest for the final output.

[0042] Preferably, training and parameter optimization are performed based on the comprehensive loss function to obtain the specific formula of the trained scrap steel instance segmentation model as follows:

[0043] Loss = αL rpn_cls +βL rpn_bbox +γL bbox +δL cls +εL mask

[0044] Among them, {α, β, γ, δ, ε} represents the weights between different types of loss functions; L rpn_cls and L rpn_bbox Respectively represent the binary cross entropy loss and positioning loss of the RPN network; L cls Represents the final predicted category c j The cross entropy loss between the real category and the object; L bbox Represents the predicted detection box b j and the SmoothL1Loss loss between the real detection frame of the object; L mask Represents the object mask m output by the mask decoder j The binary cross entropy loss between the pixels in the object's true mask and the pixels in the object's true mask.

[0045] In a second aspect, the present invention further provides a large model-based scrap steel instance segmentation system, comprising:

[0046] The sample acquisition module is used to take photos of the scrap steel pile from multiple angles to obtain high-quality sample photos;

[0047] A data processing module is used to perform instance segmentation and annotation on the types of scrap steel in the obtained sample photos to obtain annotation information, convert the annotation information into a Json file in the COCO data format, and divide the Json file into data sets;

[0048] The model training module is used to input the data set into the scrap steel instance segmentation model for training and parameter optimization based on the comprehensive loss function to obtain the trained scrap steel instance segmentation model; the trained scrap steel instance segmentation model is used to perform object mask segmentation and output the regional mask;

[0049] The image segmentation module is used to convert the regional mask output by the scrap steel instance segmentation model into a Json file in the COCO data format. Based on the image, the scrap steel monomer image is visualized in the visualization module, the material type information is stored, and the scrap steel monomer instance segmentation is completed.

[0050] In a third aspect, the present invention also provides a mobile terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the large model-based scrap steel instance segmentation method as described above when executing the computer program.

[0051] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the large model-based scrap steel instance segmentation method as described above.

[0052] Compared with the prior art, the present invention has the following beneficial technical effects:

[0053] The invention provides a scrap steel instance segmentation method based on a large model. The method comprises the following steps: taking photos of a scrap steel pile from multiple angles to obtain high-quality sample photos; performing instance segmentation and annotation on the scrap steel material types in the obtained sample photos to obtain annotation information, converting the annotation information into a Json file in a COCO data format, and dividing the Json file into a data set; inputting the data set into a scrap steel instance segmentation model to perform training and parameter optimization based on a comprehensive loss function to obtain a trained scrap steel instance segmentation model; performing object mask segmentation and outputting a regional mask using the trained scrap steel instance segmentation model; converting the regional mask output by the scrap steel instance segmentation model into a Json file in a COCO data format, taking the image as a reference, performing visualization of the scrap steel monomer image in a visualization module, storing the material type information, and completing the scrap steel monomer instance segmentation. The present invention uses the ViT model trained based on big data as the backbone network to extract scrap steel image features, combines the region proposal network and the precise region of interest pooling layer to generate target candidate boxes and use them as prompt information for the mask decoder, thereby enhancing the model's zero-sample segmentation capability and enabling it to adapt to the segmentation of scrap steel instances of unknown categories in complex scenarios, providing an important basis for subsequent intelligent grading and recognition of scrap steel.

[0054] Furthermore, the present invention utilizes the powerful feature extraction capability of the ViT model to realize feature extraction of scrap steel images, and then fuses the feature maps of each level to effectively utilize low-level spatial information and high-level semantic information. Anchor frames of different sizes are generated for multi-scale features, so that the network has good detection capabilities for steel parts of various sizes. Compared with the prior art, the coordinate quantization loss is avoided by using a precise region of interest pooling layer, and a more accurate target bounding box regression result is obtained. The target bounding box is input into the mask decoder as the prompt information for instance segmentation, which makes up for the problem that the existing segmentation method is prone to incomplete masking and fragmented segmentation in the segmentation of large-size parts, thereby effectively improving the model's zero-sample segmentation capability and generalization performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a flow chart of a method for segmenting a scrap steel material instance based on a large model in an embodiment of the present invention;

[0056] Figure 2 This is an example diagram of the network structure for segmenting a scrap steel monomer in an embodiment of the present invention;

[0057] Figure 3 It is a schematic diagram of the principle of the scrap steel material instance segmentation system based on the large model in an embodiment of the present invention;

[0058] In the figure: 1-sample acquisition module; 2-data processing module; 3-model training module; 4-image segmentation module. DETAILED DESCRIPTION

[0059] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0060] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0061] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0062] The purpose of the present invention is to provide a scrap steel instance segmentation method and related equipment based on a large model, so as to solve the technical problems in the prior art of unclear object edge segmentation in the instance segmentation of scrap steel images with severe stacking and lack of clear category information, fragmented segmentation of large-size scrap steel parts, and poor generalization of trained models.

[0063] See also Figure 1 The present invention provides a method for instance segmentation of scrap steel based on a large model, comprising the following process:

[0064] Step 1, take photos of the scrap steel pile from multiple angles to obtain high-quality sample photos;

[0065] Step 2: perform instance segmentation and annotation on the scrap steel material types in the obtained sample photos to obtain annotation information, convert the annotation information into a Json file in the COCO data format, and divide the Json file into data sets;

[0066] Specifically, CVAT annotation software is used to perform instance segmentation annotation on the collected scrap steel images, and the annotation information is saved as a Json file. The Json file divides the data set into a training set, a validation set, and a test set in proportion.

[0067] Step 3, input the data set into the scrap steel instance segmentation model for training and parameter optimization based on the comprehensive loss function to obtain a trained scrap steel instance segmentation model; use the trained scrap steel instance segmentation model to perform object mask segmentation and output region mask;

[0068] Specifically, according to Figure 2 As shown in the figure, the specific process of inputting the data set into the scrap steel instance segmentation model for training and parameter optimization based on the comprehensive loss function is as follows:

[0069] S1, scale the scrap steel image to 1024×1024 size and enhance the training samples by random horizontal flipping;

[0070] S2, using the image encoder pre-trained on big data as the backbone feature extraction network Γ img-enc , process the scrap steel image I into a feature map F img And output the intermediate layer feature F of the model i , the formula is as follows:

[0071] F img , F i =Γ img-enc (I)

[0072] in, represents the input image, F img Represents the final output image features, F i , i∈{1,…,k} represents multiple intermediate layer features;

[0073] Among them, the model adopts the ViT-B model, and the specific process is as follows:

[0074] The input image first passes through a convolution layer with a convolution kernel size of 16×16, a step size of 16, and 768 convolution kernels to obtain a feature map that is 16 times downsampled from the image.

[0075] A Class Token is added to the feature map for classification prediction, and the position code is added before it is sent to 12 stacked Transformer Blocks. Eight of these blocks are self-attention modules based on Window Partition. The feature maps are divided into 14×14 blocks within these blocks, and the other four blocks ([2, 5, 8, 11]) perform global attention.

[0076] The final image features and the intermediate features of each layer are output through two layers of convolution.

[0077] S3, in the feature aggregation module Γ img-agg In the example, the intermediate layer features F extracted by the image encoder are iTo perform feature fusion, the spatial information of the low-level feature map is used while combining the semantic information of the high-level feature map. The formula is as follows:

[0078] F agg =Γ img-agg ({F i})

[0079] Among them, {F i} represents the selected intermediate layer features; F agg Represents the multi-scale features obtained after fusion;

[0080] Among them, in the feature aggregation module Γ img-agg In the example, the intermediate layer features F extracted by the image encoder are i The specific process of feature fusion, which utilizes the spatial information of the low-level feature map and combines the semantic information of the high-level feature map, is as follows:

[0081] Select some intermediate features output by the image encoder according to the preset information, and reduce the number of channels by 16 times through a 1×1 convolution layer;

[0082] Use a convolutional layer with a kernel of 3×3 and a stride of 2 for downsampling;

[0083] Finally, two 3×3 convolutional layers and one 1×1 convolutional layer are used as fusion convolutional layers to output three feature maps F of different sizes. agg .

[0084] S4, input the multi-scale feature map into the region proposal network Γ RPN In the above example, a target candidate box is generated on each feature layer to improve the detection capability of scrap steel monomers of different sizes. The formula is as follows:

[0085] p j =Γ RPN (F agg )

[0086] Among them, F agg Represents feature maps of different sizes, p j Represents the generated candidate region;

[0087] Specifically, the multi-scale feature map is input into the region proposal network Γ RPN In , the specific process of generating target candidate boxes on each feature layer is as follows:

[0088] L1, based on the three preset aspect ratios (0.5, 1, and 2) and pixel change scales (128, 64, and 32), 9 anchor boxes of different sizes are generated with each pixel on each feature map as the center, so that the network can adapt to objects of different sizes;

[0089] L2, the classification layer is trained based on the cross entropy loss function to determine whether each anchor box contains a scrap steel instance; at the same time, the regression layer is trained based on the SmoothL1Loss loss function to fine-tune the center point coordinates and length and width of the anchor box so that the candidate box and the real detection box of the corresponding object are closer;

[0090] L3, according to the classification results, non-maximum suppression (NMS) is used to filter the candidate frames, and the top M candidate frames that are most likely to contain scrap steel monomers at each feature level are retained as the candidate regions of interest for the final output.

[0091] S5, for the feature map F agg Perform position encoding, and then use the precise region of interest pooling layer instead of the traditional RoIAlign operation to convert the non-uniformly sized target candidate boxes p output by the RPN network into j Pooling to a fixed-size feature map, the formula is as follows:

[0092] o j =Γ PrRoI (E pos (F agg ), p j )

[0093] Among them, E pos Representative calculation feature map F agg Sinusoidal position encoding in x and y directions to increase position information; o j Represents Γ PrRoI The output of the network;

[0094] Among them, the specific process of the PrRoI layer is as follows:

[0095]

[0096] Among them, bin = {(x2-x1), (y2-y1)} represents the grid sub-region divided in RoI, where (x1, y1) and (x2, y2) are the continuous coordinates of the upper left point and the lower right point respectively; F represents the corresponding feature map. By calculating the second-order integral of the sub-region and the feature map, the cumulative error caused by the bounding box quantization operation is solved, and the accuracy of the target bounding box regression is improved.

[0097] S6, Γ PrRoI The output of the network is input into the classification branch Γ cls and locate branch Γ reg , perform target classification and bounding box regression, and encode prompt information;

[0098] Specifically, two fully connected networks are used to implement the RoI region o j The classification of specific categories and the adjustment of the target bounding box position are shown in the following formula:

[0099] c j =Γ cls (o j )

[0100] b j =Γ reg (o j )

[0101] Among them, c j Represents the category probability corresponding to each detection box; b j Represents the final output object boundary detection box. The classification and regression branches are trained based on the cross entropy loss function and the SmoothL1Loss loss function respectively.

[0102] S7, the target bounding box b j After encoding, it is used as prompt information and image features F agg Input together with the frozen SAM model mask decoder Γ m-dec Perform scrap steel instance segmentation as follows:

[0103] m j =Γ m-dec (F agg , b j )

[0104] Among them, m j Represents the generated binary mask of size H×W for each scrap steel instance. The mask prediction branch is trained based on the binary cross entropy loss function.

[0105] Step 4: Convert the regional mask output by the scrap steel instance segmentation model into a Json file in the COCO data format. Based on the image, visualize the scrap steel monomer image in the visualization module, store the material type information, and complete the scrap steel monomer instance segmentation.

[0106] Example 1

[0107] This embodiment provides a method for segmenting steel scrap instances based on a large model, and the specific process is as follows:

[0108] Step 1: Taking scrap steel recycling in steel enterprises as the task background, use a high-resolution camera to capture images in the scrap steel unloading area of ​​a large steel smelter. The shooting angle should cover the entire scrap steel pile as much as possible to obtain the original scrap steel image dataset.

[0109] Step 2: Use CVAT annotation software to perform instance segmentation and annotation on the collected scrap steel images. The annotated information is stored in a Json file and divided into training set, validation set and test set according to the proportion.

[0110] Step three, perform data enhancement and scaling on the scrap steel image and then input it into the backbone network for feature extraction. The present invention uses the SAM-ViT-B (k=12) model and freezes the parameters of its image encoder part. The specific process is as follows: reduce the original captured picture from 3024×4032×3 to 1024×1024×3, and then input it into the image encoder. In the image encoder, it is first downsampled 16 times, and then input into 12 stacked Transformer Blocks for self-attention and global attention. Finally, the convolutional layer is used to obtain the image features with an output dimension size of 2×256×64×64, as well as the intermediate features of each layer with a dimension size of 2×64×64×768, where 2 represents the batch size.

[0111] Step 4: Input the selected intermediate features of the [1st, 3rd, 5th, 7th, 9th, 11th]th layers into the feature aggregation module to obtain feature maps of three different sizes, with dimensions of 2×256×128×128, 2×256×64×64, and 2×256×32×32, respectively, to improve the recognition ability of scrap steel types of different sizes.

[0112] Step 5: Based on the preset information, the region candidate network generates anchor boxes of different numbers and sizes on each feature layer, which are [294912, 73728, 18432]. The probability of each box being a background or a scrap steel instance and the fine-tuned bounding box are obtained through the classification branch and the regression branch, and then the top 1000 bounding boxes are retained on the three feature maps according to the binary classification scores, and NMS screening is performed separately (threshold is 0.7) to obtain the final M target candidate boxes.

[0113] Step 6: This embodiment uses the precise region of interest pooling layer to convert the above-mentioned target candidate boxes of different sizes into a fixed size of M×7×7×256, and inputs it into the classification and positioning branch to obtain the specific category probability and the final target detection box. Next, the target detection box is used as the prompt information and input into the prompt encoding module to generate the sparse prompt encoding that can be used by the SAM model, with a dimension size of 2×4×256.

[0114] Step 7: Input the prompt information and image features into the SAM mask decoder for scrap steel instance segmentation to obtain a binary mask of 1024×1024 size, and then restore it to the original image size of 3024×4032 to obtain the regional mask of each steel part in the current scrap steel image, and save the result as a Json file in COCO data format.

[0115] Step 8: Read the Json file in the visualization module, visualize each steel part image in the scrap steel image according to the regional mask, and complete the single instance segmentation. The bounding box detection and segmentation accuracy of the present invention on the scrap steel small data set are shown in Table 1.

[0116] Table 1 Bounding box detection and segmentation accuracy of the present invention on a small dataset of scrap steel

[0117]

[0118] In summary, this embodiment uses the powerful feature extraction capability of the ViT model to realize feature extraction of scrap steel images, and then fuses the feature maps of each level to effectively utilize low-level spatial information and high-level semantic information. Anchor frames of different sizes are generated for multi-scale features, so that the network has good detection capabilities for steel parts of various sizes. Compared with the prior art, the coordinate quantization loss is avoided by using a precise region of interest pooling layer, and a more accurate target bounding box regression result is obtained. The target bounding box is input into the mask decoder as the prompt information for instance segmentation, which makes up for the problem that the existing segmentation method is prone to incomplete masking and fragmented segmentation in the segmentation of large-size parts, thereby effectively improving the model's zero-sample segmentation capability and generalization performance.

[0119] Example 2

[0120] according to Figure 3 As shown, the present invention also provides a scrap steel material instance segmentation system based on a large model, comprising:

[0121] The sample acquisition module 1 is used to take photos of the scrap steel pile from multiple angles to obtain high-quality sample photos;

[0122] Data processing module 2, used for instance segmentation and annotation of the scrap steel material types in the obtained sample photos to obtain annotation information, converting the annotation information into a Json file in COCO data format, and dividing the Json file into data sets;

[0123] Model training module 3 is used to input the data set into the scrap steel instance segmentation model for training and parameter optimization based on the comprehensive loss function to obtain the trained scrap steel instance segmentation model; use the trained scrap steel instance segmentation model to perform object mask segmentation and output region mask;

[0124] The image segmentation module 4 is used to convert the regional mask output by the scrap steel material instance segmentation model into a Json file in the COCO data format, and based on the image, visualize the scrap steel monomer image in the visualization module, store the material type information, and complete the scrap steel monomer instance segmentation.

[0125] Example 3

[0126] The present invention also provides a mobile terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, such as a large model-based scrap steel instance segmentation program.

[0127] When the processor executes the computer program, the steps of the above-mentioned method for segmenting steel scrap instances based on a large model are implemented, for example:

[0128] Take photos of scrap steel piles from multiple angles to obtain high-quality sample photos;

[0129] Perform instance segmentation and annotation on the scrap steel types in the obtained sample photos to obtain annotation information, convert the annotation information into a Json file in the COCO data format, and divide the Json file into data sets;

[0130] The data set is input into the scrap steel instance segmentation model for training and parameter optimization based on the comprehensive loss function to obtain the trained scrap steel instance segmentation model; the trained scrap steel instance segmentation model is used to perform object mask segmentation and output the region mask;

[0131] The regional mask output by the scrap steel instance segmentation model is converted into a Json file in the COCO data format. Based on the image, the scrap steel monomer image is visualized in the visualization module, the material type information is stored, and the scrap steel monomer instance segmentation is completed.

[0132] Alternatively, when the processor executes the computer program, the functions of each module in the above system are realized, for example:

[0133] The sample acquisition module 1 is used to take photos of the scrap steel pile from multiple angles to obtain high-quality sample photos;

[0134] Data processing module 2, used for instance segmentation and annotation of the scrap steel material types in the obtained sample photos to obtain annotation information, converting the annotation information into a Json file in COCO data format, and dividing the Json file into data sets;

[0135] Model training module 3 is used to input the data set into the scrap steel instance segmentation model for training and parameter optimization based on the comprehensive loss function to obtain the trained scrap steel instance segmentation model; use the trained scrap steel instance segmentation model to perform object mask segmentation and output region mask;

[0136] The image segmentation module 4 is used to convert the regional mask output by the scrap steel material instance segmentation model into a Json file in the COCO data format, and based on the image, visualize the scrap steel monomer image in the visualization module, store the material type information, and complete the scrap steel monomer instance segmentation.

[0137] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the mobile terminal.

[0138] For example, the computer program may be divided into a sample acquisition module 1, a data processing module 2, a model training module 3 and an image segmentation module 4, and the specific functions of each module are as follows:

[0139] The sample acquisition module 1 is used to take photos of the scrap steel pile from multiple angles to obtain high-quality sample photos;

[0140] Data processing module 2, used for instance segmentation and annotation of the scrap steel material types in the obtained sample photos to obtain annotation information, converting the annotation information into a Json file in COCO data format, and dividing the Json file into data sets;

[0141] Model training module 3 is used to input the data set into the scrap steel instance segmentation model for training and parameter optimization based on the comprehensive loss function to obtain the trained scrap steel instance segmentation model; use the trained scrap steel instance segmentation model to perform object mask segmentation and output region mask;

[0142] The image segmentation module 4 is used to convert the regional mask output by the scrap steel material instance segmentation model into a Json file in the COCO data format, and based on the image, visualize the scrap steel monomer image in the visualization module, store the material type information, and complete the scrap steel monomer instance segmentation.

[0143] The mobile terminal may be a computing device such as a desktop computer, a notebook, a palm computer, a cloud server, etc. The mobile terminal may include, but is not limited to, a processor and a memory.

[0144] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the mobile terminal, and uses various interfaces and lines to connect various parts of the entire mobile terminal.

[0145] The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the mobile terminal by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0146] The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0147] Example 4

[0148] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for instance segmentation of scrap steel based on a large model are implemented.

[0149] If the module / unit integrated in the mobile terminal is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0150] Based on such understanding, the present invention implements all or part of the processes in the above method, and can also be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned method for segmenting scrap steel instances based on a large model can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc.

[0151] The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0152] It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electrical carrier signals and telecommunication signals.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for instance segmentation of scrap steel based on a large model, characterized in that: include: Take photos of scrap steel piles from multiple angles to obtain high-quality sample photos; Perform instance segmentation and annotation on the scrap steel types in the obtained sample photos to obtain annotation information, convert the annotation information into a Json file in the COCO data format, and divide the Json file into data sets; The data set is input into the scrap steel instance segmentation model for training and parameter optimization based on the comprehensive loss function to obtain the trained scrap steel instance segmentation model; the trained scrap steel instance segmentation model is used to perform object mask segmentation and output the region mask; The regional mask output by the scrap steel instance segmentation model is converted into a Json file in the COCO data format. Based on the image, the scrap steel monomer image is visualized in the visualization module, the material type information is stored, and the scrap steel monomer instance segmentation is completed.

2. The method for instance segmentation of scrap steel based on a large model according to claim 1, characterized in that: The collected scrap steel images are annotated by instance segmentation using CVAT annotation software, and the annotation information is saved as a Json file. The Json file divides the data set into a training set, a validation set, and a test set in proportion.

3. The method for instance segmentation of scrap steel based on a large model according to claim 1, characterized in that: The specific process of inputting the data set into the scrap steel instance segmentation model for training and parameter optimization based on the comprehensive loss function is as follows: S1, scale the scrap steel image to 1024×1024 size and enhance the training samples by random horizontal flipping; S2, using the image encoder pre-trained on big data as the backbone feature extraction network Γ img-enc , process the scrap steel image I into the feature map F img And output the intermediate layer feature F of the model i , the formula is as follows: F img ,F i =Γ img-enc (I) in, represents the input image, F img Represents the final output image features, F i , i∈{1,…,k} represents multiple intermediate layer features; S3, in the feature aggregation module Γ img-agg In the example, the intermediate layer features F extracted by the image encoder are i To perform feature fusion, the spatial information of the low-level feature map is used while combining the semantic information of the high-level feature map. The formula is as follows: F agg =C img-agg ({F i }) Among them, {F i } represents the selected intermediate layer features; F agg Represents the multi-scale features obtained after fusion; S4, input the multi-scale feature map into the region proposal network Γ RPN In the above example, a target candidate box is generated on each feature layer to improve the detection capability of scrap steel monomers of different sizes. The formula is as follows: p j =C RPN (F agg ) Among them, F agh Represents feature maps of different sizes, p j Represents the generated candidate region; S5, for the feature map F agg Perform position encoding, and then use the precise region of interest pooling layer instead of the traditional RoI Align operation to convert the non-uniformly sized target candidate boxes p output by the RPN network into j Pooling to a fixed-size feature map, the formula is as follows: the j =Γ PrRoI (AND pos (F agg ),p j ) Among them, E pos Representative calculation feature map F agg Sinusoidal position encoding in x and y directions to increase position information; o j Represents Γ PrRoI The output of the network; S6, Γ PrRoI The output of the network is input into the classification branch Γ cls and locate branch Γ reg , perform target classification and bounding box regression, and encode prompt information; S7, the target bounding box b j After encoding, it is used as prompt information and image features F agg Input together with the frozen SAM model mask decoder Γ m-dec Perform scrap steel instance segmentation as follows: m j =C m-dec (F agg ,b j ) Among them, m j A binary mask of size H×W representing each scrap steel instance generated; the mask prediction branch is trained based on the binary cross entropy loss function.

4. The method for instance segmentation of scrap steel based on a large model according to claim 3, characterized in that: In S2, the model adopts the ViT-B model, and the specific process is as follows: The input image first passes through a convolution layer with a convolution kernel size of 16×16, a step size of 16, and 768 convolution kernels to obtain a feature map that is 16 times downsampled from the image. A Class Token is added to the feature map for classification prediction, and the position code is added before it is sent to 12 stacked Transformer Blocks. Eight of these blocks are self-attention modules based on Window Partition. The feature maps are divided into 14×14 blocks within these blocks, and the other four blocks ([2, 5, 8, 11]) perform global attention. The final image features and the intermediate features of each layer are output through two layers of convolution.

5. The method for instance segmentation of scrap steel based on a large model according to claim 3, characterized in that: In S3, in the feature aggregation module Γ img-agg In the example, the intermediate layer features F extracted by the image encoder are i The specific process of feature fusion, which utilizes the spatial information of the low-level feature map and combines the semantic information of the high-level feature map, is as follows: Select some intermediate features output by the image encoder according to the preset information, and reduce the number of channels by 16 times through a 1×1 convolution layer; Use a convolutional layer with a kernel of 3×3 and a stride of 2 for downsampling; Finally, two 3×3 convolutional layers and one 1×1 convolutional layer are used as fusion convolutional layers to output three feature maps F of different sizes. agg .

6. The method for instance segmentation of scrap steel based on a large model according to claim 3, characterized in that: In S4, the multi-scale feature map is input into the region proposal network Γ RPN In , the specific process of generating target candidate boxes on each feature layer is as follows: L1, based on the three preset aspect ratios and pixel change scales, 9 anchor boxes of different sizes are generated with each pixel on each feature map as the center, so that the network can adapt to targets of different sizes; L2, the classification layer is trained based on the cross entropy loss function to determine whether each anchor box contains a scrap steel instance; at the same time, the regression layer is trained based on the SmoothL1Loss loss function to fine-tune the center point coordinates and length and width of the anchor box so that the candidate box and the real detection box of the corresponding object are closer; L3, according to the classification results, non-maximum suppression (NMS) is used to filter the candidate frames, and the top M candidate frames that are most likely to contain scrap steel monomers at each feature level are retained as the candidate regions of interest for the final output.

7. The method for instance segmentation of scrap steel based on a large model according to claim 1, characterized in that: Based on the comprehensive loss function, training and parameter optimization are performed to obtain the specific formula of the trained scrap steel instance segmentation model as follows: Loss=αL rpn_cls +βL rpn_bbox +γL bbox +δL cls +εL mask Among them, {α, β, γ, δ, ε} represents the weights between different types of loss functions; L rpn_cls and L rpn_bbox Respectively represent the binary cross entropy loss and positioning loss of the RPN network; L cls Represents the final predicted category c j The cross entropy loss between the real category and the object; L bbox Represents the predicted detection box b j The SmoothL1Loss loss between the real detection frame of the object; L mask Represents the object mask m output by the mask decoder j The binary cross entropy loss between the pixels in the object's true mask and the pixels in the object's true mask.

8. A scrap steel instance segmentation system based on a large model, characterized in that: include: The sample acquisition module (1) is used to take photos of the scrap steel pile from multiple angles to obtain high-quality sample photos; A data processing module (2) is used to perform instance segmentation and annotation on the types of scrap steel in the obtained sample photos to obtain annotation information, convert the annotation information into a Json file in the COCO data format, and divide the Json file into data sets; The model training module (3) is used to input the data set into the scrap steel material instance segmentation model to perform training and parameter optimization based on the comprehensive loss function to obtain a trained scrap steel material instance segmentation model; and use the trained scrap steel material instance segmentation model to perform object mask segmentation and output a region mask; The image segmentation module (4) is used to convert the regional mask output by the scrap steel material instance segmentation model into a Json file in the COCO data format, and based on the image, perform scrap steel monomer image visualization in the visualization module, store material type information, and complete scrap steel monomer instance segmentation.

9. A mobile terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the scrap steel instance segmentation method based on a large model as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for instance segmentation of scrap steel based on a large model as described in any one of claims 1 to 7 are implemented.

Images