A semi-supervised hybrid training semantic segmentation method and system

Abstract

This invention discloses a semi-supervised hybrid training semantic segmentation method and system. It collects and labels highly overlapping particle image RGB datasets to obtain labeled and unlabeled data. A pre-segmentation model is established based on a segmentation algorithm using an encoding / decoding framework. The pre-segmentation model is trained using labeled data to obtain a coarse segmentation model. The coarse segmentation model is then used to identify the unlabeled data, yielding coarse segmentation results. Pseudo-labels are obtained based on the coarse segmentation results. Based on the pseudo-labels and labeled data, the coarse segmentation model is trained to obtain a fine segmentation model. Finally, based on the fine segmentation model, image segmentation is performed on highly overlapping particle images. This method solves the problem of low segmentation accuracy in existing stacked particle images. By using a convex hull-based pseudo-label generation algorithm, it achieves high-precision segmentation of stacked particle images and significantly improves the structural integrity of the particles, reducing the degree of undersegmentation.

Description

Technical Field

[0001] The present invention mainly relates to the technical field of blast furnace image processing, and in particular to a semi-supervised hybrid training semantic segmentation method and system. Background Art

[0002] Particles generally refer to small objects, which are small and fragmented clusters, and generally refer to solid particles, droplets or bubbles with a solid shape. Particles have three most important properties: particle size, quantity and shape. Particle size refers to the physical scale of a single particle, that is, an indicator that describes the size of a single particle, ranging from centimeters to nanometers. Quantity describes the number of particles contained in a particle group, because particles usually appear in the form of particle groups. Shape refers to the appearance of a single particle, which is generally divided into round and non-circular (irregular shape). At present, there are a lot of research on particle analysis in various industries, such as ore particle size detection in industry, cell image segmentation in medicine, leaf segmentation in agronomy, etc. Among them, the most widely used method is to perform particle analysis through image segmentation.

[0003] In image segmentation, there are concepts of under-segmentation and over-segmentation, which are used to describe the difference between the segmentation results and the labels. In the process of particle image segmentation, under-segmentation affects the accuracy of segmentation more than over-segmentation. In most cases of over-segmentation, the extra pixels often cannot form a new closed connected domain, but only form branches of varying lengths. This has almost negligible impact on identifying the number of particles, calculating the size of particles, and locating the position of particles, and only has a partial impact on identifying the edges and shapes of particles.

[0004] In most cases of under-segmentation, the lack of some pixels has a fatal impact on an originally closed connected domain. The loss of some key pixels will directly lead to the fusion of multiple originally closed connected domains into one, which will have a huge impact on the calculation of the number, size, position locking, edges and shapes of particles. This impact cannot be ignored in the analysis of particle images. In addition to the influence of the algorithm itself, the main reason for a large number of under-segmentations in the particle image segmentation process is the complexity of the particle object itself, including the shape, texture and spatial position distribution of the particles. The complex texture and irregular shape greatly increase the difficulty and cost of particle image annotation, and it is difficult to obtain a large amount of labeled data. In terms of spatial distribution, the segmentation of irregular particle images can be roughly divided into three types: particle adhesion (low particle overlap rate), particle overlap but overlapping information can be used, and particle overlap and overlapping information is difficult to use. The segmentation problem of the third type of particle image is one of the most difficult challenges. That is, in the image, multiple particles overlap with each other to a high degree, blocking each other's partial edge information, and the texture information of the overlapping area cannot be obtained. Therefore, it is urgent to propose a segmentation method for the third type of irregular particle image under limited labeled data.

[0005] The invention with application publication number CN113516130A discloses a semi-supervised image semantic segmentation method based on entropy minimization. The method proposes a feature gradient mapping regularization strategy (FGMR), which uses the gradient mapping of low-level feature maps in the encoder to enhance the encoder's encoding ability for deep feature maps; then, an adaptive sharpening strategy is proposed to keep the decision boundary of unlabeled data in a low-density area; and in order to further reduce the impact of noise, a low-confidence consistency strategy is proposed to ensure the consistency of classification and segmentation.

[0006] The patent application publication number CN112381098A discloses a semi-supervised learning method and system based on self-learning in the target segmentation field. The invention uses labeled data in the training data set to train the initial segmentation network; generates pseudo labels for the unlabeled data in the training data set through the trained initial segmentation network; performs shape quality evaluation and semantic quality evaluation on the generated pseudo labels; fuses the shape quality and semantic quality to obtain the pseudo label quality; estimates the distribution of real labels and pseudo labels, and optimizes the distribution of pseudo labels; adds data with higher pseudo label quality to the training data set to expand the training data set; optimizes the trained initial segmentation network using the expanded training data set; and iteratively repeats the above steps until the performance of the segmentation network is saturated. A corresponding system, terminal and medium are also provided. The invention patent does not specifically provide a method for generating pseudo labels. Summary of the invention

[0007] The semi-supervised hybrid training semantic segmentation method and system provided by the present invention solve the existing technical problem of low segmentation accuracy of stacked particle images.

[0008] In order to solve the above technical problems, the semi-supervised hybrid training semantic segmentation method proposed in the present invention includes:

[0009] Collect highly overlapping particle image RGB datasets, and annotate the particle image RGB datasets to obtain labeled data and unlabeled data;

[0010] A pre-segmentation model is established based on the segmentation algorithm of the codec framework, and the pre-segmentation model is trained with labeled data to obtain a coarse segmentation model;

[0011] A coarse segmentation model is used to identify unlabeled data, obtain a coarse segmentation result, and obtain a pseudo label based on the coarse segmentation result;

[0012] Based on pseudo-labels and labeled data, the coarse segmentation model is trained to obtain a fine segmentation model;

[0013] Based on the fine segmentation model, image segmentation is performed on highly overlapping particle images.

[0014] Furthermore, the particle image RGB dataset is annotated to obtain labeled data including:

[0015] Mark out the particle edges of the particle image in the particle image RGB dataset;

[0016] The particle edges, particle gaps and other non-particle areas in the particle image are merged as the background, and the internal area of ​​the particle is taken as the foreground target to obtain labeled data;

[0017] The labeled data is divided into first labeled data, second labeled data, and third labeled data.

[0018] Furthermore, the pre-segmentation model is trained using labeled data to obtain a rough segmentation model:

[0019] The first labeled data is used to train the pre-segmentation model to obtain a coarse segmentation model.

[0020] Furthermore, obtaining pseudo labels based on the coarse segmentation results includes:

[0021] Extract overlapping particle connected domains from the coarse segmentation results;

[0022] Extract the convex hull and convex defects of the particle connected domain, and extract feature points based on the convex hull and convex defects;

[0023] Match the feature points and obtain the segmentation line segments according to the matching results;

[0024] The segmentation line segments are superimposed on the rough segmentation results to obtain pseudo labels.

[0025] Further, extracting the convex hull and convex defects of the particle connected domain, and extracting feature points based on the convex hull and convex defects includes:

[0026] The farthest point of the corresponding contour point in the convex defect is selected as the feature point, and the calculation formula of the feature point is:

[0027]

[0028] Among them, V n represents the set of feature points in the nth convex hull, m represents the number of convex defects in the convex hull, i represents the i-th convex defect, and p n,i represents the contour point on the i-th convex defect in the n-th convex hull, C n represents the nth convex hull, d(p n,i ,C n ) represents the contour point p n,i To the corresponding convex hull C n The Euclidean distance of the edges.

[0029] Furthermore, matching the feature points and obtaining the segmentation line segments according to the matching results include:

[0030] The feature points are matched, and the calculation formula for matching the feature points is:

[0031]

[0032] Among them, X i represents the matching coefficient of the i-th feature point. The matching coefficient specifically refers to the maximum matching probability between the i-th feature point and other feature points. i represents the i-th feature point, v j represents the jth feature point, i≠j, V n represents the set of feature points in the nth convex hull, d(v i ,v j ) indicates v i With v j The Euclidean distance, A(v i ,v j ) represents the angle formed by two line segments, one of which is formed by the feature point v i and and The other is formed by the midpoint of the feature point v j and and The midpoint of represents the starting point of the contour of the i-th convex defect of the n-th convex hull, represents the endpoint of the contour of the i-th convex defect of the n-th convex hull, Indicates the starting point of the contour To the corresponding contour end point The Euclidean distance, λ1, λ2 represent the experimental parameters for adjusting the size of the overall geometric index, l() represents the segmentation line formed by two points, and ∠ represents the angle formed by the two line segments;

[0033] The matching coefficient of each feature point is calculated and compared with the preset matching threshold. If the matching coefficient is greater than the matching threshold, the two feature points are connected by generating a two-point line segment. If it is less than the matching threshold, the point is skipped. The matching of all feature points is completed in this cycle to obtain the segmentation line segment.

[0034] Furthermore, based on the pseudo-labels and labeled data, the coarse segmentation model is trained to obtain the fine segmentation model:

[0035] Based on the pseudo-label and the second labeled data, the coarse segmentation model is trained to obtain a sub-fine segmentation model;

[0036] Based on the third labeled data, the sub-fine segmentation model is trained to obtain a fine segmentation model.

[0037] Furthermore, the loss function for training the coarse segmentation model and the sub-fine segmentation model is calculated as follows:

[0038] Loss(Y)=L CE (L)+λ·L CE (P),

[0039] Where Y represents all samples, Loss(Y) represents the overall loss function, L represents the labeled dataset, and L CE (L) represents the cross entropy loss function trained with a labeled dataset, P represents an unlabeled dataset, λ represents the training parameter, and L CE (P) represents the cross entropy loss function trained with an unlabeled dataset.

[0040] Furthermore, after obtaining the refined segmentation model, the following steps are also included:

[0041] The accuracy of the fine segmentation model is evaluated by mIOU, and the under-segmentation degree evaluation index is used to evaluate the under-segmentation degree of the segmentation result.

[0042] The semi-supervised hybrid training semantic segmentation system provided by the present invention comprises:

[0043] A memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the semi-supervised hybrid training semantic segmentation method provided by the present invention are implemented.

[0044] Compared with the prior art, the advantages of the present invention are:

[0045] The semi-supervised hybrid training semantic segmentation method and system provided by the present invention collect highly overlapping RGB data sets of particle images, and annotate the RGB data sets of particle images to obtain labeled data and unlabeled data, establish a pre-segmentation model based on the segmentation algorithm of the codec framework, and use the labeled data to train the pre-segmentation model to obtain a coarse segmentation model, use the coarse segmentation model to identify the unlabeled data to obtain a coarse segmentation result, and obtain a pseudo-label based on the coarse segmentation result, train the coarse segmentation model based on the pseudo-label and the labeled data to obtain a fine segmentation model, and perform image segmentation on the highly overlapping particle images based on the fine segmentation model, thereby solving the existing technical problem of low segmentation accuracy of stacked particle images, making full use of the implicit information contained in the unlabeled data through the pre-segmentation model, and proposing a pseudo-label generation algorithm based on the convex hull when training the unlabeled data, further segmenting the coarse segmentation result obtained on the pre-segmentation model to obtain a pseudo-label that is closer to the real label in terms of structural integrity, finally achieving high-precision segmentation of the stacked particle images, and being able to greatly improve the structural integrity of the particles and reduce the degree of under-segmentation.

[0046] The key points of the present invention include:

[0047] (1) We innovatively propose a semi-supervised hybrid training framework for semantic segmentation of complex particle images, namely, a semi-supervised training mode consisting of a pre-segmentation model, hybrid training, and fine-tuning.

[0048] (2) We innovatively propose a trainable and learnable pseudo-label generation algorithm for particle image segmentation. By exploring the convex hull and related geometric features on the segmented image, we extract and match the corresponding feature points, and then generate re-segmented line segments to obtain pseudo-labels that are closer to the true labels in terms of structural integrity.

[0049] (3) We innovatively propose a semi-supervised hybrid training semantic segmentation framework and system based on convex hull to generate pseudo-labels, which enables efficient training of unlabeled data and achieves segmentation accuracy close to that of full supervision. It can also significantly improve the structural integrity of particles and reduce the degree of under-segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is an overall structural diagram of a semi-supervised hybrid training semantic segmentation framework based on convex hull generation of pseudo labels according to the second embodiment of the present invention;

[0051] Figure 2 The convex hull and related geometric features of the rough segmentation of the second embodiment of the present invention;

[0052] Figure 3 The feature points and matching generated segmentation line segments in the rough segmentation of the second embodiment of the present invention;

[0053] Figure 4 The figure shows the comparison results of segmenting stacked particle images using different methods.

[0054] Figure 5 4 is a structural block diagram of a semi-supervised hybrid training semantic segmentation system according to an embodiment of the present invention.

[0055] Reference numerals:

[0056] 100. Memory; 200. Processor. DETAILED DESCRIPTION

[0057] In order to facilitate the understanding of the present invention, the present invention will be described more comprehensively and meticulously below in conjunction with the accompanying drawings and preferred embodiments of the present invention, but the protection scope of the present invention is not limited to the following specific embodiments.

[0058] The embodiments of the present invention are described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.

[0059] Embodiment 1

[0060] The semi-supervised hybrid training semantic segmentation method provided in the first embodiment of the present invention includes:

[0061] Step S101, collecting highly overlapping particle image RGB data sets, and annotating the particle image RGB data sets to obtain labeled data and unlabeled data;

[0062] Step S102, establishing a pre-segmentation model based on the segmentation algorithm of the codec framework, and training the pre-segmentation model with labeled data to obtain a coarse segmentation model;

[0063] Step S103, using a coarse segmentation model to identify unlabeled data, obtaining a coarse segmentation result, and obtaining a pseudo label based on the coarse segmentation result;

[0064] Step S104, training the coarse segmentation model based on the pseudo labels and the labeled data to obtain a fine segmentation model;

[0065] Step S105 , performing image segmentation on the highly overlapping particle images based on the fine segmentation model.

[0066] The semi-supervised hybrid training semantic segmentation method provided by the embodiment of the present invention collects a highly overlapping RGB data set of particle images, and annotates the RGB data set of particle images to obtain labeled data and unlabeled data, establishes a pre-segmentation model based on the segmentation algorithm of the codec framework, and uses the labeled data to train the pre-segmentation model to obtain a coarse segmentation model, uses the coarse segmentation model to identify the unlabeled data to obtain a coarse segmentation result, and obtains a pseudo-label based on the coarse segmentation result, trains the coarse segmentation model based on the pseudo-label and the labeled data to obtain a fine segmentation model, and performs image segmentation on the highly overlapping particle images based on the fine segmentation model, thereby solving the existing technical problem of low accuracy in the segmentation of stacked particle images, and fully utilizes the implicit information contained in the unlabeled data through the pre-segmentation model, and when training the unlabeled data, proposes a pseudo-label generation algorithm based on the convex hull, further segments the coarse segmentation result obtained on the pre-segmentation model, and obtains a pseudo-label that is closer to the real label in terms of structural integrity, thereby finally achieving high-precision segmentation of the stacked particle images, and can greatly improve the structural integrity of the particles and reduce the degree of under-segmentation.

[0067] Embodiment 2

[0068] Embodiment 2 of the present invention proposes a semi-supervised hybrid training semantic segmentation framework based on convex hull to generate pseudo labels. The algorithm steps of Embodiment 2 of the present invention include:

[0069] (1) Prepare a highly overlapping RGB dataset of particle images. Prepare n RGB data for training, including 0.2n labeled samples L and 0.8n unlabeled data, denoted as U. The 0.2n labeled data are further divided into three parts: 0.1n, 0.05n and 0.05n, denoted as L1, L2 and L3, which are used for pre-segmentation model, hybrid training and fine-tuning respectively.

[0070] (2) Construct a pre-segmentation model. Based on the algorithm based on the codec framework, pre-training is performed with labeled data L1 to obtain a pre-segmentation model f for rough segmentation. pre .

[0071] (3) Generate pseudo labels. During the training process, the rough segmentation result S of the unlabeled sample U is c Converted into pseudo label P. The generation algorithm is divided into four steps. The first step is to c First, extract the overlapping particle connected domain O n , then extract its convex hull C and convex defect C def , extract all feature points V on the convex defect nThen, based on the skeleton graph branch geometric features, the matching index X is proposed to match these feature points. Finally, the successfully matched feature points Connect them in the form of line segments, these line segments L s And the rough segmentation result S c Superposition forms fine segmentation S f , and added as pseudo label P to subsequent training.

[0072] (4) Randomly mix the generated pseudo-label P with L2 as a mixed data set to further train the coarse segmentation model f pre Get model f M . And add the loss of pseudo-label data to the overall loss function. After the mixed data set training is completed, use L3 to fine-tune the model and obtain the final segmented model f e .

[0073] (5) Finally, mIOU is used to evaluate the accuracy of the model, and the under-segmentation degree is used to evaluate the under-segmentation degree of the final segmentation result.

[0074] The specific implementation plan is as follows:

[0075] 1. Prepare a highly overlapping RGB dataset of irregular particle images.

[0076] 1) Select particle objects according to particle classification

[0077] There are different types of particles. The particle objects selected in the present invention are irregular in shape, highly stacked, and the information of overlapping areas is difficult to use.

[0078] 2) Create a dataset

[0079] The image of the particle pile is obtained through the camera, and some samples of the particle image are labeled using the annotation tool as the labeled samples in the data set, and the rest are used as unlabeled samples.

[0080] 3) Divide the dataset

[0081] If there are n RGB data, 0.2n of them are used as labeled samples L, and 0.8n samples are used as unlabeled data, recorded as U. At the same time, the 0.2n labeled data are further divided into three parts: 0.1n, 0.05n and 0.05n, recorded as L1, L2 and L3 respectively.

[0082] 2. Construct a pre-segmentation model for the segmentation algorithm based on the codec framework pre .

[0083] Based on the codec framework, the present invention builds a four-layer image segmentation pre-segmentation model f preThe overall network architecture is as follows: Figure 1 As shown. The network consists of two parts: encoder and decoder.

[0084] 1) Encoder

[0085] In the encoder, each layer consists of a feature extraction module consisting of pooling, two groups of 3×3 convolutions, BN (Batch Normalization), and ReLU activation functions. Max pooling is used between the two layers for dimensionality reduction. The input is a 3*s*s RGB image, and the encoder outputs an RGB feature tensor U of s*(s / 16)*(s / 16). rgb =(u0,u1,...,u s-1 ).

[0086] 2) Decoder

[0087] The decoding process adopts a strategy similar to the compression branch on the left. Each layer has upsampling, two sets of convolution, BN and ReLU activation functions, with a total of four layers. rgb =(u 0,0 ,u 1,0 ,...,u s,0 ) is input into the decoder, and after four times of pooling dimensionality reduction (maximum pooling) and 3*3 convolution operations, the final output is a binary coarse segmentation result S of size 1*s*s c .

[0088] 3) Classification target

[0089] The classification goal of general image segmentation is to classify the edge of the particle as the foreground and the other areas as the background. This segmentation currently cannot distinguish whether the pixel belongs to the internal area of ​​the particle or the gap between particles. The present invention classifies the particle edge e and the particle gap g and other non-particle areas a np Merge it and use it as background B, then use the inner area of ​​the particle a in As the foreground target T, the edge of the particle needs to be marked separately during the marking process. This embodiment adopts this classification method to avoid the existing classification method of taking the edge of the particle as the foreground and other areas as the background, which makes it impossible to distinguish whether the pixel belongs to the internal area of ​​the particle or the gap between particles.

[0090] 3. Pseudo-label generation algorithm based on convex hull

[0091] The unlabeled data U is converted into pseudo labels P during the training process by the re-segmentation method. The purpose is to further segment the image using the geometric features of the coarse segmentation image under limited labels, so that it forms pseudo labels that are closer to the real labels in terms of structural integrity, which helps the further training of the model to achieve an effect close to full supervision and reduce the occurrence of under-segmentation. The generation algorithm is divided into four steps.

[0092] 1) Identify the overlapping particle connected domain O n

[0093] In order to improve the calculation speed of the algorithm, it is not necessary to calculate every particle connected domain, but only to study the particle connected domains that have overlapped. n The K-means clustering algorithm is used to perform cluster analysis on all particle connected domains.

[0094] Step 1: Select and initialize 2 samples as the initial cluster centers a=a1, a2;

[0095] Step 2: For each sample x in the data set i Calculate its distance to the two cluster centers and assign it to the class corresponding to the cluster center with the smallest distance;

[0096] Step 3: For each category a j , recalculate its cluster center (i.e., the centroid of all samples belonging to this class);

[0097] Step 4: Repeat steps 2 and 3 above until a certain termination condition (number of iterations, minimum error change, etc.) is reached.

[0098] By clustering, the overlapping particle connected domains O are extracted. n .

[0099] 2) Extract feature point set V on the coarse segmentation image n

[0100] First, in the rough segmentation result S c Each overlapping particle connected domain O is calculated n The convex hull C n and all its convex defects D n,m , and then the corresponding contour point p in each convex defect n,m The farthest point of is taken as a feature point in the connected domain of overlapping particles, such as Figure 2 As shown, Figure 2 is the convex hull of the rough segmentation and related geometric features.

[0101]

[0102] Among them, d(p n,i ,C n ) represents the contour point p n,i To the corresponding convex hull C n The Euclidean distance of the edges.

[0103] 3) Feature point matching

[0104] Aiming at the geometric features of convex hull and convex defects in the coarse segmentation image, the present invention proposes a geometric index to match all the extracted feature points in pairs, which is defined as follows:

[0105]

[0106] Among them, X i represents the matching coefficient of the i-th feature point. The matching coefficient specifically refers to the maximum matching probability between the i-th feature point and other feature points. i represents the i-th feature point, v j represents the jth feature point, i≠j, V n represents the set of feature points in the nth convex hull, d(v i ,v j ) indicates v i With v j The Euclidean distance, A(v i ,v j ) represents the angle formed by two line segments, one of which is formed by the feature point v i and and The other is formed by the midpoint of the feature point v j and and The midpoint of represents the starting point of the contour of the i-th convex defect of the n-th convex hull, represents the endpoint of the contour of the i-th convex defect of the n-th convex hull, Indicates the starting point of the contour To the corresponding contour end point The Euclidean distance, λ1, λ2 represent the experimental parameters for adjusting the size of the overall geometric index, l() represents the segmentation line formed by two points, and ∠ represents the angle formed by the two line segments;

[0107] In a real vector space V, for a given set X, the intersection S of all convex sets that contain X is called the convex hull of X. Any deviation of a contour from its convex hull is called a convexity defect.

[0108] The reasons for selecting the convex hull method in this embodiment are mainly based on the following points:

[0109] 1. In order to solve the problem of overlapping particle segmentation, it is necessary to improve the segmentation accuracy and reduce the degree of under-segmentation. The simplest method is to generate re-segmentation line segments;

[0110] 2. The generation of the segmented line needs to determine the starting point and the end point, which can only rely on the geometric features on the graph. The convex hull and convex defects provide such geometric features.

[0111] 3. Other pseudo-labeling methods are more general methods driven by data and are not applicable to the technical problems raised by the embodiments of the present invention. The method of generating pseudo-labels based on convex hull adopted in this embodiment can find the required features more quickly, simply and accurately, thereby helping to improve the model training accuracy and further improve the segmentation accuracy of the stacked particle image.

[0112] 4) Generate segment L s

[0113] Calculate the X of each feature point i After that, it is compared with the matching threshold α. If it is greater than or equal to this value, the two feature points are connected by generating a two-point line segment. If it is less than this value, the point is skipped. Finally, all segmentation line segments L are completed through a loop. s The generation of Figure 3 As shown, Figure 3 It is the feature point in the rough segmentation and the segmentation line segment generated by matching. s And the rough segmentation result S c The final pseudo label P is formed by superposition.

[0114] P = f pre (U)+L s =S c +L s ,

[0115] Through this pseudo-label generation algorithm, a pseudo-label with higher accuracy and closer to the real label in structural integrity is obtained on the basis of coarse segmentation.

[0116] 4. Semi-supervised hybrid training framework

[0117] First, if there are n RGB data, 0.2n of them are used as labeled samples L, and 0.8n samples are used as unlabeled data, recorded as U. At the same time, the 0.2n labeled data are further divided into three parts: 0.1n, 0.05n and 0.05n, recorded as L1, L2 and L3 respectively.

[0118] f pre =f(L1),

[0119] Secondly, L2 and unlabeled data U are randomly and uniformly mixed to form a mixed dataset M, and then pre Continue training based on . If unlabeled data is trained, the pseudo-label generation algorithm is called to generate pseudo-labels and use them like real labels. After training the entire mixed data set, the model f is obtained. M .

[0120] f M =f pre(M),

[0121] Finally, use the remaining labeled data L3 in the model f M Fine-tune the final high-precision segmentation model f e .

[0122] f e =f M (L3),

[0123] During the entire hybrid training process, the loss of pseudo-labeled data is added to the overall loss function. The overall loss function is defined as:

[0124] Loss(Y)=L CE (L)+λ·L CE (P),

[0125] Where Y represents all samples, Loss(Y) represents the overall loss function, L represents the labeled dataset, and L CE (L) represents the cross entropy loss function trained with a labeled dataset, P represents an unlabeled dataset, λ represents the training parameter, and L CE (P) represents the cross entropy loss function trained with an unlabeled dataset.

[0126] The material in the blast furnace charging process (not limited to sintered ore) is a stacked particle image with highly overlapping occlusion and overlapping information that is difficult to utilize. Since such particle images with complex textures are difficult to label and the labeling cost is too high, it is difficult to obtain a large amount of labeled data. Therefore, the present invention proposes a semi-supervised hybrid training semantic segmentation framework and system based on convex hull generation of pseudo-labels, which are used to improve the accuracy of segmenting such particle images and reduce under-segmentation and unseparated connected domains that occur during the segmentation process.

[0127] The present invention proposes a semi-supervised hybrid training semantic segmentation framework and system based on convex hull to generate pseudo labels. Compared with existing methods, the present invention innovatively proposes a semi-supervised hybrid training framework for complex particle image segmentation, which fully utilizes the implicit information contained in the unlabeled data through the pre-segmentation model, hybrid training and fine-tuning training process. And when training unlabeled data, a pseudo-label generation algorithm based on convex hull is proposed to further segment the rough segmentation results obtained on the pre-segmentation model to obtain pseudo-labels that are closer to the real labels in terms of structural integrity. Ultimately, a segmentation accuracy close to that of full supervision is achieved, and the structural integrity of the particles can be greatly improved, reducing the degree of under-segmentation.

[0128] Example 3

[0129] This example uses a 2650m 3Large blast furnace material images are used for verification. The semi-supervised hybrid training semantic segmentation framework and system based on convex hull generation of pseudo labels proposed by the inventors in the embodiment are as follows:

[0130] 1. Data set preparation. In order to obtain RGB images of blast furnace materials, the present invention collects RGB image information on the blast furnace feeding belt of a steel plant through hardware devices such as industrial cameras. The industrial camera selected is the German Basler acA2500-14gm Basler industrial array camera data acquisition software is pylon Viewer. Communication is carried out in Ethernet mode. A total of 8 groups of data are collected by the industrial camera in a concentrated time period (each time a group of data is collected, a batch of ore is replaced), and each group of data includes 1000-2000 pictures, where the picture size is 2592*2048.

[0131] 2. Semi-supervised hybrid training based on convex hull to generate pseudo labels. First, if there are 8000 RGB data, 1600 of them are used as labeled samples L, and 6400 samples are used as unlabeled data, denoted as U. At the same time, the 1600 labeled data are further divided into three parts: 800, 400 and 400, denoted as L1, L2 and L3 respectively. Then the encoding branch in the pre-segmentation model is designed. Each layer consists of a feature extraction module consisting of 2*2 pooling, two groups of 3×3 convolutions, BN (BatchNormalization), and ReLU activation functions. The maximum pooling is used between the two layers for dimensionality reduction. The decoding process on the right adopts a strategy similar to the compression branch on the left. Each layer has upsampling, two groups of convolutions, BN and ReLU activation functions. 400 labeled data are used for training. During the training process, Adam is used as the optimization algorithm. The learning rate is set to 10^-4, and the cosine annealing learning rate scheduling strategy is used to adjust the learning rate, with the minimum value of 10^-6. The weight decay is set to 10^-8. The pre-segmentation model capable of binary classification is obtained through training, aiming to divide the pixels in the blast furnace material image into background and material particles. Next, the 400 labeled data and 6400 unlabeled data are randomly and evenly mixed to form a mixed data set M of size 6800, and then pre Continue training based on . If unlabeled data is trained, the pseudo-label generation algorithm is called to generate pseudo-labels and use them like real labels. After training the entire mixed data set, the model f is obtained. M Finally, the remaining 400 labeled data L3 are used in model f M Fine-tune the final high-precision segmentation model f e .

[0132] 3. Experimental results and analysis. On the same experimental platform, the labeled dataset is divided into three parts according to different proportions, and comparative experiments are carried out using U-Net, MaskRCNN, ALSS and the method in this paper. Figure 4 This is the comparison result of different methods. Figure 4 (a) represents the original image, (b) the segmentation result image using the UNet method, (c) the segmentation result image using the maskrcnn method, (d) the segmentation result image of the method of the embodiment of the present invention, (e) the skeleton image of the method of the embodiment of the present invention, and (f) the real label image. Table 1 shows the performance of commonly used evaluation indicators for image segmentation under different methods at different proportions of labeled data. It can be seen from the segmentation indicators in Table 1 that the segmentation results of the semi-supervised hybrid training semantic segmentation framework and system based on convex hull generation of pseudo-labels proposed in the present invention are better than other deep learning models. At the same time, the method of the present invention carried out comparative experiments at different ratios of 4:0:0, 2:2:0, 2:0:2, and 2:1:1, and finally concluded that 2:1:1 is the current optimal division ratio.

[0133] Table 1

[0134]

[0135] Reference Figure 5 , the semi-supervised hybrid training semantic segmentation system proposed in the embodiment of the present invention includes:

[0136] A memory 10, a processor 20, and a computer program stored in the memory 10 and executable on the processor 20, wherein the processor 20 implements the steps of the semi-supervised hybrid training semantic segmentation method proposed in this embodiment when executing the computer program.

[0137] The specific working process and working principle of the semi-supervised hybrid training semantic segmentation system of this embodiment can refer to the working process and working principle of the semi-supervised hybrid training semantic segmentation method in this embodiment.

[0138] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A semi-supervised hybrid training semantic segmentation method, characterized in that: The method comprises: Collect highly overlapping particle image RGB datasets, and annotate the particle image RGB datasets to obtain labeled data and unlabeled data; A pre-segmentation model is established based on the segmentation algorithm of the codec framework, and the pre-segmentation model is trained with labeled data to obtain a coarse segmentation model; The coarse segmentation model is used to identify the unlabeled data to obtain a coarse segmentation result, and a pseudo label is obtained based on the coarse segmentation result, wherein obtaining the pseudo label based on the coarse segmentation result includes: Extract overlapping particle connected domains from the coarse segmentation results; Extract the convex hull and convex defects of the particle connected domain, and extract feature points based on the convex hull and convex defects; Matching the feature points and obtaining the segmentation line segments according to the matching results, wherein matching the feature points and obtaining the segmentation line segments according to the matching results include: The feature points are matched, and the calculation formula for matching the feature points is: Among them, X i represents the matching coefficient of the i-th feature point. The matching coefficient specifically refers to the maximum matching probability between the i-th feature point and other feature points. i represents the i-th feature point, v j represents the jth feature point, i≠j, V n represents the set of feature points in the nth convex hull, d(v i ,v j ) indicates v i With v j The Euclidean distance of A(v i ,v j ) represents the angle formed by two line segments, one of which is formed by the feature point v i and and The other is formed by the midpoint of the feature point v j and and The midpoint of represents the starting point of the contour of the i-th convex defect of the n-th convex hull, represents the endpoint of the contour of the i-th convex defect of the n-th convex hull, Indicates the starting point of the contour To the corresponding contour end point The Euclidean distance, λ1, λ2 represent the experimental parameters for adjusting the size of the overall geometric index, l() represents the segmentation line formed by two points, and ∠ represents the angle formed by the two line segments; Calculate the matching coefficient of each feature point and compare it with the preset matching threshold. If the matching coefficient is greater than the matching threshold, connect the two feature points by generating a two-point line segment. If it is less than the matching threshold, skip the point. Repeat this cycle to complete the matching of all feature points, thereby obtaining the segmentation line segment. Superimpose the segmentation line segment with the rough segmentation result to obtain a pseudo label; Based on pseudo-labels and labeled data, the coarse segmentation model is trained to obtain a fine segmentation model; Based on the fine segmentation model, image segmentation is performed on highly overlapping particle images.

2. The semi-supervised hybrid training semantic segmentation method according to claim 1, characterized in that: The particle image RGB dataset is annotated to obtain labeled data including: Mark out the particle edges of the particle images in the particle image RGB dataset; The particle edges, particle gaps and other non-particle areas in the particle image are merged as the background, and the internal area of ​​the particle is taken as the foreground target to obtain labeled data; The labeled data is divided into first labeled data, second labeled data, and third labeled data.

3. The semi-supervised hybrid training semantic segmentation method according to claim 2, characterized in that: The pre-segmentation model is trained using labeled data to obtain a rough segmentation model: The first labeled data is used to train the pre-segmentation model to obtain a coarse segmentation model.

4. The semi-supervised hybrid training semantic segmentation method according to claim 3, characterized in that: Extracting the convex hull and convex defects of the particle connected domain, and extracting feature points based on the convex hull and convex defects include: The farthest point of the corresponding contour point in the convex defect is selected as the feature point, and the calculation formula of the feature point is: Among them, V n represents the set of feature points in the nth convex hull, m represents the number of convex defects in the convex hull, i represents the i-th convex defect, and p n,i represents the contour point on the i-th convex defect in the n-th convex hull, C n represents the nth convex hull, d(p n,i ,C n ) represents the contour point p n,i To the corresponding convex hull C n The Euclidean distance of the edges.

5. The semi-supervised hybrid training semantic segmentation method according to claim 4, characterized in that: Based on pseudo-labels and labeled data, the coarse segmentation model is trained to obtain the fine segmentation model as follows: Based on the pseudo-label and the second labeled data, the coarse segmentation model is trained to obtain a sub-fine segmentation model; Based on the third labeled data, the sub-fine segmentation model is trained to obtain a fine segmentation model.

6. The semi-supervised hybrid training semantic segmentation method according to claim 5, characterized in that: The loss function for training the coarse segmentation model and the sub-fine segmentation model is calculated as follows: Loss(Y)=L CE (L)+λ·L CE (P), Where Y represents all samples, Loss(Y) represents the overall loss function, L represents the labeled dataset, and L CE (L) represents the cross entropy loss function trained with a labeled dataset, P represents an unlabeled dataset, λ represents the training parameter, and L CE (P) represents the cross entropy loss function trained with an unlabeled dataset.

7. The semi-supervised hybrid training semantic segmentation method according to claim 6, characterized in that: After obtaining the refined segmentation model, it also includes: The accuracy of the fine segmentation model is evaluated by mIOU, and the under-segmentation degree evaluation index is used to evaluate the under-segmentation degree of the segmentation result.

8. A semi-supervised hybrid training semantic segmentation system, the system comprising: A memory (100), a processor (200), and a computer program stored in the memory (100) and executable on the processor (200), wherein the processor (200) implements the steps of any of the methods described in claims 1 to 7 when executing the computer program.

Images