A sample equalization method based on large shape model

Abstract

The present invention relates to a sample balancing method based on a large shape model, comprising the following steps: S1, extracting appearance features; splicing the obtained features, and the feature vector obtained by splicing is the appearance feature; S2, intra-class clustering; clustering the samples in each class based on the appearance features using the k-means algorithm; S3, inter-class subclass clustering; each class obtains subclasses after k-means clustering, and clusters the centers of the subclasses again; S4, calculating the number of additions and deletions of each subclass; S5, sample simplification and sample enhancement. The present invention trains a large model based on shape, combines the information of mutual conversion between text and image, generates features with strong separability, combines the machine learning algorithm to screen samples, so that the samples are evenly distributed in categories and appearances; by balancing the samples of each category and appearance, higher quality and more representative samples are obtained, the quality of training data is improved, and thus the upper limit of the model effect is improved.

Description

Technical Field

[0001] The invention relates to the technical field of machine vision detection, and in particular to a sample equalization method based on a large shape model. Background Art

[0002] Most of the existing sample balancing methods adopt the following two forms:

[0003] (1) Category balance: Classify defects and screen them according to the number of defects in each category to achieve sample balance between categories;

[0004] (2) Manual appearance balance: People observe the appearance distribution of samples in the same category and perform secondary classification, dividing the samples in each category into several subcategories. By screening based on the number of samples in each subcategory in a category, appearance balance within the category is achieved.

[0005] However, the sample balance between categories does not take into account the appearance within the category, and does not balance the different appearances in the same category, which may lead to uneven distribution of appearances in the same category, resulting in poor defect detection effect of the trained model for some appearances that account for a small proportion;

[0006] Manual appearance balancing requires human eyes to identify and judge, resulting in unstable results. Due to subjective factors of operators and possible visual fatigue, the balancing effect is poor and the efficiency is low. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a sample balancing method based on a large shape model, which solves the problem that in projects using deep learning, samples of different categories and appearances are unevenly distributed in quantity, resulting in unsatisfactory training model results.

[0008] The technical solution adopted by the present invention to solve the technical problem is: a sample equalization method based on a large shape model, comprising the following steps:

[0009] S1. Extract the appearance features; and concatenate the features, and the concatenated feature vector is the appearance feature;

[0010] S2, intra-class clustering: cluster the samples in each class based on their appearance features using the k-means algorithm;

[0011] S3, inter-class sub-class clustering; each class is clustered by k-means to obtain sub-classes, and the centers of the sub-classes are clustered again;

[0012] S4. Calculate the number of additions and deletions of each subcategory;

[0013] S5. For subcategories that need to streamline samples, delete the unselected samples according to the samples screened in step S4; for subcategories that need to increase samples, perform sample enhancement to reach the allocated number.

[0014] Furthermore, in step S1 described in the present invention, the appearance features include: shape features, image features and text features; the shape features are features extracted and output by selecting one layer of the large shape model.

[0015] Furthermore, in step S2 of the present invention, the clustering category parameter m is determined by:

[0016] (1) Construct the loss function: the horizontal axis is m, the sampling step is 1, and the vertical axis is Where k is the number of samples in the i-th class in the k-means clustering result, x j is the appearance feature vector of each sample in the i-th class, uc i is the center of all sample appearance feature vectors of the i-th class, that is, the average of the Euclidean distances from the samples in each class clustered by k-means to the center of that class;

[0017] (2) Determine the number of categories: Calculate the second-order derivative of the constructed loss function and select the horizontal coordinate of the point with the largest second-order derivative, which is the number of categories m;

[0018] (3) Intra-class classification: Based on the appearance feature vectors of samples in each class and the determined parameters, the k-means algorithm is used to cluster each class of samples and divide them into several subclasses.

[0019] Furthermore, in step S3 of the present invention, each subclass in all classes has three attributes: the category to which it belongs, the subclass within the class, and the category to which the subclass between classes belongs.

[0020] Furthermore, step S4 of the present invention includes the following steps:

[0021] 1) If the total number of samples in one of the subclasses does not reach the number allocated for the subclass, all samples of the subclass are selected, and the number that needs to be increased is calculated so that the increased number can reach the number allocated for the subclass; otherwise, the following steps are performed to screen the samples;

[0022] 2) For each sample vector in a subclass, find its distance from the center of its corresponding subclass and the distance from the center of each subclass of other classes, and give priority to selecting points whose distance from the center of a subclass of other classes is smaller than the distance from the center of the subclass to which it belongs, until the number of points allocated to the subclass is reached;

[0023] 3) If after step 2), the current subclass has not reached the number allocated to the subclass, the unselected samples of the subclass are fitted with a half-edge Gaussian according to their distance from the subclass to the center, with the farthest distance as the expectation, and then sampled according to the symmetric Gaussian to select samples. Then, the remaining samples are sampled and selected according to the Gaussian fitting method until the selected samples reach the number allocated to the subclass.

[0024] Furthermore, the construction of the shape model of the present invention includes the following steps:

[0025] S11. Dataset collection and division: obtain the annotations of all existing samples of different products and different workstations, and convert these annotations into 8-bit binary images; each piece of data of the same product and the same workstation is called a class, and is randomly divided into training sets and test sets based on the class;

[0026] S12, classifying the training set according to different preprocessing algorithms; the classification categories include linear, hairy, block and sheet;

[0027] S13, iterative training model;

[0028] S14, reclassify the samples of each category that have been classified in the training set in step S2;

[0029] S15. Repeat step S3 to obtain a large shape model that can be used to classify the annotated shapes.

[0030] The beneficial effects of the present invention are that the defects existing in the background technology are solved.

[0031] 1. For inter-class balance, the present invention obtains sufficient appearance information by extracting features of sample shape, image, and text description, and balances samples within a class through appearance, thereby solving the problem of imbalanced appearance within a class;

[0032] 2. To solve the problems of manual appearance equalization, the present invention extracts features based on a large shape model and a clip model, and combines the extracted features with a machine learning clustering algorithm to perform clustering equalization, which is more accurate and efficient than manual appearance equalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of the process of the present invention;

[0034] Figure 2 It is a flow chart of the construction of a large model of shape and the application of the present invention;

[0035] Figure 3 It is a flow chart of the appearance feature extraction of the present invention. DETAILED DESCRIPTION

[0036] The present invention will now be described in further detail in conjunction with the accompanying drawings and preferred embodiments. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0037] like Figure 1-Figure 3 A sample balancing method based on a large shape model is shown. By training a large shape-based model and combining the information of mutual conversion between text and image, features with strong separability are generated. The samples are screened with a machine learning algorithm so that the samples are evenly distributed in categories and appearances. By balancing samples of various categories and appearances, higher quality and more representative samples are obtained, which improves the quality of training data and thus increases the upper limit of the model effect.

[0038] The overall steps are as follows Figure 1 As shown:

[0039] 1. Train to get a large shape model

[0040] The overall steps of large model training are as follows Figure 2 As shown;

[0041] 1. Dataset collection and division

[0042] Get all existing samples of different products and different workstations, get their annotations, and convert these annotations into 8-bit binary images; each piece of data of the same product and the same workstation is called a class. Suppose there are N classes of samples in total. The N classes of samples are randomly divided into training sets and test sets based on classes. The training set has n_train classes and the test set has n_test classes.

[0043] N=n_train+n_test,n_train:n_test=4:1;

[0044] The present invention requires all sample small images to be of the same size. The size used in the present invention is 256*256. For small images with a size less than 256, the four sides are expanded with equal boundaries, and the grayscale of the expanded pixels is 0. The expanded size is 256*256. For images with a size greater than 256, a 256*256 window is used for sliding window cutting.

[0045] 2. Initialize the classification of the training set

[0046] The training set is classified into the following categories according to the following preprocessing algorithm:

[0047] (1) Linear

[0048] That is, a long and straight linear area is recorded as area A. Hereinafter, area A refers to the linear area;

[0049] a. First, obtain the equivalent ellipse parameters of the annotated area of ​​the current sample image:

[0050] M20=∑(r0-r) 2

[0051] M02=∑(c0-c) 2

[0052] M11=∑(r0-r)(c0-c)

[0053]

[0054] in:

[0055] r0, c0 are the row and column coordinates of the center of gravity of the region, r, c are the row and column coordinates of each point on the region;

[0056] Ra, Rb refer to the major and minor semi-axis of the regional equivalent ellipse, respectively;

[0057] b. Then calculate the anisotropy of the region based on the equivalent ellipse parameters:

[0058] anisometry = Ra / Rb;

[0059] c. Classify samples with anisometry>14.75 as linear;

[0060] (2) Hairy

[0061] That is, the short and curved linear region is recorded as region B. Hereinafter, region B refers to the hairy region;

[0062] Perform the following operations on the remaining samples in the previous step:

[0063] a. Get the central axis of the region:

[0064] That is, if the region satisfies: at least two boundary points of the region have the same distance to the point, then the point is a point on the axis of the region;

[0065] Obtain all points that meet the requirements in the annotated area of ​​the current sample image to form the central axis of the area;

[0066] b. Get the narrow length of the area:

[0067] dist=mean(d1,d2,...dm), where d1,d2,...dm represent the minimum distance from the point on the central axis to the regional boundary, m is the number of points on the central axis, and mean represents the average;

[0068] c. Classification as required:

[0069] Samples with dist < 2.5, m < 200, and anisometry < 8 (the meaning and calculation method of anisometry are the same as in step 1) are classified as hairy;

[0070] (3) Block

[0071] That is, the area is large, not very narrow and long, and the number of connected domains is small, which is recorded as region C. The following region C refers to the block region;

[0072] Perform the following operations on the remaining samples in the previous step:

[0073] a. Requirement 1:

[0074] max(a1,a2...ac)>100, c<=3;

[0075] Where a1, a2...ac are the areas of each connected domain, and c is the number of connected domains;

[0076] b. Requirement 2:

[0077] anisometry < 6;

[0078] If the annotated area of ​​the current sample image meets the above requirements, it is classified as blocky;

[0079] (4) Flake

[0080] That is, the total area is large, but the area of ​​a single connected domain is small, not very narrow and long, and the number of connected domains is large, recorded as region D. The following region D refers to the sheet region;

[0081] Perform the following operations on the remaining samples in the previous step:

[0082] a. Requirement 1:

[0083] max(a1,a2...ac)<60, c>=5;

[0084] b. Requirement 2:

[0085] anisometry < 6;

[0086] If the annotated area of ​​the current sample image meets the above requirements, it is classified as sheet-like;

[0087] (5) Other categories

[0088] The samples in the training set that are not classified into the above categories are classified as other categories, recorded as area E. The following area E refers to other category areas;

[0089] At this point, the training set is divided into five categories: linear, hairy, blocky, flake-like, and other categories;

[0090] 3. Iterative training model

[0091] Select a classic network. The present invention uses resnet-50, and change the output layer to 4 categories, which correspond to the four categories of linear, hairy, blocky, and flake in step 2;

[0092] Send these 4 types of samples in the training set into the network for training;

[0093] After training is completed, use the test set for testing;

[0094] Observe the classification of the test set. For the misclassified test samples, manually judge which category they should belong to. Select samples with similar appearance to the misclassified samples from other classes in the training set and put them into the corresponding class participating in the training.

[0095] Repeat this step for iterative training until the network classifies the test set in a way that meets human judgment standards;

[0096] 4. Refine your training

[0097] Based on the classified samples of the training set involved in the training, the samples in the training set are further classified by manual observation or algorithm design according to the needs of the specific scene. In the present invention, each of the above 4 types of samples is further divided into 2 categories:

[0098] (1) Linear

[0099] Divided into straight lines and curves;

[0100] (2) Hairy

[0101] It is divided into single-arc curved hair wire (such as "C" shape) and multi-arc curved hair wire (such as "S" shape);

[0102] (3) Block

[0103] It is divided into single block (i.e., only one connected domain) and multi-block (i.e., more than one connected domain);

[0104] (4) Flake

[0105] It is divided into dense flakes (the shortest average distance between connected domains is less than or equal to 5) and sparse flakes (the shortest average distance between connected domains is greater than 5);

[0106] After the above subdivision, the samples involved in training are divided into 8 categories. The previously trained network is used, and the output layer is replaced with 8 categories (the number of categories in other application scenarios is replaced with the corresponding classification output layer). The network model is iteratively trained according to the previous method until it meets the standard of manual judgment on the test set;

[0107] At this point, a large shape model that can be used to classify the annotated shapes is obtained, which can be used for subsequent data enhancement.

[0108] 2. Sample balancing based on large shape model

[0109] 1. Appearance feature extraction

[0110] like Figure 3 As shown:

[0111] (1) Shape characteristics

[0112] Select the features extracted from a certain layer of the shape model (the layer before the output layer is used in the present invention), and the output obtained is the shape feature, which is used as part of the appearance feature;

[0113] (2) Image features

[0114] The image feature extractor is the image encoder of the clip network. The original image is input into the clip image encoder, and the output obtained is the image feature, which is part of the appearance feature.

[0115] (3) Text features

[0116] The original image is input into a graph-to-text network to generate a text description of the image. The graph-to-text network of the present invention adopts gpt-4. The generated text description is input into a text feature extractor. The text feature extractor of the present invention adopts a text encoder of the clip network. The output obtained is the text feature, which is a part of the appearance feature;

[0117] (4) Splicing

[0118] The obtained shape features, image features, and text features are spliced ​​together, and the feature vector obtained by splicing is the appearance feature, due to the subsequent processing steps;

[0119] 2. Clustering within a class

[0120] The samples in each category are clustered based on their appearance features using the k-means algorithm, where the clustering category parameter m is determined as follows:

[0121] (1) Constructing loss function

[0122] The horizontal axis is m, the value of n in the present invention is 0-10, the sampling step is 1, and the vertical axis is Where k is the number of samples in the i-th class in the k-means clustering result, x j is the appearance feature vector of each sample in the i-th class, uc iis the center of all sample appearance feature vectors of the i-th class, that is, the average of the Euclidean distances from the samples in each class clustered by k-means to the center of that class;

[0123] (2) Determination of the number of categories

[0124] Find the second-order derivative of the constructed loss function, and select the horizontal coordinate of the point with the largest second-order derivative, which is the number of categories m;

[0125] (3) Classification within a category

[0126] According to the appearance feature vector of samples in each category and the determined parameters, the k-means algorithm is used to cluster each category of samples and divide them into several subcategories;

[0127] 3. Inter-class sub-class clustering

[0128] Assuming that there are n classes of samples, each class is clustered by k-means to obtain subclasses, and all subclasses of all classes are recorded as C 1,i_1 C 1,i_2 ...C 1,i_s1 ...C n,i_s1 ...C n,i_sn ;

[0129] Where s1, s2...sn is the number of subclasses of the 1st, 2nd...nth class, i_s1, i_s2...i_sn represents the label of each subclass in each class;

[0130] The centers of these subclasses are denoted as u 1,i_1 u 1,i_2 ...u 1,i_s1 ...u n,i_s1 ...u n,i_sn ;

[0131] Use the method in step 2 to convert u 1,i_1 u 1,i_2 ...u 1,i_s1 ...u n,i_s1 ...u n,i_sn Clustering is performed, and the clustering results are recorded as W1 W2W3...W g , where g is the number of categories into which these subclass centers are clustered as samples;

[0132] So far, each subclass in all classes has 3 attributes:

[0133] (1) The category to which it belongs, denoted as a,

[0134] (2) The subclass within the class, denoted as b,

[0135] (3) The category to which the inter-class subclass belongs, denoted as d;

[0136] Each subclass can be denoted by S a,b,d ,in:

[0137] a=1,2...n;

[0138] b=1,2...s a ;s a is the number of subclasses of the ath class;

[0139] d=1,2,...g;

[0140] 4. Calculate the number of additions and deletions of each subcategory

[0141] Assume that the total number of samples required is n_need, and the minimum number of samples in the current sample category is n_min;

[0142] (1) Quantity allocation:

[0143] The number of samples selected for each category is n_s = n_need / a;

[0144] Balance the appearance within each class:

[0145] Assume that the number of subclasses in the current class is n_b, then the number of samples taken from each subclass is n_s_b = n_s / n_b;

[0146] (2) Select by allocation quantity

[0147] 1) If the total number of samples of the subclass does not reach the number allocated for the subclass, all samples of the subclass are selected and the number that needs to be increased is calculated so that the increased number can reach the number allocated for the subclass;

[0148] Otherwise, perform the following steps to screen the samples;

[0149] 2) For each sample vector in a subclass, find its distance from the center of its corresponding subclass and the distance from the center of each subclass of other classes, and give priority to selecting points whose distance from the center of a subclass of other classes is smaller than the distance from the center of the subclass to which it belongs, until the number of points allocated to the subclass is reached;

[0150] 3) If after step 2), the current subclass has not reached the number allocated to the subclass, the unselected samples of the subclass are subjected to half-edge Gaussian fitting according to their distance from the subclass to the center, with the farthest distance as the expectation, and then sampled according to the symmetric Gaussian to select samples, and then the remaining samples are subjected to Gaussian fitting sampling and selection according to this method until the selected samples reach the number allocated to the subclass;

[0151] Step 1) determines the number of samples to be added to each subclass, and the number of samples to be added to each subclass. Steps 2) and 3) filter out the samples, that is, determine the samples to be deleted. So far, the number of samples added and deleted in each subclass has been obtained.

[0152] 5. Sample simplification and enhancement

[0153] (1) Sample reduction

[0154] For the subclasses that need to streamline samples, delete the unselected samples according to the samples screened in step 4;

[0155] (2) Sample enhancement

[0156] For subclasses that need additional samples, a data augmentation method based on a large model (in another patent) is used to enhance the samples to reach the allocated number.

[0157] At this point, we have obtained samples that are balanced in category and appearance. This sample can be used for training, which will greatly improve the data quality and training effect.

[0158] The above description only describes the specific implementation mode of the present invention. Various examples do not limit the essential content of the present invention. After reading the description, ordinary technicians in the relevant technical field can modify or deform the specific implementation modes described above without departing from the essence and scope of the invention.

Claims

1. A sample equalization method based on a large shape model, characterized in that: The following steps are included: S1. Extract the appearance features; and concatenate the features, and the concatenated feature vector is the appearance feature; S2, intra-class clustering: cluster the samples in each class based on their appearance features using the k-means algorithm; S3, inter-class sub-class clustering; each class is clustered by k-means to obtain sub-classes, and the centers of the sub-classes are clustered again; S4. Calculate the number of additions and deletions of each subcategory; S5. For subcategories that need to streamline samples, delete the unselected samples according to the samples screened in step S4; for subcategories that need to increase samples, perform sample enhancement to reach the allocated number.

2. A sample equalization method based on a large shape model as claimed in claim 1, characterized in that: In the step S1, the appearance features include: shape features, image features and text features; the shape features are features extracted and output by selecting one layer of the shape model.

3. The sample equalization method based on a large shape model as claimed in claim 1, characterized in that: In step S2, the clustering category parameter m is determined by: (1) Construct the loss function: the horizontal axis is m, the sampling step is 1, and the vertical axis is Where k is the number of samples in the i-th class in the k-means clustering result, x j is the appearance feature vector of each sample in the i-th class, uci is the center of the appearance feature vectors of all samples in the i-th class, that is, the average Euclidean distance from the sample in each class clustered by k-means to the center of the class; (2) Determine the number of categories: Calculate the second-order derivative of the constructed loss function and select the horizontal coordinate of the point with the largest second-order derivative, which is the number of categories m; (3) Intra-class classification: Based on the appearance feature vectors of samples in each class and the determined parameters, the k-means algorithm is used to cluster each class of samples and divide them into several subclasses.

4. The sample equalization method based on a large shape model as claimed in claim 1, characterized in that: In the step S3, each subclass in all classes has three attributes: the category to which it belongs, the subclass within the class, and the category to which the subclass between classes belongs.

5. The sample equalization method based on a large shape model as claimed in claim 1, characterized in that: In the step S4, The following steps are included: 1) If the total number of samples in a subclass does not reach the number allocated for that subclass, all samples of that subclass are selected and the number that needs to be increased is calculated so that the increased number can reach the number allocated for that subclass; Otherwise, perform the following steps to screen the samples; 2) For each sample vector in a subclass, find its distance from the center of its corresponding subclass and the distance from the center of each subclass of other classes, and give priority to selecting points whose distance from the center of a subclass of other classes is smaller than the distance from the center of the subclass to which it belongs, until the number of points allocated to the subclass is reached; 3) If after step 2), the current subclass has not reached the number allocated to the subclass, the unselected samples of the subclass are fitted with a half-edge Gaussian according to their distance from the subclass to the center, with the farthest distance as the expectation, and then sampled according to the symmetric Gaussian to select samples. Then, the remaining samples are sampled and selected according to the Gaussian fitting method until the selected samples reach the number allocated to the subclass.

6. A sample equalization method based on a large shape model as claimed in claim 2, characterized in that: The construction of the shape model includes the following steps: S11. Dataset collection and division: obtain the annotations of all existing samples of different products and different workstations, and convert these annotations into 8-bit binary images; each piece of data of the same product and the same workstation is called a class, and is randomly divided into training sets and test sets based on the class; S12, classifying the training set according to different preprocessing algorithms; the classification categories include linear, hairy, block and sheet; S13, iterative training model; S14, reclassify the samples of each category that have been classified in the training set in step S2; S15. Repeat step S3 to obtain a large shape model that can be used to classify the annotated shapes.

Images