A shape-based large model and its application

Abstract

The present invention relates to a large model based on shape and its application, including the following construction steps: S1, data set collection and division; S2, classification of training sets according to different preprocessing algorithms; S3, iterative training model; S4, reclassification of samples of each category that have been classified in the training set of step S2; S5, repeating step S3 to obtain a large shape model that can be used to classify the labeled shapes. The present invention uses shape information to preprocess data, and then combines various subsequent methods such as similarity comparison and automatic clustering algorithm to meet specific needs, greatly reducing the difficulty of realizing sample retrieval, sample cleaning, sample balancing and other requirements, thereby greatly reducing the workload of human sample management and optimization, not only solving the problems caused by manual processing, but also improving efficiency and saving labor and time costs.

Description

Technical Field

[0001] The invention relates to the technical field of machine vision detection, in particular to a shape-based large model and application thereof. Background Art

[0002] In machine vision industrial inspection projects, especially in deep learning scenarios, sample cleaning and sample balancing are currently mainly handled manually in both academia and industry. Data cleaning and sample optimization are performed entirely manually. The human eye observes each sample image and adjusts the sample data based on experience and the test results of the network model. Adjusting sample data includes sample classification error correction, sample category ratio balancing, outlier sample deletion and other operations.

[0003] However, manual processing requires people to observe each image, which consumes manpower and time. When the number of samples is too large, such as reaching millions, it is difficult for people to observe them one by one. In addition, when people observe images, there are memory and sensory fluctuations, which may affect the quality of sample processing.

[0004] The algorithm-assisted approach is mostly to design algorithms based on the appearance characteristics of the current sample and specific needs, so as to achieve sample cleaning, sample balancing and other sample optimization tasks;

[0005] However, algorithm assistance requires algorithm personnel to design algorithms based on the image features and network training effects in the current project to assist manual processing. Algorithms need to be designed manually, and the designed algorithms can only be used for samples of the current project and cannot be universal.

[0006] In addition, for e-commerce image retrieval systems, the current systems used for image retrieval (which is considered sample retrieval because the sample itself is an image), such as Taobao's "Pai Li Tao", process product images on the platform. There are many types of images, and the differences between images are large; the product samples processed by the e-commerce image retrieval system are characterized by many types and large differences between different samples; while the sample images in industrial inspection based on machine vision are of fewer types, and the differences between samples are also smaller. The attribute characteristics of the objects processed are different, so it is not suitable for industrial inspection projects based on machine vision. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a large shape-based model and its application, which can be used in machine vision industrial inspection projects to solve the needs of sample cleaning, sample preliminary classification, sample balancing, sample retrieval, etc.

[0008] The technical solution adopted by the present invention to solve the technical problem is: a large model based on shape, including the following construction steps:

[0009] S1. Dataset collection and division: obtain the annotations of all existing samples of different products and different workstations, and convert these annotations into 8-bit binary images; each piece of data of the same product and the same workstation is called a class, and is randomly divided into training sets and test sets based on the class;

[0010] S2, classify the training set according to different preprocessing algorithms;

[0011] S3, iterative training model;

[0012] S4, reclassify the samples of each category that have been classified in the training set in step S2;

[0013] S5. Repeat step S3 to obtain a large shape model that can be used to classify the annotated shapes.

[0014] Furthermore, in step S2 of the present invention, the classification categories include linear, filamentous, block and sheet.

[0015] Furthermore, the classification method of the linear categories described in the present invention is:

[0016] A. Obtain the equivalent ellipse parameters of the long and straight linear area;

[0017] B. Calculate the anisotropy of the region based on the equivalent ellipse parameters;

[0018] C. Compare the value calculated in step B with the set threshold, and classify samples that are greater than the set threshold as linear.

[0019] Furthermore, the classification method of the hairy category of the present invention is as follows:

[0020] a. Get the median axis of the short and curved linear region:

[0021] b. Get the narrow length of the area;

[0022] c. Classify the samples that meet the set threshold as hairy.

[0023] Furthermore, the block categories described in the present invention are classified as follows:

[0024] I. Get a large area that is not very narrow and long and has a small number of connected domains;

[0025] II. The number of connected domains, the area of ​​each connected domain, and the anisotropy of the region that meet the set threshold are classified as blocky.

[0026] Furthermore, the classification method of the flake category of the present invention is as follows:

[0027] i. Get a region with a large total area but a small single connected domain area, which is not very narrow and long and has a large number of connected domains;

[0028] ii. The number of connected domains, the area of ​​each connected domain, and the anisotropy of the region that meet the set threshold are classified as flakes.

[0029] The shape-based model structure described in the present invention can be applied to sample retrieval, sample cleaning, sample balancing and other applications in the field of computer vision.

[0030] The beneficial effects of the present invention are that the defects existing in the background technology are solved.

[0031] 1. In order to solve the problem that manual processing consumes manpower and human memory and sensory fluctuations lead to unstable sample processing effects, the present invention trains a large shape model, uses shape information to pre-process the data, and then combines similarity comparison and automatic clustering algorithms and other subsequent methods to meet specific needs, greatly reducing the difficulty of realizing sample retrieval, sample cleaning, sample balancing and other needs, thereby greatly reducing the workload of human sample management and optimization, not only solving the problems caused by manual processing, but also improving efficiency and saving labor and time costs.

[0032] 2. For algorithm-assisted problems, the large shape model can be applied to any labeled data set, thus solving the problem of algorithm non-universality and saving the cost of algorithm personnel;

[0033] 3. The method for sample design in industrial inspection projects based on machine vision is more suitable for the application field targeted by the present invention (i.e., the industrial inspection field) than the e-commerce image retrieval system. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flow chart of the model structure construction and application of the present invention;

[0035] Figure 2 is a sample retrieval application flow chart of the present invention;

[0036] Figure 3 is a sample cleaning application flow chart of the present invention;

[0037] Figure 4 It is a sample equalization application flow chart of the present invention. DETAILED DESCRIPTION

[0038] The present invention will now be described in further detail in conjunction with the accompanying drawings and preferred embodiments. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0039] like Figure 1-Figure 4A large shape-based model and its application are shown in the figure. The overall steps are as follows: Figure 1 As shown:

[0040] 1. Dataset collection and division

[0041] Get all existing samples of different products and different workstations, get their annotations, and convert these annotations into 8-bit binary images; each piece of data of the same product and the same workstation is called a class. Suppose there are N classes of samples in total. The N classes of samples are randomly divided into training sets and test sets based on classes. The training set has n_train classes and the test set has n_test classes.

[0042] N=n_train+n_test,n_train:n_test=4:1;

[0043] The present invention requires all sample small images to be of the same size. The size used in the present invention is 256*256. For small images with a size less than 256, the four sides are expanded with equal boundaries, and the grayscale of the expanded pixels is 0. The expanded size is 256*256. For images with a size greater than 256, a 256*256 window is used for sliding window cutting.

[0044] 2. Initialize the classification of the training set

[0045] The training set is classified into the following categories according to the following preprocessing algorithm:

[0046] (1) Linear

[0047] That is, a long and straight linear area is recorded as area A. Hereinafter, area A refers to the linear area;

[0048] a. First, obtain the equivalent ellipse parameters of the annotated area of ​​the current sample image:

[0049] M20=∑(r0-r) 2

[0050] M02=∑(c0-c) 2

[0051] M11=∑(r0-r)(c0-c)

[0052]

[0053] in:

[0054] r0, c0 are the row and column coordinates of the center of gravity of the region, and r, c are the row and column coordinates of each point on the region;

[0055] Ra, Rb refer to the major and minor semi-axis of the regional equivalent ellipse, respectively;

[0056] b. Then calculate the anisotropy of the region based on the equivalent ellipse parameters:

[0057] anisometry = Ra / Rb;

[0058] c. Classify samples with anisometry>14.75 as linear;

[0059] (2) Hairy

[0060] That is, the short and curved linear region is recorded as region B. Hereinafter, region B refers to the hairy region;

[0061] Perform the following operations on the remaining samples in the previous step:

[0062] a. Get the central axis of the region:

[0063] That is, if the region satisfies: at least two boundary points of the region have the same distance to the point, then the point is a point on the axis of the region;

[0064] Obtain all points that meet the requirements in the annotated area of ​​the current sample image to form the central axis of the area;

[0065] b. Get the narrow length of the area:

[0066] dist=mean(d1,d2,...dm), where d1,d2,...dm represent the minimum distance from the point on the central axis to the regional boundary, m is the number of points on the central axis, and mean represents the average;

[0067] c. Classification as required:

[0068] Samples with dist < 2.5, m < 200, and anisometry < 8 (the meaning and calculation method of anisometry are the same as in step 1) are classified as hairy;

[0069] (3) Block

[0070] That is, the area is large, not very narrow and long, and the number of connected domains is small, which is recorded as region C. The following region C refers to the block region;

[0071] Perform the following operations on the remaining samples in the previous step:

[0072] a. Requirement 1:

[0073] max(a1,a2...ac)>100, c<=3;

[0074] Where a1, a2...ac are the areas of each connected domain, and c is the number of connected domains;

[0075] b. Requirement 2:

[0076] anisometry < 6;

[0077] If the annotated area of ​​the current sample image meets the above requirements, it is classified as blocky;

[0078] (4) Flake

[0079] That is, the total area is large, but the area of ​​a single connected domain is small, not very narrow and long, and the number of connected domains is large, recorded as region D. The following region D refers to the sheet region;

[0080] Perform the following operations on the remaining samples in the previous step:

[0081] a. Requirement 1:

[0082] max(a1,a2...ac)<60, c>=5;

[0083] b. Requirement 2:

[0084] anisometry < 6;

[0085] If the annotated area of ​​the current sample image meets the above requirements, it is classified as sheet-like;

[0086] (5) Other categories

[0087] The samples in the training set that are not classified into the above categories are classified as other categories, recorded as area E. The following area E refers to other category areas;

[0088] At this point, the training set is divided into five categories: linear, hairy, blocky, flake-like, and other categories;

[0089] 3. Iterative training model

[0090] Select a classic network. The present invention uses resnet-50, and change the output layer to 4 categories, which correspond to the four categories of linear, hairy, blocky, and flake in step 2;

[0091] Send these 4 types of samples in the training set into the network for training;

[0092] After training is completed, use the test set for testing;

[0093] Observe the classification of the test set. For the misclassified test samples, manually judge which category they should belong to. Select samples with similar appearance to the misclassified samples from other classes in the training set and put them into the corresponding class participating in the training.

[0094] Repeat this step for iterative training until the network classifies the test set in accordance with human judgment standards;

[0095] 4. Refine your training

[0096] Based on the classified samples of the training set involved in the training, the samples in the training set are further classified by manual observation or algorithm design according to the needs of the specific scene. In the present invention, each of the above 4 types of samples is further divided into 2 categories:

[0097] (1) Linear

[0098] Divided into straight lines and curves;

[0099] (2) Hairy

[0100] It is divided into single-arc curved hair wire (such as "C" shape) and multi-arc curved hair wire (such as "S" shape);

[0101] (3) Block

[0102] It is divided into single block (i.e., only one connected domain) and multi-block (i.e., more than one connected domain);

[0103] (4) Flake

[0104] It is divided into dense flakes (the shortest average distance between connected domains is less than or equal to 5) and sparse flakes (the shortest average distance between connected domains is greater than 5);

[0105] After the above subdivision, the samples involved in training are divided into 8 categories. The previously trained network is used, and the output layer is replaced with 8 categories (the number of categories in other application scenarios is replaced with the corresponding classification output layer). The network model is iteratively trained according to the previous method until it meets the standard of manual judgment on the test set;

[0106] At this point, a large shape model that can be used to classify the annotated shapes is obtained, that is, a shape-based model structure;

[0107] 5. Model application

[0108] The large shape model trained in the above steps can currently be applied to the following scenarios:

[0109] (1) Sample retrieval

[0110] Application steps: Figure 2 As shown:

[0111] The annotations of the samples to be searched and the annotations of the searched data set are sent to the shape model for shape classification. The samples in the searched data set that are classified into the same category as the samples to be searched are used as the coarse positioning sample set. Further original image similarity matching, such as histogram matching, feature point matching, etc., is performed in the coarse positioning sample set to achieve further matching in the coarse positioning sample set.

[0112] (2) Sample cleaning

[0113] Application steps: Figure 3 As shown:

[0114] The annotations of the dataset to be cleaned are sent to the large shape model for classification, and then humans decide which shape samples to reduce or delete, thereby achieving sample cleaning at the shape level.

[0115] (3) Sample Balance

[0116] Application steps: Figure 4 As shown:

[0117] The annotations of the data set to be balanced are sent to the shape model for classification, and then the sample balancing algorithm (such as selecting an equal number of samples for each shape) is used to obtain the shape balanced samples. Then, based on the shape balanced samples, a further balancing algorithm (such as balancing by statistical grayscale features) is used to further balance the original sample images to obtain the final balanced samples.

[0118] The above description only describes the specific implementation mode of the present invention. Various examples do not limit the essential content of the present invention. After reading the description, ordinary technicians in the relevant technical field can modify or deform the specific implementation modes described above without departing from the essence and scope of the invention.

Claims

1. A large shape-based model, characterized in that: The build steps include: S1. Dataset collection and division: obtain the annotations of all existing samples of different products and different workstations, and convert these annotations into 8-bit binary images; each piece of data of the same product and the same workstation is called a class, and is randomly divided into training sets and test sets based on the class; S2, classify the training set according to different preprocessing algorithms; S3, iterative training model; S4, reclassify the samples of each category that have been classified in the training set in step S2; S5. Repeat step S3 to obtain a large shape model that can be used to classify the annotated shapes.

2. A shape-based model structure as claimed in claim 1, characterized in that: In the step S2, the classification categories include line, filament, block and sheet.

3. A shape-based model structure as claimed in claim 2, characterized in that: The linear categories are classified as follows: A. Obtain the equivalent ellipse parameters of the long and straight linear area; B. Calculate the anisotropy of the region based on the equivalent ellipse parameters; C. Compare the value calculated in step B with the set threshold, and classify samples that are greater than the set threshold as linear.

4. A shape-based model structure as claimed in claim 2, characterized in that: The classification of the hairy and filamentous categories is as follows: a. Get the median axis of the short and curved linear region: b. Get the narrow length of the area; c. Classify the samples that meet the set threshold as hairy.

5. A shape-based model structure as claimed in claim 2, characterized in that: The block categories are classified as follows: I. Get a large area that is not very narrow and long and has a small number of connected domains; II. The number of connected domains, the area of ​​each connected domain, and the anisotropy of the region that meet the set threshold are classified as blocky.

6. A shape-based model structure as claimed in claim 2, characterized in that: The classification method of the flake category is: i. Get a region with a large total area but a small single connected domain area, which is not very narrow and long and has a large number of connected domains; ii. The number of connected domains, the area of ​​each connected domain, and the anisotropy of the region that meet the set threshold are classified as flakes.

7. An application of the shape-based model structure as claimed in claim 1, characterized in that: The shape macro model is applied to sample retrieval.

8. An application of the shape-based model structure as claimed in claim 1, characterized in that: The shape model is used for sample cleaning.

9. An application of the shape-based model structure as claimed in claim 1, characterized in that: The described shape model is applied to sample equalization.

10. An application of the shape-based model structure according to claim 1, characterized in that: The described large shape model has applications in the field of computer vision.

Images