A sample retrieval method based on large shape model

Abstract

The present invention relates to a sample retrieval method based on a large shape model, comprising steps S1, inputting over-killed and missed samples in a test set to be retrieved, i.e., template samples, and training set samples of different classes from the template samples, i.e., labeled images of different classes of samples, into the large shape model for shape classification; S2, large model pre-classification; S3, feature similarity statistics; S4, display sample selection; S5, interactive feedback; S6, final retrieval. As a deep learning over-kill and missed detection analysis tool, the present invention analyzes and solves the over-kill and missed detection problems from the sample level, which can fundamentally solve the problem compared to strategies based on the network level, and is more efficient and simple than strategies based on the network level, saving time and manpower for adjusting the network as well as equipment costs.

Description

Technical Field

[0001] The invention relates to the technical field of machine vision detection, in particular to a sample retrieval method based on a large shape model. Background Art

[0002] Whether in deep learning competitions or in actual projects, sample retrieval is a key link in data processing. Its accuracy and efficiency are directly related to the effect of subsequent model training and the success or failure of project implementation. In the process of sample retrieval, manual screening, as a traditional and intuitive method, has certain applicability in specific situations. However, for manual screening, manual sample retrieval is a relatively effective method when the sample size is small, but when the sample size is large, the workload will be very large and time-consuming, and due to unstable factors such as human fatigue and emotions, the sample retrieval effect will also be unstable.

[0003] In addition, based on the characteristics of over-killing and missed detection samples, the problem of over-killing and missed detection can be solved by modifying the network structure, adjusting network parameters, etc.; however, for network-level strategies, a lot of analysis and experiments are required. Whether it is modifying the network structure or adjusting the parameters, it is a very complex and tedious process, and the controllability is not strong. The effect is often unstable and difficult to guarantee. In essence, this method allows the network to make a special distinction between over-killing and missed detection samples, which improves the network's fitting ability and easily leads to overfitting. Summary of the invention

[0004] The technical problem to be solved by the present invention is: to provide a sample retrieval method based on a large shape model, which solves the problem of optimizing the over-kill and missed detection rates from the sample level by retrieving samples similar to over-kill and missed detection samples in a training set and processing them in projects using deep learning.

[0005] The technical solution adopted by the present invention to solve the technical problem is: a sample retrieval method based on a large shape model, comprising the following steps:

[0006] S1. Input the over-detected and missed samples in the test set to be retrieved, i.e., the template samples, and the training set samples of different classes from the template samples, i.e., the labeled images of samples of different classes, into the shape large model for shape classification;

[0007] S2, large model pre-classification: select samples whose shape classification belongs to the same shape category as the template sample among samples of different classes, i.e., samples to be retrieved; use the template sample to search for samples similar to the template sample among samples to be retrieved;

[0008] S3. Feature similarity statistics: perform multi-feature similarity comparison between over-detected and missed samples and samples to be retrieved, and obtain similarity scores of each feature of samples to be retrieved;

[0009] S4, sample selection for display: select the samples with the highest similarity ranking in each feature of the samples to be retrieved in turn, and display the selected samples after the number of samples that meets the set threshold is reached;

[0010] S5, interactive feedback: similar samples are manually confirmed;

[0011] S6. Final retrieval: The selected samples are selected with the top-ranked similarity as the final retrieval similarity, and the final retrieval similarity is used to retrieve the samples to be retrieved to obtain the retrieval results.

[0012] Furthermore, in step S3 described in the present invention, features are extracted from the template image and the image to be retrieved, and the features include: grayscale histogram of the original image, gradient direction histogram of the original image, gradient direction histogram of the label image, average grayscale of the original image, average grayscale of the label part of the original image, maximum connected domain area of ​​the label image, number of connected domains of the label image, entropy of the label part of the original image, and contrast between the label part of the original image and the surrounding background part.

[0013] Furthermore, in step S3 described in the present invention, the similarity of the average grayscale of the original image, the average grayscale of the label part of the original image, the maximum connected domain area of ​​the label map, the number of connected domains of the label map, the entropy of the label part of the original image, and the contrast between the label part of the original image and the surrounding background part is calculated by directly using the absolute value of the difference between the eigenvalue of the template sample and the eigenvalue of the image to be matched. The smaller the value, the higher the similarity.

[0014] Furthermore, in step S3 of the present invention, the similarity of the average grayscale of the original image is calculated by using the Bhattacharyya distance, and the formula is:

[0015]

[0016] in:

[0017] H1 and H2 are two histograms whose similarity is to be measured. and are the average values ​​of the two histograms, I is the value of each bin of the histogram, N is the number of bins, and the smaller the value is, the higher the similarity is.

[0018] Furthermore, in step S3 of the present invention, the similarity of the average grayscale of the label part in the original image and the maximum connected domain area of ​​the label image is calculated by cyclically shifting each bin of H2; assuming H2 to be (h(1), h(2)...h(N)); then the cyclic shift is expanded to (h(2)...h(N), h(1))...(h(N), h(N-1)...h(1)) for a total of N histograms, according to the formula:

[0019]

[0020] Calculate the similarity between the histogram of H1 and the N histograms obtained by expanding H2, and take the smallest value among the N values ​​as the result. The smaller the value, the higher the similarity.

[0021] Among them: H1 and H2 are two histograms to measure similarity. and are the average values ​​of the two histograms, I is the value of each bin of the histogram, and N is the number of bins.

[0022] Furthermore, in step S1 of the present invention, the construction of the large shape model includes the following steps:

[0023] S11. Dataset collection and division: obtain the annotations of all existing samples of different products and different workstations, and convert these annotations into 8-bit binary images; each piece of data of the same product and the same workstation is called a class, and is randomly divided into training sets and test sets based on the class;

[0024] S12, classifying the training set according to different preprocessing algorithms; the classification categories include linear, hairy, block and sheet;

[0025] S13, iterative training model;

[0026] S14, reclassify the samples of each category that have been classified in the training set in step S2;

[0027] S15. Repeat step S3 to obtain a large shape model that can be used to classify the annotated shapes.

[0028] The beneficial effects of the present invention are that the defects existing in the background technology are solved.

[0029] 1. For the problems existing in manual screening, the present invention uses a designed method to automatically execute on the program side, almost without manual intervention, thus solving the problems that may exist due to manual operation;

[0030] 2. For the problems existing in the network-side strategy, the present invention uses over-killed and missed samples to retrieve similar samples in the training set. These retrieved similar samples can be eliminated or processed in other ways to solve the problem at the sample level. The effect is stable and can solve the problem more fundamentally. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a flow chart of the present invention;

[0032] Figure 2It is a flow chart of the construction of a large shape model and the application of the present invention. DETAILED DESCRIPTION

[0033] The present invention will now be described in further detail in conjunction with the accompanying drawings and preferred embodiments. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0034] like Figure 1-Figure 2 The sample retrieval method based on a large shape model is shown. By training a large shape-based model and combining it with other features and subsequent interactive processing methods, it automatically retrieves samples that the user feels are similar, thereby helping the user analyze whether over-killing or missed detection is caused by sample problems, and then processes similar samples.

[0035] The specific steps are as follows:

[0036] 1. Train to get a large shape model. The overall steps of training a large shape model are as follows: Figure 2 As shown;

[0037] 1. Dataset collection and division

[0038] Get all existing samples of different products and different workstations, get their annotations, and convert these annotations into 8-bit binary images; each piece of data of the same product and the same workstation is called a class. Suppose there are N classes of samples in total. The N classes of samples are randomly divided into training sets and test sets based on classes. The training set has n_train classes and the test set has n_test classes.

[0039] N=n_train+n_test,n_train:n_test=4:1;

[0040] The present invention requires all sample small images to be of the same size. The size used in the present invention is 256*256. For small images with a size less than 256, the four sides are expanded with equal boundaries, and the grayscale of the expanded pixels is 0. The expanded size is 256*256. For images with a size greater than 256, a 256*256 window is used for sliding window cutting.

[0041] 2. Initialize the classification of the training set

[0042] The training set is classified into the following categories according to the following preprocessing algorithm:

[0043] (1) Linear

[0044] That is, a long and straight linear area is recorded as area A. Hereinafter, area A refers to the linear area;

[0045] a. First, obtain the equivalent ellipse parameters of the annotated area of ​​the current sample image:

[0046] M20=∑(r0-r) 2

[0047] M02=∑(c0-c) 2

[0048] M11=∑(r0-r)(c0-c)

[0049]

[0050] in:

[0051] r0, c0 are the row and column coordinates of the center of gravity of the region, and r, c are the row and column coordinates of each point on the region;

[0052] Ra, Rb refer to the major and minor semi-axis of the regional equivalent ellipse, respectively;

[0053] b. Then calculate the anisotropy of the region based on the equivalent ellipse parameters:

[0054] anisometry = Ra / Rb;

[0055] c. Classify samples with anisometry>14.75 as linear;

[0056] (2) Hairy

[0057] That is, the short and curved linear region is recorded as region B. Hereinafter, region B refers to the hairy region;

[0058] Perform the following operations on the remaining samples in the previous step:

[0059] a. Get the central axis of the region:

[0060] That is, if the region satisfies: at least two boundary points of the region have the same distance to the point, then the point is a point on the axis of the region;

[0061] Obtain all points that meet the requirements in the annotated area of ​​the current sample image to form the central axis of the area;

[0062] b. Get the narrow length of the area:

[0063] dist=mean(d1,d2,...dm), where d1,d2,...dm represent the minimum distance from the point on the central axis to the regional boundary, m is the number of points on the central axis, and mean represents the average;

[0064] c. Classification as required:

[0065] Samples with dist < 2.5, m < 200, and anisometry < 8 (the meaning and calculation method of anisometry are the same as in step 1) are classified as hairy;

[0066] (3) Block

[0067] That is, the area is large, not very narrow and long, and the number of connected domains is small, which is recorded as region C. The following region C refers to the block region;

[0068] Perform the following operations on the remaining samples in the previous step:

[0069] a. Requirement 1:

[0070] max(a1,a2...ac)>100, c<=3;

[0071] Where a1, a2...ac are the areas of each connected domain, and c is the number of connected domains;

[0072] b. Requirement 2:

[0073] anisometry < 6;

[0074] If the annotated area of ​​the current sample image meets the above requirements, it is classified as blocky;

[0075] (4) Flake

[0076] That is, the total area is large, but the area of ​​a single connected domain is small, not very narrow and long, and the number of connected domains is large, recorded as region D. The following region D refers to the sheet region;

[0077] Perform the following operations on the remaining samples in the previous step:

[0078] a. Requirement 1:

[0079] max(a1,a2...ac)<60, c>=5;

[0080] b. Requirement 2:

[0081] anisometry < 6;

[0082] If the annotated area of ​​the current sample image meets the above requirements, it is classified as sheet-like;

[0083] (5) Other categories

[0084] The samples in the training set that are not classified into the above categories are classified as other categories, recorded as area E. The following area E refers to other category areas;

[0085] At this point, the training set is divided into five categories: linear, hairy, blocky, flake-like, and other categories;

[0086] 3. Iterative training model

[0087] Select a classic network. The present invention uses resnet-50, and change the output layer to 4 categories, which correspond to the four categories of linear, hairy, blocky, and flake in step 2;

[0088] Send these 4 types of samples in the training set into the network for training;

[0089] After training is completed, use the test set for testing;

[0090] Observe the classification of the test set. For the misclassified test samples, manually judge which category they should belong to. Select samples with similar appearance to the misclassified samples from other classes in the training set and put them into the corresponding class participating in the training.

[0091] Repeat this step for iterative training until the network classifies the test set in accordance with human judgment standards;

[0092] 4. Refine your training

[0093] Based on the classified samples of the training set involved in the training, the samples in the training set are further classified by manual observation or algorithm design according to the needs of the specific scene. In the present invention, each of the above 4 types of samples is further divided into 2 categories:

[0094] (1) Linear

[0095] Divided into straight lines and curves;

[0096] (2) Hairy

[0097] It is divided into single-arc curved hair wire (such as "C" shape) and multi-arc curved hair wire (such as "S" shape);

[0098] (3) Block

[0099] It is divided into single block (i.e., only one connected domain) and multi-block (i.e., more than one connected domain);

[0100] (4) Flake

[0101] It is divided into dense flakes (the shortest average distance between connected domains is less than or equal to 5) and sparse flakes (the shortest average distance between connected domains is greater than 5);

[0102] After the above subdivision, the samples involved in training are divided into 8 categories. The previously trained network is used, and the output layer is replaced with 8 categories (the number of categories in other application scenarios is replaced with the corresponding classification output layer). The network model is iteratively trained according to the previous method until it meets the standard of manual judgment on the test set;

[0103] At this point, a large shape model is obtained that can be used to classify the annotated shapes and can be used for subsequent image retrieval.

[0104] 2. Pre-classification of large shape models

[0105] Input the labeled images of the over-killed and missed samples (hereinafter referred to as template samples) in the test set to be retrieved and the training set samples of different classes from the template samples (hereinafter referred to as different class samples) into the large model for shape classification, and select the samples whose shape classification belongs to the same shape category as the template samples from the different class samples, which are referred to as samples to be retrieved, that is, use the template samples to search for samples similar to the template samples in the samples to be retrieved;

[0106] 3. Feature Similarity Statistics

[0107] 1. Feature Statistics

[0108] Extract features from the template image and the image to be retrieved. The features include:

[0109] (1) Grayscale histogram of the original image; the grayscale range is 0-255, and the number of bins is 64;

[0110] (2) Gradient direction histogram of the original image; the value range is 0-360, and the number of bins is 36;

[0111] (3) Label image gradient direction histogram; the value range is 0-360, and the number of bins is 36;

[0112] (4) Average grayscale of the original image;

[0113] (5) The average grayscale of the label part in the original image;

[0114] (6) The maximum connected domain area of ​​the label graph;

[0115] (7) The number of connected domains in the label graph;

[0116] (8) Entropy of the label part of the original image;

[0117] (9) The contrast between the label part and the surrounding background part in the original image;

[0118] That is, the label is expanded by a 5*5 circle center element. The difference between the expanded area and the label area is called the surrounding area. The required contrast is the difference between the average grayscale of the label area and the average grayscale of the surrounding area.

[0119] 2. Calculate similarity

[0120] For features (4)(5)(6)(7)(8)(9), directly use the difference between the feature value of the template sample and the feature value of the image to be matched and take the absolute value to get the result. The smaller the value, the higher the similarity.

[0121] For (1)

[0122] Use Bhattacharyya distance to measure, the formula is:

[0123]

[0124] in:

[0125] H1 and H2 are two histograms whose similarity is to be measured. and are the average values ​​of the two histograms, I is the value of each bin of the histogram, N is the number of bins, and the smaller the value is, the higher the similarity is;

[0126] For (2) and (3)

[0127] Circularly shift each bin of H2 and set H2 to (h(1),h(2).....h(N));

[0128] Then, it can be expanded into N histograms (h(2).....h(N), h(1))...(h(N), h(N-1))....h(1)) through cyclic shift. The similarity between the histogram of H1 and the N histograms obtained by expanding H2 is calculated according to formula (1). The smallest of the N values ​​is taken as the result. The smaller the value, the higher the similarity.

[0129] 4. Selection of display samples

[0130] Select the samples with the highest similarity ranking in various feature selections in turn, remove the duplicated samples, until 50 samples are selected, and display the selected samples on the interface;

[0131] 5. Interactive Feedback

[0132] The user selects samples that are considered similar in the interface;

[0133] 6. Final Search

[0134] The samples selected by the user are taken as the final retrieval similarity with the top ranking similarity. The final retrieval similarity is used to perform a new round of retrieval on the samples to be retrieved. The rankings of all samples to be retrieved in all features are counted. The sum of all ranking numbers of each sample is taken as the retrieval value. The samples to be retrieved are sorted according to the size of the retrieval value, and the samples to be retrieved are provided to the user according to the sorting. The higher the ranking, the more similar it is to the template sample. Users refer to the ranking to select similar samples and process them.

[0135] The above description only describes the specific implementation mode of the present invention. Various examples do not limit the essential content of the present invention. After reading the description, ordinary technicians in the relevant technical field can modify or deform the specific implementation modes described above without departing from the essence and scope of the invention.

Claims

1. A sample retrieval method based on a large shape model, characterized in that: The following steps are included: S1. Input the over-detected and missed samples in the test set to be retrieved, i.e., the template samples, and the training set samples of different classes from the template samples, i.e., the labeled images of samples of different classes, into the shape large model for shape classification; S2, large model pre-classification: select samples whose shape classification belongs to the same shape category as the template sample among samples of different classes, i.e., samples to be retrieved; use the template sample to search for samples similar to the template sample among samples to be retrieved; S3. Feature similarity statistics: perform multi-feature similarity comparison between over-detected and missed samples and samples to be retrieved, and obtain similarity scores of each feature of samples to be retrieved; S4, sample selection for display: select the samples with the highest similarity ranking in each feature of the samples to be retrieved in turn, and display the selected samples after the number of samples that meets the set threshold is reached; S5, interactive feedback: similar samples are manually confirmed; S6. Final retrieval: The selected samples are selected with the top-ranked similarity as the final retrieval similarity, and the final retrieval similarity is used to retrieve the samples to be retrieved to obtain the retrieval results.

2. A sample retrieval method based on a large shape model as claimed in claim 1, characterized in that: In the step S3, features are extracted from the template image and the image to be retrieved, and the features include: grayscale histogram of the original image, gradient direction histogram of the original image, gradient direction histogram of the label image, average grayscale of the original image, average grayscale of the label part of the original image, maximum connected domain area of ​​the label image, number of connected domains of the label image, entropy of the label part of the original image, and contrast between the label part of the original image and the surrounding background part.

3. A sample retrieval method based on a large shape model as claimed in claim 2, characterized in that: In the step S3, the similarity of the average grayscale of the original image, the average grayscale of the label part in the original image, the maximum connected domain area of ​​the label map, the number of connected domains in the label map, the entropy of the label part in the original image, and the contrast between the label part in the original image and the surrounding background part is calculated by directly using the absolute value of the difference between the eigenvalue of the template sample and the eigenvalue of the image to be matched. The smaller the value, the higher the similarity.

4. A sample retrieval method based on a large shape model as claimed in claim 2, characterized in that: In step S3, the similarity of the average grayscale of the original image is calculated by using the Bhattacharyya distance, and the formula is: in: H1 and H2 are two histograms whose similarity is to be measured. and are the average values ​​of the two histograms, I is the value of each bin of the histogram, N is the number of bins, and the smaller the value is, the higher the similarity is.

5. The sample retrieval method based on a large shape model as claimed in claim 2, characterized in that: In step S3, the similarity between the average grayscale of the label part in the original image and the maximum connected domain area of ​​the label image is calculated by cyclically shifting each bin of H2; assuming H2 is (h(1), h(2)...h(N)); then the cyclic shift is expanded to (h(2)...h(N), h(1))...(h(N), h(N-1)...h(1)) for a total of N histograms, according to the formula: Calculate the similarity between the histogram of H1 and the N histograms obtained by expanding H2, and take the smallest value among the N values ​​as the result. The smaller the value, the higher the similarity. Among them: H1 and H2 are two histograms to measure similarity. and are the average values ​​of the two histograms, I is the value of each bin of the histogram, and N is the number of bins.

6. A sample retrieval method based on a large shape model as claimed in claim 1, characterized in that: In the step S1, the construction of the large shape model includes the following steps: S11. Dataset collection and division: obtain the annotations of all existing samples of different products and different workstations, and convert these annotations into 8-bit binary images; each piece of data of the same product and the same workstation is called a class, and is randomly divided into training sets and test sets based on the class; S12, classifying the training set according to different preprocessing algorithms; the classification categories include linear, hairy, block and sheet; S13, iterative training model; S14, reclassify the samples of each category that have been classified in the training set in step S2; S15. Repeat step S3 to obtain a large shape model that can be used to classify the annotated shapes.

Images