A general detection method for machine vision

Abstract

The present invention relates to a general detection method for machine vision, comprising the following steps: S1, data labeling; S2, deep learning model pre-selection; S3, detection area rough extraction; S4, detection area fine extraction; S5, detection area fitting; S6, defect detection; S7, defect screening; S8, data integration. The present invention summarizes and classifies the traditional algorithms and deep learning algorithms commonly used in machine vision projects into various modules, uses these modules as various modules of the detection system, and adds a recommendation mechanism. Manually select modules according to experimental results for different application scenarios to form a detection system that meets the needs, and the entire module uses an open source library or is developed by itself, and no longer uses a commercial library; it not only reduces the difficulty and complexity of the work, reduces the technical level requirements of personnel, but also shortens the development cycle and reduces labor and time costs.

Description

Technical Field

[0001] The invention relates to the field of machine vision detection technology, and in particular to a universal machine vision detection method. Background Art

[0002] In the automation industry, machine vision projects have cyclical characteristics. Partners, inspection targets, inspection requirements, etc. often change over time. In view of this feature, current machine vision inspection companies or teams usually adopt the following model when doing projects:

[0003] (1) Algorithm personnel use commercial algorithm libraries to design and debug algorithms for projects and different application scenarios. In this model, both the design and debugging of algorithms require the participation of algorithm personnel, that is, algorithm personnel act as the project changes.

[0004] (2) Algorithm developers use open source algorithm libraries or develop algorithms completely from the bottom up, which is similar to (1), except that they do not need to use paid commercial algorithm libraries;

[0005] (3) Based on the platform software, some algorithm libraries are developed for calling, such as the publicly available FSLighting. The modules in this algorithm library are all designed to meet the application needs of the project.

[0006] The above model has the following disadvantages:

[0007] (1) Algorithm developers use commercial libraries to develop algorithms for different application scenarios, and the algorithms they develop cannot be applied to other projects and application scenarios. Commercial libraries require payment, which increases costs.

[0008] (2) Algorithm developers use open source algorithm libraries or develop algorithms completely from the bottom up, getting rid of the limitations of commercial libraries, but increasing the difficulty of development, which leads to higher technical requirements for algorithm developers, and increases the development cycle, labor and time costs;

[0009] (3) Currently, the self-developed algorithm libraries based on platform software known in the market are all designed based on the application-level requirements of the project. The application-level requirement descriptions are often disconnected from the algorithm theory logic, making them difficult to be universal and often producing poor results. Summary of the invention

[0010] The technical problem to be solved by the present invention is to provide a universal machine vision detection method, which solves the problem that in machine vision detection projects, algorithm personnel are required to design specific detection algorithms for different projects and different application scenarios, and the designed algorithms are difficult to reuse in other projects and application scenarios, thus resulting in a waste of manpower and time, and if a commercial algorithm library is used, it will increase costs.

[0011] The technical solution adopted by the present invention to solve the technical problem is: a general machine vision detection method, comprising the following steps:

[0012] S1. Annotate the defects of the image to be inspected to form a data set containing images and corresponding annotations;

[0013] S2. Send the image and annotation datasets to the deep learning module, randomly divide all the data into training sets and validation sets according to a certain ratio, train all the models separately or sequentially, and after the training is completed, select the model according to the intelligent recommendation results of the intelligent recommendation mechanism based on the validation set score;

[0014] S3, sending the image and the annotated data set to the detection area extraction module for global binarization, and arranging the output results of all global binarization algorithms in descending order of scores according to the intelligent recommendation mechanism; selecting the optimal extraction result as the rough extraction result of the detection area;

[0015] S4, the image and the annotated data set and the detection area extraction result of the previous step are sent to the detection area correction module, and the detection area extracted in the previous step is subjected to the region of interest roi operation on the image to obtain the roi image; the boundaries of the roi image are extracted respectively, and the segmented area is obtained by the boundary extraction-based method, and the segmented area is brought into the intelligent recommendation mechanism to obtain the score of the image in the boundary detection algorithm, and the detection area extracted by the optimal result is selected as the detection area fine extraction result;

[0016] S5, sending the image and annotation data as well as the result of the previous step to the detection area positioning module, manually inputting the shape of the detection area, and selecting the optimal result as the new detection area extraction result according to the intelligent recommendation mechanism;

[0017] S6. Send the image and the annotation data as well as the result obtained in the previous step to the defect detection module for local binarization, bring the detected area in according to the intelligent recommendation mechanism, and select the best result as the preliminary defect detection result by manual judgment;

[0018] S7, sending the image and the annotation data and the result obtained in the previous step to the defect detection module, and selecting the corresponding feature combination by manual judgment according to the detection result of the previous step, and obtaining the feature adjustment threshold for each feature extraction algorithm;

[0019] S8. Take the intersection and union of the result obtained in the previous step and the model obtained in step S2 respectively, and select the best combination according to the intelligent recommendation method and finally combined with manual judgment; use this combination to actually detect the image of the object to be tested, perform data synthesis on the detection results, and the result obtained by data synthesis is the final defect detection result.

[0020] Furthermore, in step S1 of the present invention, the labeling includes the detection area labeling and defect labeling of the area to be detected. In step S2, the scores of each model on the validation set are arranged in descending order according to the scores, and the top N models are selected, the parameters are manually adjusted, and the training is repeated; the top N' models are selected according to the above-mentioned recommendation mechanism as the deep learning pre-selected models; wherein N and N' are set values. In step S3, the score of the intelligent recommendation mechanism is equal to the average value of the detection intersection and union ratio of all images; wherein the labeling is the detection area labeling. In step S4, the labeling is the area labeling. In step S5, if the detection area is not a regular common shape, this step is not performed. In step S6, the labeling is the defect labeling. In step S7, the defect screening algorithm is combined by manually selecting feature extraction algorithms and the intersection or union relationship between them.

[0021] The present invention summarizes and classifies traditional algorithms and deep learning algorithms commonly used in machine vision projects into various modules, uses these modules as modules of the detection system, and adds a recommendation mechanism. Manually select modules according to experimental results for different application scenarios to form a detection system that meets the needs. The entire module uses open source libraries or is self-developed and implemented, and no longer uses commercial libraries.

[0022] The beneficial effect of the present invention is that it solves the defects existing in the background technology. After using the present invention, only the module selection stage of the system requires manual operation, and it is only responsible for the module parameter adjustment work, which reduces the difficulty and complexity of the work, thereby reducing the technical level requirements of the personnel, shortening the development cycle, and ultimately reducing labor and time costs. In addition, the development of all modules does not use commercial libraries, which further reduces costs compared to the development model using commercial libraries. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flow chart of the detection algorithm of the present invention. DETAILED DESCRIPTION

[0024] The present invention will now be described in further detail in conjunction with the accompanying drawings and preferred embodiments. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0025] like Figure 1 A general machine vision detection method is shown, which combines traditional algorithms with deep learning, builds a general defect detection system with modules (tool sets), divides the modules (tool sets) and corresponds them to the algorithm library, uses label data to score the intelligent recommendation mechanism, and finally obtains the optimal combination between the traditional algorithms and the combination with deep learning; specifically, the following steps are included:

[0026] 1. Data annotation

[0027] Defects are annotated on the images to be inspected to form a data set containing images and corresponding annotations, where the annotations include:

[0028] Inspection area marking, i.e. the area that needs to be inspected;

[0029] Defect annotation;

[0030] 2. Pre-selection of deep learning models

[0031] Select classic semantic segmentation models, such as FCN, UNet, UNet++, etc., to form a model library, which can be continuously updated with the release of new academic results. Each model in the model library, combined with a data set containing images and annotations, constitutes a deep learning module. The data set in the deep learning module is used to train each model in the model library to enable it to have defect detection capabilities;

[0032] The image and annotation data set are sent to the deep learning module. The deep learning module essentially corresponds to various models in various model libraries. All data are randomly divided into training sets and validation sets according to a certain ratio. In the present invention, the division ratio is 5:1. All models are trained separately or sequentially (when computing or storage resources are insufficient). After the training is completed, the model is selected according to the validation set score and the intelligent recommendation result. The mechanism of intelligent recommendation is:

[0033]

[0034] in:

[0035] n connection is the number of all connected domains in the detection results,

[0036] n label To mark the number of connected domains,

[0037] n true is the number of connected domains of defects detected correctly;

[0038]

[0039] in:

[0040] n is the number of images,

[0041] io i is the detection intersection-of-union ratio of the i-th image, i.e., the iou (intersection of union) value,

[0042] c label is the marked pixel part, c detectis the detected and annotated part, that is, the ratio of the intersection area of ​​the detected and annotated parts to the union area of ​​the two;

[0043] score = λ1score connection +λ2score miou

[0044] in:

[0045]

[0046] λ2=1-λ1

[0047] The scores of each model on the validation set are obtained according to the above formula, and the models are sorted in descending order according to the scores. The top 10 models are selected, and the parameters are manually adjusted and trained again. The top 5 models are selected according to the above recommendation mechanism as the pre-selected models for deep learning.

[0048] 3. Rough extraction of detection area

[0049] By consulting papers, cooperating with scientific research institutions, or developing on your own, you can collect various global grayscale binarization algorithms, implement them through programming, and form a global binarization algorithm library, which can be continuously updated with the release of new academic results;

[0050] When collecting algorithms, the selection criteria for the algorithms were: they could perform binary segmentation on the entire image and distinguish the foreground from the background;

[0051] The image and annotation dataset are sent to the detection area extraction module. The detection area extraction module is actually a global binarization algorithm library, which contains various global binarization algorithms. The output results of all global binarization algorithms are arranged in descending order according to the intelligent recommendation mechanism.

[0052] Smart recommendation scoring method:

[0053] score=score miou

[0054] score miou The method of finding is the same as that in step 2. The difference is that step 2 uses defect annotation, while here we use inspection area annotation. The top 10 algorithms are selected according to the scores, and then the parameters are manually adjusted to select the top 3 algorithms. Their extraction results are used as the rough extraction results of the inspection area.

[0055] 4. Fine extraction of the test area

[0056] By consulting papers, cooperating with scientific research institutions, or developing on your own, you can collect various boundary extraction algorithms, implement them through programming, and form a boundary extraction algorithm library, which can be continuously updated with the release of new academic results;

[0057] When collecting algorithms, the selection criteria for the algorithms are: the ability to detect the boundary part of the object in the image;

[0058] The image and the annotation data set as well as the rough extraction result of the detection area in the previous step are sent to the detection area correction module. The detection area correction module is actually a boundary extraction algorithm library, which contains various boundary extraction algorithms. The three groups of detection areas extracted in the previous step are respectively subjected to roi operation on the image, and the roi is expanded by n_roi pixels. n_roi can be set as a parameter in the system, and the default value is 5. Thus, three groups of roi images are obtained.

[0059] For the three groups of roi images, all algorithms in the boundary extraction algorithm library are used to extract boundaries. The segmented regions are obtained by the boundary extraction-based method. The segmented regions are introduced in accordance with the intelligent recommendation method in step 3. The region annotation is still used to obtain the scores of the three groups of images in each boundary detection algorithm. The top three algorithms are selected for each group, and then the parameters are manually adjusted to obtain better results. The top three algorithm combinations among the better results are selected, and the extracted detection regions are used as the detection region fine extraction results.

[0060] 5. Detection area fitting

[0061] By consulting papers, cooperating with scientific research institutions, or developing on your own, you can collect various shape fitting algorithms, including circle fitting, ellipse fitting, rectangle fitting, regular polygon fitting, straight line fitting, and curve fitting, and implement them through programming to form a shape fitting algorithm library, which can be continuously updated with the release of new academic achievements;

[0062] When collecting algorithms, the selection criteria of the algorithms are: they can fit a polygon based on the set of point position coordinates;

[0063] If the detection area is not a regular common shape, such as a matrix, circle, ellipse, regular polygon, etc., ignore this step;

[0064] If the detection area is a regular common shape, the image and annotation data as well as the result of the previous step are sent to the detection area positioning module. The detection area positioning module is actually a shape fitting algorithm library. The shape of the detection area is manually input, and the top 3 algorithm combinations are obtained according to step 3 as the new detection area extraction result.

[0065] 6. Defect Detection

[0066] By consulting papers, cooperating with scientific research institutions, or developing on your own, you can collect various local grayscale binarization algorithms, implement them through programming, and form a local binarization algorithm library, which can be continuously updated with the release of new academic results;

[0067] When collecting algorithms, the selection criteria of the algorithms are: the local foreground part of the image can be segmented, that is, the pixels with obvious grayscale differences from the surrounding within a certain range, and the size of the local range can be adjusted, such as the local rectangular area - the length and width can be adjusted, and the local circular area - the radius can be adjusted;

[0068] The image and annotation data as well as the results obtained in the previous step are sent to the defect detection module. The defect detection module is actually a local binarization algorithm library. Following the intelligent recommendation method in step 3, the detected area is brought in and the defect annotation area is annotated to obtain the top 10 algorithm combinations.

[0069] Then, manual parameter optimization is performed. According to the detection results, the best algorithm combination is selected from these 10 combinations by manual judgment, and its detection result is used as the preliminary defect detection result;

[0070] 7. Defect screening

[0071] By consulting papers, cooperating with scientific research institutions, or developing on your own, you can collect various grayscale and shape feature calculation algorithms, implement them through programming, and form a feature extraction algorithm library, which can be continuously updated with the release of new academic results;

[0072] When collecting algorithms, the selection criteria for algorithms are: the ability to extract grayscale features (such as Hu moments) and shape features (such as area and roundness), and the ability to describe and distinguish different grayscale distributions and shapes;

[0073] The image and annotation data as well as the results obtained in the previous step are sent to the defect detection module. The defect detection module is actually a feature extraction algorithm library, which contains various grayscale feature and shape feature algorithms. According to the detection results of the previous step, there is manual judgment to select the corresponding feature combination, that is, to select the feature extraction algorithm and the intersection or union relationship between them, and combine the defect screening algorithm. Then, the threshold of the features obtained by each feature extraction algorithm in the defect screening algorithm is adjusted to achieve the best effect. The optimal method can refer to the score given by the artificial intelligence recommendation method in step 3, and the final result is manual comprehensive judgment.

[0074] 8. Data synthesis

[0075] Take the intersection and union of the results obtained in the previous step and the 5 models obtained in step 2, and get 10 groups of results, of which 5 groups are obtained by intersection and 5 groups are obtained by union. Refer to the intelligent recommendation method in step 3, and finally combine manual judgment to select the best combination;

[0076] This combined method is used to actually detect the image of the object under test. In the end, there are traditional algorithm results obtained by combining various modules, deep learning results obtained by the deep learning network, and the best screening method (intersection, union) obtained by defect screening. The detection results are synthesized, and the result obtained by data synthesis is the final defect detection result.

[0077] Regarding the issue of increased costs due to using commercial libraries:

[0078] The original intention of the design of this invention is to break away from the constraints of the commercial library halcon, completely use open source libraries or self-implemented bottom-level implementation, and no longer rely on any commercial image processing library;

[0079] Regarding the issue of increased labor costs:

[0080] The present invention only requires operators to select functional modules based on their intuitive feelings and experience, and with the assistance of intelligent recommendation functions, the technical level requirements for operators are greatly reduced;

[0081] Regarding the issue of increasing time costs:

[0082] The present invention only requires manual intervention in the functional module selection stage. It only needs to qualitatively select the algorithm based on intuitive feelings and project experience, and adjust the parameters to further optimize the effect. In addition, the whole process is assisted by intelligent recommendation and intelligent parameter setting functions, which greatly improves efficiency.

[0083] The demand description at the application level is often out of touch with the algorithm logic, making it difficult to be universal and often having poor results:

[0084] The present invention divides functional modules according to algorithm logic, giving full play to the basic algorithm knowledge of users, so that people with low algorithm level can also combine high-quality algorithms through the various modules of the system, so that the system functions can be flexibly used in various application scenarios of various projects, thereby making the system universal and the effect greatly improved.

[0085] The above description only describes the specific implementation mode of the present invention. Various examples do not limit the essential content of the present invention. After reading the description, ordinary technicians in the relevant technical field can modify or deform the specific implementation modes described above without departing from the essence and scope of the invention.

Claims

1. A general machine vision detection method, characterized in that: The following steps are included: S1. Annotate the defects of the image to be inspected to form a data set containing images and corresponding annotations; S2. Send the image and annotation datasets to the deep learning module, randomly divide all the data into training sets and validation sets according to a certain ratio, train all the models separately or sequentially, and after the training is completed, select the model according to the intelligent recommendation results of the intelligent recommendation mechanism based on the validation set score; S3, sending the image and annotation data set to the detection area extraction module for global binarization, and arranging the output results of all global binarization algorithms in descending order of scores according to the intelligent recommendation mechanism; Select the best extraction result as the rough extraction result of the detection area; S4, sending the image and the annotation data set and the detection area extraction result of the previous step to the detection area correction module, performing the region of interest roi operation on the image of the detection area extracted in the previous step to obtain the roi image; Extract the boundaries of the roi image respectively, obtain the segmented area through the boundary extraction method, bring the segmented area into the intelligent recommendation mechanism, obtain the score of the image in the boundary detection algorithm, and select the detection area extracted by the optimal result as the detection area extraction result; S5, sending the image and annotation data as well as the result of the previous step to the detection area positioning module, manually inputting the shape of the detection area, and selecting the optimal result as the new detection area extraction result according to the intelligent recommendation mechanism; S6. Send the image and the annotation data as well as the result obtained in the previous step to the defect detection module for local binarization, bring the detected area in according to the intelligent recommendation mechanism, and select the best result as the preliminary defect detection result by manual judgment; S7, sending the image and the annotation data and the result obtained in the previous step to the defect detection module, and selecting the corresponding feature combination by manual judgment according to the detection result of the previous step, and obtaining the feature adjustment threshold for each feature extraction algorithm; S8. Take the intersection and union of the result obtained in the previous step and the model obtained in step S2 respectively, and select the best combination according to the intelligent recommendation method and finally combined with manual judgment; use this combination to actually detect the image of the object to be tested, perform data synthesis on the detection results, and the result obtained by data synthesis is the final defect detection result.

2. A general machine vision detection method as claimed in claim 1, characterized in that: In the step S1, the marking includes marking of the inspection area and defect marking of the area to be inspected.

3. A general machine vision detection method as claimed in claim 1, characterized in that: In the step S2, the scores of each model on the validation set are arranged in descending order, and the top N models are selected, and the parameters are manually adjusted and trained again; the top N' models are selected according to the above-mentioned recommendation mechanism as the pre-selected models for deep learning; wherein N and N' are set values.

4. A general machine vision detection method as claimed in claim 1, characterized in that: In step S3, the score of the intelligent recommendation mechanism is equal to the average value of the detection intersection-over-union ratios of all images; The markings are detection area markings.

5. A general machine vision detection method as claimed in claim 1, characterized in that: In the step S4, the marking is area marking.

6. A general machine vision detection method as claimed in claim 1, characterized in that: In the step S5, if the detection area is not a regular common shape, this step is not performed.

7. A general machine vision detection method as claimed in claim 1, characterized in that: In the step S6, the marking is a defect marking.

8. A general machine vision detection method as claimed in claim 1, characterized in that: In step S7, a defect screening algorithm is formed by manually selecting feature extraction algorithms and the intersection or union relationship between them.

Images