A visual inspection method, device, terminal and storage medium

Abstract

The present invention discloses a visual inspection method, device, terminal, and storage medium. The method includes: obtaining an image of an object to be inspected, recorded as a picture of the object to be inspected; based on the picture of the object to be inspected, using a pre-trained micro-convolutional deep learning model, determining whether a first feature to be detected exists in the object to be inspected; if the first feature to be detected exists in the object to be inspected, using a pre-modeled template library of target detection objects based on the picture of the object to be inspected, identifying whether a second feature to be detected exists in the object to be inspected; if the second feature to be detected exists in the object to be inspected, determining that the object to be inspected is a target detection object and that the object to be inspected has no assembly defects. This solution, by enabling a machine to perform visual defect inspection on the object to be inspected using the micro-convolutional deep learning model, can automatically model the template of the object to be inspected, significantly saving manual labor.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and specifically relates to a visual inspection method, device, terminal and storage medium, and more particularly to an implementation method, device, terminal and storage medium of an automatic modeling visual inspection system based on a self-learning algorithm. Background Art

[0002] When inspecting product quality defects on an object, manual inspection is possible, but the workload is prohibitive. Alternatively, machines can be used to perform visual defect inspection on the object. Specifically, during machine visual defect inspection, the object must be matched against a template for similarity. If the matching similarity does not meet a predetermined requirement, the object is deemed defective. However, the template must be manually modeled, which still requires considerable manual effort.

[0003] The above content is only used to assist in understanding the technical solution of the present invention and does not constitute an admission that the above content is prior art. Summary of the Invention

[0004] The purpose of the present invention is to provide a visual inspection method, device, terminal and storage medium to solve the problem that in the process of using a machine to perform visual defect inspection on the target to be inspected, it is necessary to match the similarity of the target to be inspected with a template, but the template of the target to be inspected is manually modeled, and there is still a problem of large amount of manual labor. By enabling the machine to use a micro-convolution deep learning model to perform visual defect inspection on the target to be inspected, it is possible to realize automatic modeling of the template of the target to be inspected, thereby greatly saving the amount of manual labor.

[0005] The present invention provides a visual inspection method, which is applied to perform visual defect detection on an object to be inspected; the visual inspection method includes: obtaining an image of the object to be inspected, recorded as a picture of the object to be inspected; based on the picture of the object to be inspected, using a pre-trained convolutional deep learning model, determining whether a first feature to be detected exists in the object to be inspected; if it is determined that the first feature to be detected exists in the object to be inspected, based on the picture of the object to be inspected, using a pre-modeled template library of target detection objects, identifying whether a second feature to be detected exists in the object to be inspected; when it is determined that the second feature to be detected also exists in the object to be inspected, determining that the object to be inspected is a target detection object, and determining that the object to be inspected has no assembly defects.

[0006] In some embodiments, based on the picture of the object to be detected, a pre-trained convolutional deep learning model is used to determine whether a first feature to be detected exists in the object to be detected, including: performing a window search on the picture of the object to be detected with a window size of a first set pixel to obtain a first set number of window pictures; using the convolutional deep learning model, determining whether a reference picture of the first feature to be detected exists in each window picture of the first set number of window pictures; if it is determined that a reference picture of the first feature to be detected exists in at least one window picture of the first set number of window pictures, then it is determined that the first feature to be detected exists in the object to be detected.

[0007] In some embodiments, the convolutional deep learning model is used to determine whether there is a reference image of the first feature to be detected in each window image of the first set number of window images, including: performing feature extraction on each window image in the first set number of window images through the convolutional layer in the convolutional deep learning model to obtain a feature map corresponding to each window image; obtaining a first set number of feature maps for the first set number of window images; performing dimensionality reduction sampling on each feature map in the first set number of feature maps through the pooling layer in the convolutional deep learning model to obtain a reduced dimensionality feature map corresponding to each feature map; obtaining a first set number of reduced dimensionality feature maps for the first set number of feature maps; classifying each reduced dimensionality feature map in the first set number of reduced dimensionality feature maps through the classification layer in the convolutional deep learning model to determine the corresponding category of each reduced dimensionality feature map in the reference category range; if the corresponding category of at least one of the reduced dimensionality feature maps in the first set number of reduced dimensionality feature maps belongs to the category of the reference image of the first feature to be detected, it is determined that there is a reference image of the first feature to be detected in at least one window image in the first set number of window images.

[0008] In some embodiments, based on the picture of the object to be detected, a template library of the target detection object obtained by pre-modeling is used to identify whether a second feature to be detected exists in the object to be detected, including: expanding the area after a second set pixel from the position of the picture of the object to be detected where the picture of the first feature to be detected is located in the picture of the object to be detected, as the picture of the second feature to be detected; comparing the picture of the second feature to be detected with the reference picture in the template library of the second feature to be detected in the target detection object to obtain a comparison result; the comparison result is a similarity value between the picture of the second feature to be detected and at least one reference picture in the template library of the second feature to be detected; determining whether the similarity value between the picture of the second feature to be detected and at least one reference picture in the template library of the second feature to be detected is greater than a first set similarity threshold; if the similarity value between the picture of the second feature to be detected and at least one reference picture in the template library of the second feature to be detected is greater than the first set similarity threshold, it is determined that the second feature to be detected exists in the object to be detected.

[0009] In some embodiments, the picture of the second feature to be detected is compared with the reference picture in the template library of the second feature to be detected in the target detection object to obtain a comparison result, including: performing edge detection on the picture of the second feature to be detected to obtain a gradient grayscale image of the picture of the second feature to be detected; and performing edge detection on each reference picture in the template library of the second feature to be detected in the target detection object to obtain a gradient grayscale image of each reference picture of the second feature to be detected; performing Euclidean distance calculation on the gradient grayscale image of the picture of the second feature to be detected and the gradient grayscale image of each reference picture of the second feature to be detected to obtain a corresponding second set number of similarity values; comparing each similarity value in the second set number of similarity values ​​with a first set similarity threshold to determine that the second feature to be detected exists in the object to be detected when at least one of the similarity values ​​in the second set number of similarity values ​​is greater than the first set similarity threshold.

[0010] In some embodiments, based on the picture of the object to be detected, using the template library of the target detection object obtained by pre-modeling, identifying whether the second feature to be detected exists in the object to be detected, further comprising: if the similarity value between the picture of the second feature to be detected and at least one reference picture in the template library of the second feature to be detected is less than or equal to a first set similarity threshold, then determining whether the similarity value between the picture of the second feature to be detected and at least one reference picture in the template library of the second feature to be detected is greater than a second set similarity threshold; the first set similarity threshold is greater than the second set similarity threshold; if the similarity value between the picture of the second feature to be detected and at least one reference picture in the template library of the second feature to be detected is greater than a second set similarity threshold At the second set similarity threshold, it is determined that the second feature to be detected in the object to be detected is a new second feature to be detected, and the picture of the new second feature to be detected is stored in the template library of the second feature to be detected as a reference picture of the new second feature to be detected in the template library of the second feature to be detected; and it is determined that the object to be detected containing the first object to be detected and the new second feature to be detected is a target detection object, and it is determined that the object to be detected has no assembly defects; if the similarity value between the picture of the second feature to be detected and at least one reference picture in the template library of the second feature to be detected is less than or equal to the second set similarity threshold, it is determined that the object to be detected is not the target detection object.

[0011] In some embodiments, the method further includes: if it is determined that the first feature to be detected does not exist in the object to be detected, determining whether the second feature to be detected exists in the object to be detected based on a picture of the object to be detected and using a pre-trained micro-convolutional deep learning model; when it is determined that the second feature to be detected exists in the object to be detected, determining that the object to be detected is a target detection object and determining that the object to be detected has an assembly defect.

[0012] Matching the above method, the present invention provides a visual inspection device on the other hand, which is used to perform visual defect detection on an object to be inspected; the visual inspection device includes: an acquisition unit, configured to acquire an image of the object to be inspected, recorded as a picture of the object to be inspected; a control unit, configured to determine whether a first feature to be detected exists in the object to be inspected based on the picture of the object to be inspected and using a pre-trained convolutional deep learning model; the control unit is also configured to, if it is determined that the first feature to be detected exists in the object to be inspected, identify whether a second feature to be detected exists in the object to be inspected based on the picture of the object to be inspected and using a template library of target detection objects obtained by pre-modeling; the control unit is also configured to determine that the object to be inspected is a target detection object and that the object to be inspected has no assembly defects when it is determined that the second feature to be detected also exists in the object to be inspected.

[0013] In some embodiments, the control unit, based on the picture of the object to be detected, uses a pre-trained convolutional deep learning model to determine whether the first feature to be detected exists in the object to be detected, including: performing a window search on the picture of the object to be detected with a window size of a first set pixel to obtain a first set number of window pictures; using the convolutional deep learning model to determine whether there is a reference picture of the first feature to be detected in each window picture of the first set number of window pictures; if it is determined that there is a reference picture of the first feature to be detected in at least one window picture among the first set number of window pictures, it is determined that the first feature to be detected exists in the object to be detected.

[0014] In some embodiments, the control unit uses the convolutional deep learning model to determine whether there is a reference image of the first feature to be detected in each window image of the first set number of window images, including: performing feature extraction on each window image in the first set number of window images through the convolutional layer in the convolutional deep learning model to obtain a feature map corresponding to each window image; obtaining a first set number of feature maps for the first set number of window images; performing dimensionality reduction sampling on each feature map in the first set number of feature maps through the pooling layer in the convolutional deep learning model to obtain a reduced dimensionality feature map corresponding to each feature map; obtaining a first set number of reduced dimensionality feature maps for the first set number of feature maps; classifying each reduced dimensionality feature map in the first set number of reduced dimensionality feature maps through the classification layer in the convolutional deep learning model to determine the corresponding category of each reduced dimensionality feature map in the reference category range; if the corresponding category of at least one of the reduced dimensionality feature maps in the first set number of reduced dimensionality feature maps belongs to the category of the reference image of the first feature to be detected, it is determined that there is a reference image of the first feature to be detected in at least one window image in the first set number of window images.

[0015] In some embodiments, the control unit, based on the picture of the object to be detected, uses a template library of the target detection object obtained by pre-modeling to identify whether a second feature to be detected exists in the object to be detected, including: expanding the area after a second set pixel from the position of the picture of the object to be detected where the picture of the first feature to be detected is located in the picture of the object to be detected, as the picture of the second feature to be detected; comparing the picture of the second feature to be detected with the reference picture in the template library of the second feature to be detected in the target detection object to obtain a comparison result; the comparison result is a similarity value between the picture of the second feature to be detected and at least one reference picture in the template library of the second feature to be detected; determining whether the similarity value between the picture of the second feature to be detected and at least one reference picture in the template library of the second feature to be detected is greater than a first set similarity threshold; if the similarity value between the picture of the second feature to be detected and at least one reference picture in the template library of the second feature to be detected is greater than the first set similarity threshold, it is determined that the second feature to be detected exists in the object to be detected.

[0016] In some embodiments, the control unit compares the picture of the second feature to be detected with the reference picture in the template library of the second feature to be detected in the target detection object to obtain a comparison result, including: performing edge detection on the picture of the second feature to be detected to obtain a gradient grayscale image of the picture of the second feature to be detected; and performing edge detection on each reference picture in the template library of the second feature to be detected in the target detection object to obtain a gradient grayscale image of each reference picture of the second feature to be detected; performing Euclidean distance calculation on the gradient grayscale image of the picture of the second feature to be detected and the gradient grayscale image of each reference picture of the second feature to be detected to obtain a corresponding second set number of similarity values; comparing each similarity value in the second set number of similarity values ​​with a first set similarity threshold to determine that the second feature to be detected exists in the object to be detected if at least one of the similarity values ​​in the second set number of similarity values ​​is greater than the first set similarity threshold.

[0017] In some embodiments, the control unit, based on the picture of the object to be detected, uses the template library of the target detection object obtained by pre-modeling to identify whether the second feature to be detected exists in the object to be detected, and also includes: if the similarity value between the picture of the second feature to be detected and at least one reference picture in the template library of the second feature to be detected is less than or equal to a first set similarity threshold, then determining whether the similarity value between the picture of the second feature to be detected and at least one reference picture in the template library of the second feature to be detected is greater than a second set similarity threshold; the first set similarity threshold is greater than the second set similarity threshold; if the similarity between the picture of the second feature to be detected and at least one reference picture in the template library of the second feature to be detected is less than or equal to a first set similarity threshold If the similarity value is greater than the second set similarity threshold, it is determined that the second feature to be detected in the object to be detected is a new second feature to be detected, and the picture of the new second feature to be detected is stored in the template library of the second feature to be detected as a reference picture of the new second feature to be detected in the template library of the second feature to be detected; and it is determined that the object to be detected containing the first object to be detected and the new second feature to be detected is a target detection object, and it is determined that the object to be detected has no assembly defects; if the similarity value between the picture of the second feature to be detected and at least one reference picture in the template library of the second feature to be detected is less than or equal to the second set similarity threshold, it is determined that the object to be detected is not the target detection object.

[0018] In some embodiments, it also includes: the control unit is further configured to, if it is determined that the first feature to be detected does not exist in the object to be detected, determine whether the second feature to be detected exists in the object to be detected based on the image of the object to be detected and using a pre-trained micro-convolutional deep learning model; the control unit is further configured to, when it is determined that the second feature to be detected exists in the object to be detected, determine that the object to be detected is a target detection object and determine that the object to be detected has an assembly defect.

[0019] Matching the above-mentioned device, the present invention further provides a terminal on the other hand, including: the above-mentioned visual detection device.

[0020] In accordance with the above method, the present invention further provides a storage medium, which includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the above-mentioned visual inspection method.

[0021] Therefore, the solution of the present invention obtains a picture of the object to be detected by photographing the object to be detected entering the detection area; uses a micro-convolution deep learning model to identify a first feature to be detected in the picture of the object to be detected; when the first feature to be detected is identified in the picture of the object to be detected, the template image of the object to be detected pre-stored in the template library is used to identify the second feature to be detected in the picture of the object to be detected that contains the first feature to be detected; when the second feature to be detected is also identified in the picture of the object to be detected, it is considered that the object to be detected is a target detection object; when the second feature to be detected is not identified in the picture of the object to be detected, it is considered that the object to be detected is a new detection object, and the picture of the object to be detected is stored in the template library to realize automatic modeling of the template image of the new detection object; thus, by enabling the machine to perform visual defect inspection on the target to be detected using the micro-convolution deep learning model, automatic modeling of the template of the target to be detected can be realized, which greatly saves manual labor.

[0022] Other features and advantages of the present invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the present invention.

[0023] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 1 is a flow chart of an embodiment of a visual inspection method of the present invention;

[0025] Figure 21. A flow chart of an embodiment of the method of the present invention for determining whether a first feature to be detected exists in the object to be detected;

[0026] Figure 3 A schematic diagram of a flow chart of an embodiment of the method of the present invention for determining whether a reference image having the first feature to be detected exists in each window image of the first set number of window images;

[0027] Figure 4 1. A flow chart of an embodiment of the method of the present invention for identifying whether a second feature to be detected exists in the object to be detected;

[0028] Figure 5 A schematic diagram of a flow chart of an embodiment of the method of the present invention for comparing a reference image in a template library of a second feature to be detected in the target detection object;

[0029] Figure 6 Schematic diagram of a flow chart of an embodiment of automatically modeling a new second feature to be detected in the method of the present invention;

[0030] Figure 7 A flow chart of an embodiment of the method of the present invention for determining that the object to be inspected is the target inspection object and that the object to be inspected has an assembly defect;

[0031] Figure 8 Schematic diagram of the structure of a visual inspection device according to an embodiment of the present invention;

[0032] Figure 9 This is a schematic diagram of the inspection process of an automatic modeling visual inspection system based on a self-learning algorithm;

[0033] Figure 10 This is a schematic diagram of the interface for establishing a wind blade template diagram;

[0034] Figure 11 Schematic diagram of templates for multiple types of fan blades, where (a)-(r) are schematic diagrams of templates for different types of fan blades;

[0035] Figure 12 Schematic diagram of the interface for detection and automatic modeling of multiple types of fan blades;

[0036] Figure 13 Schematic diagram of the interface for filtering non-detection targets, where (a) is a schematic diagram of the interface for non-detection targets, and (b) is a schematic diagram of the interface for detection targets;

[0037] Figure 14 Schematic diagram of the interface for manual error correction assisted learning.

[0038] In conjunction with the accompanying drawings, the reference numerals in the embodiments of the present invention are as follows:

[0039] 102 - acquisition unit; 104 - control unit. DETAILED DESCRIPTION

[0040] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments of the present invention and corresponding drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0041] In the related scheme, in the process of using a machine to perform visual defect detection on the target to be detected, the target to be detected and the template are matched for similarity, which can specifically include: calculating the Euclidean distance between each pixel in the image of the target to be detected and each corresponding pixel in the template image, and using this Euclidean distance as the similarity. If the similarity is greater than a set threshold, it is considered that the image of the target to be detected and the template image are matched.

[0042] Among them, the inspected targets may differ by product model, product type, etc. Therefore, during manual inspection, inspectors can easily distinguish the product model and product type of different inspected targets, as well as whether there are defects, through visual observation. However, when using machines to perform visual defect inspection on the inspected targets, there will be some problems when the machine analyzes the acquired images. For example, manual template creation is required, and the corresponding inspected targets need to be manually classified according to product model, product type, etc. Whenever a new type of inspected target is added, if the machine's current template library does not have a corresponding template for this type of inspected target, the machine cannot use it to perform visual defect inspection on this type of inspected target.

[0043] According to an embodiment of the present invention, a visual detection method is provided, such as Figure 1A flow chart of an embodiment of the method of the present invention is shown. The visual inspection method is applied to perform visual defect detection on an object to be inspected, specifically to detect whether the object to be inspected is a target inspection object, and to detect whether the object to be inspected has assembly defects when the object to be inspected is the target inspection object and has no assembly defects; when the object to be inspected is the target inspection object, the object to be inspected includes: a first feature to be inspected and a second feature to be inspected, the first feature to be inspected being such as a nut, and the second feature to be inspected being such as a fan blade; if the second feature to be inspected in the object to be inspected is equipped with the first feature to be inspected, it is determined that the object to be inspected has no assembly defects; if the second feature to be inspected in the object to be inspected is not equipped with the first feature to be inspected, it is determined that the object to be inspected has assembly defects. The visual inspection method includes: steps S110 to S140.

[0044] At step S110, an image of the object to be detected is obtained, recorded as the picture of the object to be detected. The object to be detected is a product that has entered a preset detection area. For example, a determination is made as to whether a product has entered the preset detection area. If a product has entered the detection area, the product is taken as the object to be detected, and a photo of the object to be detected is taken to obtain an image of the object to be detected, recorded as the picture of the object to be detected.

[0045] At step S120, based on the image of the object to be detected, a pre-trained convolutional deep learning model is used to determine whether the first feature to be detected exists in the object to be detected.

[0046] In some embodiments, the convolutional deep learning model is pre-loaded with a reference image of the first feature to be detected. Based on the image of the object to be detected, using the pre-trained convolutional deep learning model, determining whether the first feature to be detected exists in the object to be detected includes:

[0047] The following combination Figure 2 The flowchart of an embodiment of determining whether the first feature to be detected exists in the object to be detected in the method of the present invention is shown, which further illustrates the specific process of determining whether the first feature to be detected exists in the object to be detected in step S120, including: steps S210 to S230.

[0048] Step S210 , performing a window search on the image of the object to be detected with a window size of a first set pixel to obtain a first set number of window images.

[0049] Step S220: Using the convolutional deep learning model, determine whether there is a reference image of the first feature to be detected in each window image of the first set number of window images.

[0050] In some embodiments, the convolutional deep learning model includes a convolutional layer, a pooling layer, and a classification layer. The specific process of using the convolutional deep learning model to determine whether a reference image for the first feature to be detected exists in each of the first set number of window images in step S220 is described in the following exemplary embodiment.

[0051] The following combination Figure 3 The flowchart of an embodiment of determining whether a reference image of the first feature to be detected exists in each window image of the first set number of window images in the method of the present invention is shown, further illustrating the specific process of determining whether a reference image of the first feature to be detected exists in each window image of the first set number of window images in step S220, including: steps S310 to S340.

[0052] Step S310: Perform feature extraction on each of the first set number of window images using the convolution layer in the convolutional deep learning model to obtain a feature map corresponding to each of the window images. For the first set number of window images, a first set number of feature maps are obtained.

[0053] Step S320: Perform dimensionality reduction sampling on each of the first set number of feature maps through the pooling layer in the convolutional deep learning model to obtain a reduced-dimensional feature map corresponding to each of the feature maps. For the first set number of feature maps, a first set number of reduced-dimensional feature maps are obtained.

[0054] Step S330: Classify each of the first set number of reduced dimensionality feature maps through the classification layer in the convolutional deep learning model to determine the corresponding category of each of the reduced dimensionality feature maps in the reference category range.

[0055] Step S340: If the corresponding category of at least one of the first set number of reduced dimensionality feature maps belongs to the category of the reference image of the first feature to be detected, it is determined that there is a reference image of the first feature to be detected in at least one window image among the first set number of window images.

[0056] The structure of the micro-convolutional deep learning model is exemplified below.

[0057] In some examples, the micro-convolutional deep learning model has a seven-layer structure: 30*9*9 convolutional layer -> 30*9*9 pooling layer -> 90*13*13 convolutional layer -> 90*13*13 pooling layer -> 60*2*2 convolutional layer -> 60*2*2 pooling layer -> classification layer. The ">" indicates the direction of transmission: the image is transmitted from the convolutional layer to the pooling layer, and then to the classification layer.

[0058] The convolutional layer extracts features from the input variables, producing a feature map. In a 30x9x9 convolutional layer, 9x9 represents sliding a 9x9 weighted window across the input image. Each weight within the window is multiplied by the corresponding image grayscale value, and the sum is calculated as the output. After sliding the window across the entire input image, a feature map is obtained. 30 represents 30 windows with different weights, resulting in 30 different feature maps.

[0059] The pooling layer downsamples the feature map obtained by the previous convolutional layer to produce a reduced-dimensional feature map. In a 30x9x9 pooling layer, 9x9 represents sliding a 9x9 window across the input feature map, taking the maximum grayscale value in the window as the output, and producing a downsampled feature map. 30 represents downsampling all 30 feature maps in the previous layer.

[0060] The function of the classification layer is to classify the input dimensionality reduction feature map. For example, the classification layer generally uses softmax for logistic regression for classification.

[0061] The following is an example of how to use the micro-convolutional deep learning model.

[0062] In some examples, the mini-convolutional deep learning model is first trained using two corresponding types of sample data, ultimately resulting in a mini-convolutional deep learning model that can distinguish between these two types of sample data. The two corresponding types of sample data can be sample data of several images of nuts and sample data of several images of non-nuts. Once the mini-convolutional deep learning model is trained, the image to be tested is input into the trained mini-convolutional deep learning model, which then determines whether the image is a nut or not.

[0063] The processing details within this micro-convolutional deep learning model can be as follows: After the image input undergoes convolution processing in the convolution layer and pooling processing in the pooling layer, a reduced-dimensional feature map is generated, which is recorded as the reduced-dimensional feature map. The reduced-dimensional feature map is then sent to the classification layer for classification, such as using a softmax classifier in the classification layer to determine the category to which the input image belongs. Determining the category of the input image allows the image to be judged to belong to the category, that is, whether it is a nut or not.

[0064] Step S230: If it is determined that at least one window image among the first set number of window images contains a reference image of the first feature to be detected, it is determined that the first feature to be detected exists in the object to be detected.

[0065] The following takes the defect detection of whether the fan blades of the fan in the air conditioner outdoor unit have been nutted as an example to exemplify the specific implementation process of the solution of the present invention.

[0066] Taking the defect detection of whether the fan blades have been nutted as an example, the solution of the present invention provides an online visual inspection system based on automatic modeling and autonomous learning mechanism. Figure 9 The following is a schematic diagram of the inspection process of an automatic modeling visual inspection system based on a self-learning algorithm. Figure 9 As shown, in the solution of the present invention, the detection process of the automatic modeling visual inspection system based on the self-learning algorithm includes:

[0067] Step S10: When the device is turned on, the machine (such as the first air-conditioning outdoor unit) enters the detection area, and the fan blades of the first air-conditioning outdoor unit pass through the device, and then step S20 is executed.

[0068] The device may be a machine loaded with an automatic modeling visual inspection system based on a self-learning algorithm.

[0069] Step S20: When the fan blades of the first air conditioner outdoor unit pass by the device, the camera takes a picture and then executes step S30.

[0070] Step S30: Detect whether there is a nut in the image captured by the camera: if so, execute step S40. Otherwise, execute step S50.

[0071] In step S30, detecting whether there is a nut in the image captured by the camera may specifically include:

[0072] Step S301: The automatic modeling visual inspection system based on the self-learning algorithm performs a window search on the image. The window size is based on the size of the actual nut in the image, such as a window size of 60*60 pixels.

[0073] Window search uses a window of a specified size (e.g., 60*60 pixels) to slide across the entire image and capture a screenshot of the window size. The actual nut size in the image can be measured using your computer's built-in drawing software, using the pixel size of the nut in the image.

[0074] Step S302: Use the micro-convolutional deep learning model to determine whether the image in each window is a nut.

[0075] Among them, the process of using the micro-convolution deep learning model to judge is to input the picture of each window size into the micro-convolution deep learning model, and then the micro-convolution deep learning model performs feature extraction and classification on the picture of each window size to obtain the result of whether the picture is a nut.

[0076] In some examples, feature extraction and classification are performed on images of each window size to determine whether the image is a nut. Specifically, a miniature convolutional deep learning model is first trained using two corresponding types of sample data, ultimately obtaining a miniature convolutional deep learning model that can distinguish between the two corresponding types of sample data. The two corresponding types of sample data can be sample data of several nut images and sample data of several non-nut images. Once the miniature convolutional deep learning model is trained, the image to be tested is input into the trained miniature convolutional deep learning model, which then determines whether the image is a nut or not.

[0077] In step S130, if it is determined that the first feature to be detected exists in the object to be detected, based on the image of the object to be detected, a template library of target detection objects obtained by pre-modeling is used to identify whether the second feature to be detected exists in the object to be detected.

[0078] In some embodiments, the specific process of identifying whether the second feature to be detected exists in the object to be detected based on the image of the object to be detected and using the template library of the target detection object obtained by pre-modeling in step S130 is described in the following exemplary embodiments.

[0079] The following combination Figure 4 The flowchart of an embodiment of the method of the present invention for identifying whether the second feature to be detected exists in the object to be detected further illustrates the specific process of identifying whether the second feature to be detected exists in the object to be detected in step S130, including: steps S410 to S440.

[0080] Step S410 , expanding the area of ​​the image of the object to be detected where the image of the first feature to be detected is located to the periphery beyond the second set pixel, as the image of the second feature to be detected.

[0081] Step S420: Compare the image of the second feature to be detected with reference images in a template library of the second feature to be detected in the target detection object to obtain a comparison result. The comparison result is a similarity value between the image of the second feature to be detected and at least one reference image in the template library of the second feature to be detected.

[0082] Step S430 , determining whether a similarity value between the picture of the second feature to be detected and at least one reference picture in the template library of the second feature to be detected is greater than a first set similarity threshold.

[0083] Step S440: If the similarity value between the picture of the second feature to be detected and at least one reference picture in the template library of the second feature to be detected is greater than a first set similarity threshold, it is determined that the second feature to be detected exists in the object to be detected.

[0084] like Figure 9 As shown, in the solution of the present invention, the detection process of the automatic modeling visual inspection system based on the self-learning algorithm also includes:

[0085] Step S40: If there is a nut in the picture taken by the camera, the position of the nut in the picture taken by the camera is expanded outward by a certain range as the fan blade image to be inspected, and then step S41 is executed.

[0086] Specifically, in step S40, if there is a nut in the picture taken by the camera, the position of the nut is expanded outward by 200 pixels. The pixel value is the size of the wind blade in the picture taken by the camera, and then the expanded area is used as the wind blade image to be inspected.

[0087] The size of the fan blades in the picture taken by the camera can be obtained by measuring the size of the pixels occupied by the fan blades in the picture using a drawing tool provided on the computer.

[0088] Step S41: Compare the blade image to be inspected with the blade template images in the blade template image library to obtain a first comparison result, and then execute step S42. The first comparison result may be a similarity value between the blade image to be inspected and at least one blade template image in the blade template image library.

[0089] Step S42: Based on the first comparison result, determine whether the similarity between the blade image to be inspected and at least one blade template image in the blade template image library is greater than a set similarity threshold: if so, execute step S43. Otherwise, execute step S44.

[0090] Specifically, in step S42, if there is a blade template image in the blade template image library, after comparing the blade image to be inspected with all blade template images in the blade template image library using the edge matching method to obtain a first comparison result, it is determined whether the similarity value between the blade image to be inspected and at least one blade template image in the blade template image library in the comparison result is greater than a normal threshold (e.g., a set similarity threshold). If so, step S43 is executed. Otherwise, step S44 is executed.

[0091] Step S43: If the similarity between the fan blade image to be inspected and at least one fan blade template image in the fan blade template library is greater than the set similarity threshold, it is determined that the machine (such as the first air conditioner outdoor unit) is the target machine, and it is determined that nuts have been installed on the fan blades of the first air conditioner outdoor unit.

[0092] Wherein, determining that the machine (such as the first air-conditioner outdoor unit) is the target machine can prevent false alarms caused by the detection of other sundries such as cartons passing through the detection device.

[0093] Specifically, in step S43, when there is a blade template image in the blade template library, after the blade image to be inspected is compared with all the blade template images in the blade template library using the edge matching method to obtain a first comparison result, if the similarity value between the blade image to be inspected and at least one blade template image in the blade template library in the comparison result is greater than a normal threshold (such as a set similarity threshold), it is considered that the product in the blade image to be inspected is a blade, and it is judged that the blade of the machine (such as the first air conditioner outdoor unit) is nutted.

[0094] In some embodiments, in step S420, the image of the second feature to be detected is compared with the reference image in the template library of the second feature to be detected in the target detection object to obtain the specific process of the comparison result, as shown in the following exemplary description.

[0095] The following combination Figure 5 The method of the present invention is shown as a flow chart of an embodiment of comparing the reference image in the template library of the second feature to be detected in the target detection object, further illustrating the specific process of comparing the reference image in the template library of the second feature to be detected in the target detection object in step S420, including: steps S510 to S530.

[0096] Step S510: Perform edge detection on the image of the second feature to be detected to obtain a gradient grayscale image of the image of the second feature to be detected. Also, perform edge detection on each reference image in the template library of the second feature to be detected in the target detection object to obtain a gradient grayscale image of each reference image of the second feature to be detected.

[0097] Step S520 , performing Euclidean distance calculation on the gradient grayscale image of the picture of the second feature to be detected and the gradient grayscale image of each reference picture of the second feature to be detected, to obtain a corresponding second set number of similarity values.

[0098] Step S530: Compare each similarity value in the second set number of similarity values ​​with the first set similarity threshold to determine that a second feature to be detected exists in the object to be detected if at least one similarity value in the second set number of similarity values ​​is greater than the first set similarity threshold.

[0099] like Figure 9 As shown, in the solution of the present invention, the detection process of the automatic modeling visual inspection system based on the self-learning algorithm also includes:

[0100] Specifically, in step S41, if there is a blade template image in the blade template library, an edge matching method is used to compare the blade image to be inspected with all blade template images in the blade template library to obtain a first comparison result. It can be seen that the automatic modeling visual inspection system based on the self-learning algorithm provided by the present invention first automatically establishes multiple pre-designed blade template images for inspecting various types of blade nuts. Figure 10 The following is a schematic diagram of the interface for establishing a wind blade template diagram. Automatically collect the wind blade diagram as the modeling interface of the wind blade template diagram, which can be as follows Figure 10 shown.

[0101] The edge matching method is used to compare the wind blade image to be inspected with all wind blade template images in the wind blade template library, which may specifically include:

[0102] Step S411: Use the Sobel operator to perform edge detection on the fan blade image to be detected to obtain a gradient grayscale image of the fan blade image to be detected. Also use the Sobel operator to perform edge detection on all fan blade template images to obtain a gradient grayscale image of each fan blade template image.

[0103] Step S412: perform Euclidean distance calculation on the gradient grayscale image of the wind blade image to be inspected and the gradient grayscale image of each wind blade template image in the gradient grayscale images of all wind blade template images to obtain a similarity value in the range of 0 to 1, where 0 represents completely different and 1 represents completely the same.

[0104] Step S413: Finally, based on the comparison between the similarity value and the threshold, it can be determined whether the product included in the fan blade image to be inspected is really a fan blade. According to experience, the normal threshold can be 0.8 and the minimum threshold can be 0.6.

[0105] Among them, judging whether the wind blade image to be inspected is a real wind blade includes: performing Sobel edge extraction on the wind blade image to be inspected to obtain the edge wind blade image of the wind blade image to be inspected, and at the same time performing Sobel edge extraction on the wind blade template image to obtain the edge template wind blade image, and then performing Euclidean distance calculation on the two images to obtain a similarity value. When the similarity value is greater than a set threshold, it is considered that the wind blade image to be inspected is a real wind blade image, otherwise it is not a real wind blade image.

[0106] At step S140, if it is determined that the second feature to be detected is also present in the object to be detected, the object to be detected is determined to be a target detection object, and it is determined that the object to be detected has no assembly defects. The target detection object is, for example, a fan blade of an air conditioner outdoor unit, the first feature to be detected is, for example, a nut, and the second feature to be detected is, for example, a fan blade. If the first feature to be detected is assembled on the second feature to be detected in the object to be detected, it is determined that the object to be detected has no assembly defects. If the first feature to be detected is not assembled on the second feature to be detected in the object to be detected, it is determined that the object to be detected has an assembly defect.

[0107] The present invention provides an automatic modeling visual inspection system based on a self-learning algorithm. This system can automatically replace manual modeling, such as automatically capturing a wind blade image as a template. This eliminates the need for manual modeling when inspecting product quality defects. This reduces staffing and increases efficiency, eliminating the need for employee intervention. The system automatically models and performs quality inspections upon startup.

[0108] In some embodiments, step S420 identifies whether a second feature to be detected exists in the object to be detected based on the image of the object to be detected and using a template library of the target detection object obtained by pre-modeling, and also includes: when there is no reference image of the new second feature to be detected in the template library, a process of automatically modeling the new second feature to be detected.

[0109] The following combination Figure 6 The flowchart of an embodiment of automatically modeling the new second feature to be detected in the method of the present invention further illustrates the specific process of automatically modeling the new second feature to be detected, which may include: steps S610 to S630.

[0110] At step S610, if the similarity value between the image of the second feature to be detected and at least one reference image in the template library of the second feature to be detected is less than or equal to a first set similarity threshold, then it is determined whether the similarity value between the image of the second feature to be detected and at least one reference image in the template library of the second feature to be detected is greater than a second set similarity threshold. The first set similarity threshold is greater than the second set similarity threshold.

[0111] At step S620, if the similarity between the image of the second feature to be detected and at least one reference image in the template library of the second feature to be detected is greater than the second set similarity threshold, the second feature to be detected in the object to be detected is determined to be a new second feature to be detected, and the image of the new second feature to be detected is stored in the template library of the second feature to be detected as a reference image for the new second feature to be detected in the template library of the second feature to be detected. Furthermore, the object to be detected, which includes the first object to be detected and the new second feature to be detected, is determined to be a target detection object, and it is determined that the object to be detected has no assembly defects.

[0112] At step S630, if the similarity value between the picture of the second feature to be detected and at least one reference picture in the template library of the second feature to be detected is less than or equal to the second set similarity threshold, it is determined that the object to be detected is not the target detection object.

[0113] like Figure 9 As shown, in the solution of the present invention, the detection process of the automatic modeling visual inspection system based on the self-learning algorithm also includes:

[0114] Step S44: If the similarity between the blade image to be inspected and at least one blade template image in the blade template image library is less than or equal to a normal threshold (e.g., a set similarity threshold), determine whether the similarity between the blade image to be inspected and at least one blade template image in the blade template image library is greater than a minimum blade similarity value. If so, proceed to step S45. Otherwise, proceed to step S46. The normal threshold (e.g., a set similarity threshold) is greater than the minimum blade similarity value.

[0115] Specifically, in step S44, when there is a wind blade template image in the wind blade template library, after comparing the wind blade image to be inspected with all the wind blade template images in the wind blade template library using the edge matching method to obtain a first comparison result, if the similarity value between the wind blade image to be inspected and at least one wind blade template image in the wind blade template library in the comparison result is less than or equal to a normal threshold (such as a set similarity threshold), then it is determined whether the similarity value between the wind blade image to be inspected and at least one wind blade template image in the wind blade template library is greater than the minimum wind blade similarity value.

[0116] Step S45: If the similarity between the blade image to be inspected and at least one blade template image in the blade template image library is less than or equal to the set similarity threshold, but the similarity between the blade image to be inspected and at least one blade template image in the blade template image library is greater than the minimum blade similarity value, then the blade image to be inspected is used as a new blade template image and added to the blade template image library. Thereafter, step S43 is executed: if the similarity between the blade image to be inspected and at least one blade template image in the blade template image library is less than or equal to the set similarity threshold, but the similarity between the blade image to be inspected and at least one blade template image in the blade template image library is greater than the minimum blade similarity value, then the unit (e.g., the first air conditioner outdoor unit) is determined to be the target unit, and it is determined that the blades on the unit (e.g., the first air conditioner outdoor unit) have been nuts.

[0117] Specifically, in step S45, when there is a blade template image in the blade template library, after the edge matching method is used to compare the blade image to be inspected with all the blade template images in the blade template library to obtain a first comparison result, if the similarity value between the blade image to be inspected and at least one blade template image in the blade template library in the comparison result is greater than a normal threshold (such as a set similarity threshold), but the similarity value between the blade image to be inspected and at least one blade template image in the blade template library is greater than the minimum blade similarity value, then the blade is considered to be a new blade type, and the picture of the blade is directly added to the blade template library. Similarly, it is determined that the blades of the machine (such as the first air conditioner outdoor unit) are nutted.

[0118] Step S46: If the similarity value between the fan blade image to be inspected and at least one fan blade template image in the fan blade template image library is less than or equal to the set similarity threshold, and the similarity value between the fan blade image to be inspected and at least one fan blade template image in the fan blade template image library is less than or equal to the minimum fan blade similarity value, it is determined that the machine (such as the first air conditioner outdoor unit) is not the target machine, and the product in the fan blade image to be inspected of the machine is a non-fan blade.

[0119] Specifically, in step S46, in step S45, when there is a blade template image in the blade template library, after the edge matching method is used to compare the blade image to be inspected with all the blade template images in the blade template library to obtain a first comparison result, if the similarity value between the blade image to be inspected and at least one blade template image in the blade template library in the comparison result is greater than a normal threshold (such as a set similarity threshold), and the similarity value between the blade image to be inspected and at least one blade template image in the blade template library is less than or equal to the minimum blade similarity value, it is considered that the product in the blade image to be inspected is not a blade, and it is judged that the machine (such as the first air conditioner outdoor unit) is not the target machine, and subsequent detection is skipped, that is, further detection is stopped.

[0120] Obviously, as more and more products are tested, there are more and more models, and the sizes and shapes of the fan blades are different, and the corresponding fan blade nuts are also different in size and shape. Figure 11 Schematic diagrams of templates for multiple types of fan blades, where (a)-(r) are schematic diagrams of templates for different types of fan blades. Figure 12 This is a schematic diagram of the interface for the detection and automatic modeling of multiple types of fan blades. If there is no fan blade template in the fan blade template library, the fan blade to be detected will be saved as the first fan blade template in the template library. Figure 12 Example shown.

[0121] It can be seen that the solution of the present invention, in addition to the automatic establishment of multiple pre-designed fan blade templates for detecting various types of fan blade nuts, adds the self-learning function of an automatic modeling visual inspection system based on a self-learning algorithm. Thus, the solution of the present invention is capable of autonomous learning and modeling without human intervention when adding new types of targets to be inspected. For example, it can be applied to the detection of whether various types of fan blades have nuts installed, solving the problem of being unable to continue using the machine to detect product quality defects due to the wide variety of products in the later stages of the inspection process. Furthermore, changing the model has no impact on the equipment, and the equipment can automatically model and generate templates, with high compatibility to achieve sustainable operation.

[0122] In some embodiments, the visual inspection method according to the solutions of the present invention further includes: a process of determining that the object to be inspected is the target inspection object and that the object to be inspected has an assembly defect.

[0123] The following combination Figure 7 The flowchart of an embodiment of determining that the object to be detected is the target detection object and that the object to be detected has assembly defects in the method of the present invention is shown, further illustrating the specific process of determining that the object to be detected is the target detection object and that the object to be detected has assembly defects, which may include: steps S710 to S720.

[0124] In step S710, if it is determined that the first feature to be detected does not exist in the object to be detected, based on the image of the object to be detected, a pre-trained micro-convolutional deep learning model is used to determine whether the second feature to be detected exists in the object to be detected.

[0125] Step S720 : When it is determined that the second feature to be detected exists in the object to be detected, it is determined that the object to be detected is a target detection object, and it is determined that an assembly defect exists in the object to be detected.

[0126] like Figure 9 As shown, in the solution of the present invention, the detection process of the automatic modeling visual inspection system based on the self-learning algorithm also includes:

[0127] Step S50: When there is no nut in the picture taken by the camera, search the current picture to be inspected according to the fan blade template in the fan blade template library to find the area most similar to the fan blade, and then execute step S51.

[0128] Specifically, in step S50, when there is no nut in the picture taken by the camera, a window search is performed on the entire picture of the wind blade template image according to the wind blade template image in the wind blade template library, and the area with the largest second comparison result is found as the wind blade image to be inspected, and then step S51 is executed.

[0129] The second comparison result may be a similarity value between the image of the fan blade to be inspected and the image of the area where at least one window is located in the fan blade template image.

[0130] Step S51: Determine whether the second comparison result is greater than the minimum fan blade similarity value: if so, execute step S52. Otherwise, execute step S46.

[0131] Specifically, in step S51, if the image captured by the camera does not contain a nut, a window search is performed on the entire image of the fan blade template in the fan blade template library. After finding the area with the largest second comparison result as the fan blade image to be inspected, a determination is made as to whether the second comparison result is greater than the minimum fan blade similarity value. If so, step S52 is executed. Otherwise, step S46 is executed.

[0132] Step S52: Determine that the machine is the target machine, and at the same time determine that the fan blade is missing a nut.

[0133] Specifically, in step S52, when there is no nut in the picture taken by the camera, a window search is performed on the entire picture of the fan blade template image in the fan blade template library, and the area with the largest second comparison result is found as the fan blade image to be inspected. If the second comparison result is greater than the minimum fan blade similarity value, it is considered that the product in the fan blade image to be inspected is a fan blade, and it is judged that the fan blade of the machine (such as the first air conditioner outdoor unit) is missing a nut.

[0134] Of course, in step S46 executed after step S51, if there is no nut in the picture taken by the camera, a window search is performed on the entire picture of the fan blade template image in the fan blade template library, and the area with the largest second comparison result is found as the fan blade image to be inspected. If the second comparison result is less than or equal to the minimum fan blade similarity value, it is considered that the product in the fan blade image to be inspected is not a fan blade, and it is judged that the machine (such as the first air conditioner outdoor unit) is not the target machine, and the detection is skipped, that is, subsequent detection is stopped.

[0135] In this way, filtering of non-detection targets is achieved. Figure 13Schematic diagram of the interface for filtering non-detection targets, where (a) is a schematic diagram of the interface for non-detection targets, and (b) is a schematic diagram of the interface for detection targets. The solution of the present invention can filter non-detection targets, such as automatically filtering non-target machines, thereby improving detection reliability.

[0136] Among them, filtering out non-target machines can reduce false alarms caused by other debris. For example, if a box passes through the device without a filtering function, the device will determine that the target has no nuts and will alarm.

[0137] In some embodiments, the solution of the present invention can realize the function of manually assisted machine learning of wind blades. For example, for template images with "cognitive errors" of the equipment, manual assistance can be provided to correct the errors. If the inspection personnel do not perform auxiliary corrections for a long time, the inspection pass rate of the online visual inspection system based on automatic modeling and autonomous learning mechanism may be slightly reduced. For example, the inspection pass rate of the online visual inspection system based on automatic modeling and autonomous learning mechanism may be reduced to 99.3%. If the inspection personnel regularly check the template images (such as once a week, which takes about 30 seconds each time) and perform auxiliary corrections, the inspection pass rate of the online visual inspection system based on automatic modeling and autonomous learning mechanism will be as high as 99.9%.

[0138] The device misidentifies a template image as an incorrect template, mistaking it for a blade image and saving it as a template. Assisted correction involves manually deleting the incorrect template.

[0139] Figure 14 The figure is a diagram of the interface for manual error correction and assisted learning. Among them, the software system of the online visual inspection system based on automatic modeling and autonomous learning mechanism provides an interface for viewing the template view, which is convenient for employees to check whether the template is abnormal. If the template is abnormal, manual error correction can be performed. Manual error correction can be specifically to manually check whether the acquired picture is a wind blade picture. If it is not a wind blade picture, it will be deleted and the picture of the wind blade picture will be left. Subsequently, the correct wind blade picture can be used as the wind blade template picture to match with the subsequently acquired pictures to filter out objects that are not wind blades, such as Figure 13 If NF appears in the image, it means that the subsequent image is a non-wind blade image.

[0140] The solution of the present invention can detect whether the nuts on the air conditioner outdoor unit's fan blades are fastened, and can also detect whether other products have similar defects. The process for detecting whether other products have similar defects is similar to that for detecting whether the nuts on the air conditioner outdoor unit's fan blades are fastened. In both cases, features are first extracted from the image of the product to be inspected, then matched with the corresponding product template image, and finally a result is obtained to determine whether the image of the product to be inspected and the corresponding product template image are similar.

[0141] According to the technical solution of the present embodiment, a picture of the object to be detected is obtained by taking a picture of the object to be detected that enters the detection area. The first feature to be detected in the picture of the object to be detected is identified using a micro-convolution deep learning model. When the first feature to be detected is identified in the picture of the object to be detected, the second feature to be detected in the picture of the object to be detected that contains the first feature to be detected is identified using the template image of the object to be detected pre-stored in the template library. When the second feature to be detected is also identified in the picture of the object to be detected, the object to be detected is considered to be a target detection object. When the second feature to be detected is not identified in the picture of the object to be detected, the object to be detected is considered to be a new detection object, and the picture of the object to be detected is stored in the template library to realize automatic modeling of the template image of the new detection object. Thus, by enabling the machine to perform visual defect inspection on the detected object using a micro-convolution deep learning model, automatic modeling of the template of the detected object can be realized, which greatly saves manual labor.

[0142] According to an embodiment of the present invention, a visual inspection device corresponding to the visual inspection method is also provided. Figure 8 The structural diagram of an embodiment of the device of the present invention is shown. The visual inspection device is used to perform visual defect detection on an object to be inspected, specifically to detect whether the object to be inspected is a target inspection object, and when the object to be inspected is the target inspection object and has no assembly defects, to detect whether the object to be inspected has assembly defects. In the case that the object to be inspected is the target inspection object, the object to be inspected includes: a first feature to be inspected and a second feature to be inspected, the first feature to be inspected being such as a nut, and the second feature to be inspected being such as a fan blade. If the second feature to be inspected in the object to be inspected is equipped with the first feature to be inspected, it is determined that the object to be inspected has no assembly defects. If the second feature to be inspected in the object to be inspected is not equipped with the first feature to be inspected, it is determined that the object to be inspected has assembly defects. The visual inspection device includes: an acquisition unit 102 and a control unit 104.

[0143] The acquisition unit 102 is configured to acquire an image of the object to be detected, which is recorded as a picture of the object to be detected. The object to be detected is a product that has entered a predetermined detection area. For example, a determination is made as to whether a product has entered the predetermined detection area. If a product has entered the detection area, the product is taken as the object to be detected, and a picture of the object to be detected is taken to obtain an image of the object to be detected, which is recorded as the picture of the object to be detected. The specific functions and processing of the acquisition unit 102 are described in step S110.

[0144] The control unit 104 is configured to determine whether the first feature to be detected exists in the object to be detected based on the image of the object to be detected and using a pre-trained convolutional deep learning model. The specific functions and processing of the control unit 104 are shown in step S120.

[0145] In some embodiments, the convolutional deep learning model is pre-loaded with a reference image of the first feature to be detected. The control unit 104 determines whether the first feature to be detected exists in the object to be detected based on the image of the object to be detected and using the pre-trained convolutional deep learning model, including:

[0146] The control unit 104 is further configured to perform a window search on the image of the object to be detected with a window size of a first set pixel to obtain a first set number of window images. The specific functions and processing of the control unit 104 are also shown in step S210.

[0147] The control unit 104 is further configured to use the convolutional deep learning model to determine whether a reference image of the first feature to be detected exists in each of the first set number of window images. The specific functions and processing of the control unit 104 are also described in step S220.

[0148] In some embodiments, the convolutional deep learning model includes: a convolution layer, a pooling layer, and a classification layer. The control unit 104 uses the convolutional deep learning model to determine whether a reference image of the first feature to be detected exists in each window image of the first set number of window images, including:

[0149] The control unit 104 is further configured to perform feature extraction on each of the first set number of window images using a convolutional layer in the convolutional deep learning model to obtain a feature map corresponding to each of the window images. A first set number of feature maps are obtained for the first set number of window images. The specific functions and processing of the control unit 104 are further described in step S310.

[0150] The control unit 104 is further configured to perform dimensionality reduction sampling on each of the first set number of feature maps through a pooling layer in the convolutional deep learning model to obtain a reduced-dimensionality feature map corresponding to each feature map. A first set number of reduced-dimensionality feature maps are obtained for the first set number of feature maps. The specific functions and processing of the control unit 104 are further described in step S320.

[0151] The control unit 104 is further configured to classify each of the first set number of reduced dimensionality feature maps using a classification layer in the convolutional deep learning model to determine a corresponding category of each reduced dimensionality feature map within a reference category range. The specific functions and processing of the control unit 104 are further described in step S330.

[0152] The control unit 104 is further configured to determine that at least one of the first set number of window images contains a reference image for the first feature to be detected, if the corresponding category of at least one of the first set number of reduced dimensionality feature images belongs to the category of the reference image for the first feature to be detected. The specific functions and processing of the control unit 104 are further described in step S340.

[0153] The structure of the micro-convolutional deep learning model is exemplified below.

[0154] In some examples, the structure of the micro-convolutional deep learning model is divided into 7 layers, specifically: 30*9*9 convolution layer -> 30*9*9 pooling layer -> 90*13*13 convolution layer -> 90*13*13 pooling layer -> 60*2*2 convolution layer -> 60*2*2 pooling layer -> classification layer.

[0155] The convolutional layer extracts features from the input variables, producing a feature map. In a 30x9x9 convolutional layer, 9x9 represents sliding a 9x9 weighted window across the input image. Each weight within the window is multiplied by the corresponding image grayscale value, and the sum is calculated as the output. After sliding the window across the entire input image, a feature map is obtained. 30 represents 30 windows with different weights, resulting in 30 different feature maps.

[0156] The pooling layer downsamples the feature map obtained by the previous convolutional layer to produce a reduced-dimensional feature map. In a 30x9x9 pooling layer, 9x9 represents sliding a 9x9 window across the input feature map, taking the maximum grayscale value in the window as the output, and producing a downsampled feature map. 30 represents downsampling all 30 feature maps in the previous layer.

[0157] The function of the classification layer is to classify the input dimensionality reduction feature map. For example, the classification layer generally uses softmax for logistic regression for classification.

[0158] The following is an example of how to use the micro-convolutional deep learning model.

[0159] In some examples, the mini-convolutional deep learning model is first trained using two corresponding types of sample data, ultimately resulting in a mini-convolutional deep learning model that can distinguish between these two types of sample data. The two corresponding types of sample data can be sample data of several images of nuts and sample data of several images of non-nuts. Once the mini-convolutional deep learning model is trained, the image to be tested is input into the trained mini-convolutional deep learning model, which then determines whether the image is a nut or not.

[0160] The processing details within the micro-convolutional deep learning model can be as follows: After the image input undergoes convolution processing in the convolution layer and pooling processing in the pooling layer, a reduced-dimensional feature map is obtained, which is recorded as the reduced-dimensional feature map. The reduced-dimensional feature map is then sent to the classification layer for classification, such as using a softmax classifier in the classification layer to determine which category the input image belongs to.

[0161] The control unit 104 is further configured to determine that the first feature to be detected exists in the object to be detected if at least one window image in the first set number of window images contains a reference image of the first feature to be detected. The specific functions and processing of the control unit 104 are further described in step S230.

[0162] The following takes the defect detection of whether the fan blades of the fan in the air conditioner outdoor unit have been nutted as an example to exemplify the specific implementation process of the solution of the present invention.

[0163] Taking the defect detection of whether the fan blades have been nutted as an example, the solution of the present invention provides an online visual inspection system based on automatic modeling and autonomous learning mechanism. Figure 9 The following is a schematic diagram of the inspection process of an automatic modeling visual inspection system based on a self-learning algorithm. Figure 9 As shown, in the solution of the present invention, the detection process of the automatic modeling visual inspection system based on the self-learning algorithm includes:

[0164] Step S10: When the device is turned on, the machine (such as the first air-conditioning outdoor unit) enters the detection area, and the fan blades of the first air-conditioning outdoor unit pass through the device, and then step S20 is executed.

[0165] The device may be a machine loaded with an automatic modeling visual inspection system based on a self-learning algorithm.

[0166] Step S20: When the fan blades of the first air conditioner outdoor unit pass by the device, the camera takes a picture and then executes step S30.

[0167] Step S30: Detect whether there is a nut in the image captured by the camera: if so, execute step S40. Otherwise, execute step S50.

[0168] In step S30, detecting whether there is a nut in the image captured by the camera may specifically include:

[0169] Step S301: The automatic modeling visual inspection system based on the self-learning algorithm performs a window search on the image. The window size is based on the size of the actual nut in the image, such as a window size of 60*60 pixels.

[0170] Among them, window search is to use a window of a specified size (such as 60*60 pixels) to slide across the entire image and extract a picture of the window size.

[0171] Step S302: Use the micro-convolutional deep learning model to determine whether the image in each window is a nut.

[0172] Among them, the process of using the micro-convolution deep learning model to judge is to input the picture of each window size into the micro-convolution deep learning model, and then the micro-convolution deep learning model performs feature extraction and classification on the picture of each window size to obtain the result of whether the picture is a nut.

[0173] The control unit 104 is further configured to, if it is determined that the first feature to be detected exists in the object to be detected, identify whether the second feature to be detected exists in the object to be detected using a pre-modeled template library of target detection objects based on the image of the object to be detected. The specific functions and processing of the control unit 104 are further described in step S130.

[0174] In some embodiments, the control unit 104 identifies whether the second feature to be detected exists in the object to be detected based on the image of the object to be detected and using a template library of target detection objects obtained by pre-modeling, including:

[0175] The control unit 104 is further configured to expand the area of ​​the image of the object to be detected where the image of the first feature to be detected is located to the periphery beyond the second set pixel, to serve as the image of the second feature to be detected. The specific functions and processing of the control unit 104 are further described in step S410.

[0176] The control unit 104 is further configured to compare the image of the second feature to be detected with reference images in a template library of the second feature to be detected in the target object to obtain a comparison result. The comparison result is a similarity value between the image of the second feature to be detected and at least one reference image in the template library of the second feature to be detected. The specific functions and processing of the control unit 104 are further described in step S420.

[0177] The control unit 104 is further configured to determine whether a similarity value between the image of the second feature to be detected and at least one reference image in the template library of the second feature to be detected is greater than a first set similarity threshold. The specific functions and processing of the control unit 104 are further described in step S430.

[0178] The control unit 104 is further configured to determine that the second feature to be detected exists in the object to be detected if a similarity value between the image of the second feature to be detected and at least one reference image in the template library of the second feature to be detected is greater than a first predetermined similarity threshold. The specific functions and processing of the control unit 104 are further described in step S440.

[0179] like Figure 9 As shown, in the solution of the present invention, the detection process of the automatic modeling visual inspection system based on the self-learning algorithm also includes:

[0180] Step S40: If there is a nut in the picture taken by the camera, the position of the nut in the picture taken by the camera is expanded outward by a certain range as the fan blade image to be inspected, and then step S41 is executed.

[0181] Specifically, in step S40, if there is a nut in the picture taken by the camera, the position of the nut is expanded outward by 200 pixels. The pixel value is the size of the wind blade in the picture taken by the camera, and then the expanded area is used as the wind blade image to be inspected.

[0182] Step S41: Compare the blade image to be inspected with the blade template images in the blade template image library to obtain a first comparison result, and then execute step S42. The first comparison result may be a similarity value between the blade image to be inspected and at least one blade template image in the blade template image library.

[0183] Step S42: Based on the first comparison result, determine whether the similarity between the blade image to be inspected and at least one blade template image in the blade template image library is greater than a set similarity threshold: if so, execute step S43. Otherwise, execute step S44.

[0184] Specifically, in step S42, if there is a blade template image in the blade template image library, after comparing the blade image to be inspected with all blade template images in the blade template image library using the edge matching method to obtain a first comparison result, it is determined whether the similarity value between the blade image to be inspected and at least one blade template image in the blade template image library in the comparison result is greater than a normal threshold (e.g., a set similarity threshold). If so, step S43 is executed. Otherwise, step S44 is executed.

[0185] Step S43: If the similarity between the fan blade image to be inspected and at least one fan blade template image in the fan blade template library is greater than the set similarity threshold, it is determined that the machine (such as the first air conditioner outdoor unit) is the target machine, and it is determined that nuts have been installed on the fan blades of the first air conditioner outdoor unit.

[0186] Specifically, in step S43, when there is a blade template image in the blade template library, after the blade image to be inspected is compared with all the blade template images in the blade template library using the edge matching method to obtain a first comparison result, if the similarity value between the blade image to be inspected and at least one blade template image in the blade template library in the comparison result is greater than a normal threshold (such as a set similarity threshold), it is considered that the product in the blade image to be inspected is a blade, and it is judged that the blade of the machine (such as the first air conditioner outdoor unit) is nutted.

[0187] In some embodiments, the control unit 104 compares the image of the second feature to be detected with a reference image in a template library of the second feature to be detected in the target detection object to obtain a comparison result, including:

[0188] The control unit 104 is further configured to perform edge detection on the image of the second feature to be detected to obtain a gradient grayscale image of the image of the second feature to be detected. Furthermore, edge detection is performed on each reference image in the template library of the second feature to be detected in the target object to obtain a gradient grayscale image of each reference image of the second feature to be detected. The specific functions and processing of the control unit 104 are further described in step S510.

[0189] The control unit 104 is further configured to perform a Euclidean distance calculation between the gradient grayscale image of the image of the second feature to be detected and the gradient grayscale image of each reference image of the second feature to be detected, to obtain a corresponding second set number of similarity values. The specific functions and processing of the control unit 104 are further described in step S520.

[0190] The control unit 104 is further configured to compare each similarity value in the second set number of similarity values ​​with a first set similarity threshold, and to determine that the second feature to be detected exists in the object to be detected if at least one similarity value in the second set number of similarity values ​​is greater than the first set similarity threshold. The specific functions and processing of the control unit 104 are further described in step S530.

[0191] like Figure 9 As shown, in the solution of the present invention, the detection process of the automatic modeling visual inspection system based on the self-learning algorithm also includes:

[0192] Specifically, in step S41, if there is a blade template image in the blade template library, an edge matching method is used to compare the blade image to be inspected with all blade template images in the blade template library to obtain a first comparison result. It can be seen that the automatic modeling visual inspection system based on the self-learning algorithm provided by the present invention first automatically establishes multiple pre-designed blade template images for inspecting various types of blade nuts. Figure 10 The following is a schematic diagram of the interface for establishing a wind blade template diagram. Automatically collect the wind blade diagram as the modeling interface of the wind blade template diagram, which can be as follows Figure 10 shown.

[0193] The edge matching method is used to compare the wind blade image to be inspected with all wind blade template images in the wind blade template library, which may specifically include:

[0194] Step S411: Use the Sobel operator to perform edge detection on the fan blade image to be detected to obtain a gradient grayscale image of the fan blade image to be detected. Also use the Sobel operator to perform edge detection on all fan blade template images to obtain a gradient grayscale image of each fan blade template image.

[0195] Step S412: perform Euclidean distance calculation on the gradient grayscale image of the wind blade image to be inspected and the gradient grayscale image of each wind blade template image in the gradient grayscale images of all wind blade template images to obtain a similarity value in the range of 0 to 1, where 0 represents completely different and 1 represents completely the same.

[0196] Step S413: Finally, based on the comparison between the similarity value and the threshold, it can be determined whether the product included in the fan blade image to be inspected is really a fan blade. According to experience, the normal threshold can be 0.8 and the minimum threshold can be 0.6.

[0197] Among them, judging whether the wind blade image to be inspected is a real wind blade includes: performing Sobel edge extraction on the wind blade image to be inspected to obtain the edge wind blade image of the wind blade image to be inspected, and at the same time performing Sobel edge extraction on the wind blade template image to obtain the edge template wind blade image, and then performing Euclidean distance calculation on the two images to obtain a similarity value. When the similarity value is greater than a set threshold, it is considered that the wind blade image to be inspected is a real wind blade image, otherwise it is not a real wind blade image.

[0198] The control unit 104 is further configured to determine that the object to be detected is a target detection object and to determine that the object to be detected has no assembly defects when it is determined that the second feature to be detected also exists in the object to be detected. The specific functions and processing of the control unit 104 can also be found in step S140. The target detection object is such as a fan blade of an air conditioner outdoor unit, the first feature to be detected is such as a nut, and the second feature to be detected is such as a fan blade. If the second feature to be detected in the object to be detected is equipped with the first feature to be detected, it is determined that the object to be detected has no assembly defects. If the second feature to be detected in the object to be detected is not equipped with the first feature to be detected, it is determined that the object to be detected has an assembly defect.

[0199] The present invention provides an automatic modeling visual inspection system based on a self-learning algorithm. This system can automatically replace manual modeling, such as automatically capturing a wind blade image as a template. This eliminates the need for manual modeling when inspecting product quality defects. This reduces staffing and increases efficiency, eliminating the need for employee intervention. The system automatically models and performs quality inspections upon startup.

[0200] In some embodiments, the control unit 104, based on the image of the object to be detected and using a pre-modeled template library of target detection objects, identifies whether the object to be detected has a second feature to be detected. The process further includes: if there is no reference image of the new second feature to be detected in the template library, automatically modeling the new second feature to be detected. Specifically, the process may be as follows:

[0201] The control unit 104 is further configured to determine whether the similarity between the image of the second feature to be detected and at least one reference image in the template library of the second feature to be detected is greater than a second set similarity threshold if the similarity between the image of the second feature to be detected and at least one reference image in the template library of the second feature to be detected is less than or equal to a first set similarity threshold. The first set similarity threshold is greater than the second set similarity threshold. The specific functions and processing of the control unit 104 are also described in step S610.

[0202] The control unit 104 is further configured to determine that the second feature to be detected in the object to be detected is a new second feature to be detected if the similarity value between the image of the second feature to be detected and at least one reference image in the template library of the second feature to be detected is greater than the second set similarity threshold, and to store the image of the new second feature to be detected in the template library of the second feature to be detected as a reference image of the new second feature to be detected in the template library of the second feature to be detected. Furthermore, the control unit 104 determines that the object to be detected, which includes the first object to be detected and the new second feature to be detected, is a target detection object, and determines that the object to be detected has no assembly defects. The specific functions and processing of the control unit 104 are also described in step S620.

[0203] The control unit 104 is further configured to determine that the object to be detected is not the target detection object if the similarity between the image of the second feature to be detected and at least one reference image in the template library of the second feature to be detected is less than or equal to the second predetermined similarity threshold. The specific functions and processing of the control unit 104 are further described in step S630.

[0204] like Figure 9 As shown, in the solution of the present invention, the detection process of the automatic modeling visual inspection system based on the self-learning algorithm also includes:

[0205] Step S44: If the similarity between the blade image to be inspected and at least one blade template image in the blade template image library is less than or equal to a normal threshold (e.g., a set similarity threshold), determine whether the similarity between the blade image to be inspected and at least one blade template image in the blade template image library is greater than a minimum blade similarity value. If so, proceed to step S45. Otherwise, proceed to step S46. The normal threshold (e.g., a set similarity threshold) is greater than the minimum blade similarity value.

[0206] Specifically, in step S44, when there is a wind blade template image in the wind blade template library, after comparing the wind blade image to be inspected with all the wind blade template images in the wind blade template library using the edge matching method to obtain a first comparison result, if the similarity value between the wind blade image to be inspected and at least one wind blade template image in the wind blade template library in the comparison result is less than or equal to a normal threshold (such as a set similarity threshold), then it is determined whether the similarity value between the wind blade image to be inspected and at least one wind blade template image in the wind blade template library is greater than the minimum wind blade similarity value.

[0207] Step S45: If the similarity between the blade image to be inspected and at least one blade template image in the blade template image library is less than or equal to the set similarity threshold, but the similarity between the blade image to be inspected and at least one blade template image in the blade template image library is greater than the minimum blade similarity value, then the blade image to be inspected is used as a new blade template image and added to the blade template image library. Thereafter, step S43 is executed: if the similarity between the blade image to be inspected and at least one blade template image in the blade template image library is less than or equal to the set similarity threshold, but the similarity between the blade image to be inspected and at least one blade template image in the blade template image library is greater than the minimum blade similarity value, then the unit (e.g., the first air conditioner outdoor unit) is determined to be the target unit, and it is determined that the blades on the unit (e.g., the first air conditioner outdoor unit) have been nuts.

[0208] Specifically, in step S45, when there is a blade template image in the blade template library, after the edge matching method is used to compare the blade image to be inspected with all the blade template images in the blade template library to obtain a first comparison result, if the similarity value between the blade image to be inspected and at least one blade template image in the blade template library in the comparison result is greater than a normal threshold (such as a set similarity threshold), but the similarity value between the blade image to be inspected and at least one blade template image in the blade template library is greater than the minimum blade similarity value, then the blade is considered to be a new blade type, and the picture of the blade is directly added to the blade template library. Similarly, it is determined that the blades of the machine (such as the first air conditioner outdoor unit) are nutted.

[0209] Step S46: If the similarity value between the fan blade image to be inspected and at least one fan blade template image in the fan blade template image library is less than or equal to the set similarity threshold, and the similarity value between the fan blade image to be inspected and at least one fan blade template image in the fan blade template image library is less than or equal to the minimum fan blade similarity value, it is determined that the machine (such as the first air conditioner outdoor unit) is not the target machine, and the product in the fan blade image to be inspected of the machine is a non-fan blade.

[0210] Specifically, in step S46, in step S45, when there is a blade template image in the blade template library, after the edge matching method is used to compare the blade image to be inspected with all the blade template images in the blade template library to obtain a first comparison result, if the similarity value between the blade image to be inspected and at least one blade template image in the blade template library in the comparison result is greater than a normal threshold (such as a set similarity threshold), and the similarity value between the blade image to be inspected and at least one blade template image in the blade template library is less than or equal to the minimum blade similarity value, it is considered that the product in the blade image to be inspected is not a blade, and it is judged that the machine (such as the first air conditioner outdoor unit) is not the target machine, and subsequent detection is skipped, that is, further detection is stopped.

[0211] Obviously, as more and more products are tested, there are more and more models, and the sizes and shapes of the fan blades are different, and the corresponding fan blade nuts are also different in size and shape. Figure 11 Schematic diagrams of templates for multiple types of fan blades, where (a)-(r) are schematic diagrams of templates for different types of fan blades. Figure 12 This is a schematic diagram of the interface for the detection and automatic modeling of multiple types of fan blades. If there is no fan blade template in the fan blade template library, the fan blade to be detected will be saved as the first fan blade template in the template library. Figure 12 Example shown.

[0212] It can be seen that the solution of the present invention, in addition to the automatic establishment of multiple pre-designed fan blade templates for detecting various types of fan blade nuts, adds the self-learning function of an automatic modeling visual inspection system based on a self-learning algorithm. Thus, the solution of the present invention is capable of autonomous learning and modeling without human intervention when adding new types of targets to be inspected. For example, it can be applied to the detection of whether various types of fan blades have nuts installed, solving the problem of being unable to continue using the machine to detect product quality defects due to the wide variety of products in the later stages of the inspection process. Furthermore, changing the model has no impact on the equipment, and the equipment can automatically model and generate templates, with high compatibility to achieve sustainable operation.

[0213] In some embodiments, the visual inspection device according to the solution of the present invention further includes: a process of determining that the object to be inspected is the target inspection object and that the object to be inspected has an assembly defect, which may be specifically as follows:

[0214] The control unit 104 is further configured to, if it is determined that the first feature to be detected is not present in the object to be detected, determine whether the second feature to be detected is present in the object to be detected using a pre-trained miniature convolutional deep learning model based on the image of the object to be detected. The specific functions and processing of the control unit 104 are further described in step S710.

[0215] The control unit 104 is further configured to, when determining that the second feature to be detected exists in the object to be detected, determine that the object to be detected is a target detection object and determine that the object to be detected has an assembly defect. The specific functions and processing of the control unit 104 are also shown in step S720.

[0216] like Figure 9 As shown, in the solution of the present invention, the detection process of the automatic modeling visual inspection system based on the self-learning algorithm also includes:

[0217] Step S50: When there is no nut in the picture taken by the camera, search the current picture to be inspected according to the fan blade template in the fan blade template library to find the area most similar to the fan blade, and then execute step S51.

[0218] Specifically, in step S50, when there is no nut in the picture taken by the camera, a window search is performed on the entire picture of the wind blade template image according to the wind blade template image in the wind blade template library, and the area with the largest second comparison result is found as the wind blade image to be inspected, and then step S51 is executed.

[0219] The second comparison result may be a similarity value between the image of the fan blade to be inspected and the image of the area where at least one window is located in the fan blade template image.

[0220] Step S51: Determine whether the second comparison result is greater than the minimum fan blade similarity value: if so, execute step S52. Otherwise, execute step S46.

[0221] Specifically, in step S51, if the image captured by the camera does not contain a nut, a window search is performed on the entire image of the fan blade template in the fan blade template library. After finding the area with the largest second comparison result as the fan blade image to be inspected, a determination is made as to whether the second comparison result is greater than the minimum fan blade similarity value. If so, step S52 is executed. Otherwise, step S46 is executed.

[0222] Step S52: Determine that the machine is the target machine, and at the same time determine that the fan blade is missing a nut.

[0223] Specifically, in step S52, when there is no nut in the picture taken by the camera, a window search is performed on the entire picture of the fan blade template image in the fan blade template library, and the area with the largest second comparison result is found as the fan blade image to be inspected. If the second comparison result is greater than the minimum fan blade similarity value, it is considered that the product in the fan blade image to be inspected is a fan blade, and it is judged that the fan blade of the machine (such as the first air conditioner outdoor unit) is missing a nut.

[0224] Of course, in step S46 executed after step S51, if there is no nut in the picture taken by the camera, a window search is performed on the entire picture of the fan blade template image in the fan blade template library, and the area with the largest second comparison result is found as the fan blade image to be inspected. If the second comparison result is less than or equal to the minimum fan blade similarity value, it is considered that the product in the fan blade image to be inspected is not a fan blade, and it is judged that the machine (such as the first air conditioner outdoor unit) is not the target machine, and the detection is skipped, that is, subsequent detection is stopped.

[0225] In this way, filtering of non-detection targets is achieved. Figure 13 Schematic diagram of the interface for filtering non-detection targets, where (a) is a schematic diagram of the interface for non-detection targets, and (b) is a schematic diagram of the interface for detection targets. The solution of the present invention can filter non-detection targets, such as automatically filtering non-target machines, thereby improving detection reliability.

[0226] In some embodiments, the solution of the present invention can realize the function of manually assisted machine learning of wind blades. For example, for template images with "cognitive errors" of the equipment, manual assistance can be provided to correct the errors. If the inspection personnel do not perform auxiliary corrections for a long time, the inspection pass rate of the online visual inspection system based on automatic modeling and autonomous learning mechanism may be slightly reduced. For example, the inspection pass rate of the online visual inspection system based on automatic modeling and autonomous learning mechanism may be reduced to 99.3%. If the inspection personnel regularly check the template images (such as once a week, which takes about 30 seconds each time) and perform auxiliary corrections, the inspection pass rate of the online visual inspection system based on automatic modeling and autonomous learning mechanism will be as high as 99.9%.

[0227] Figure 14 The figure is a diagram of the interface for manual error correction and assisted learning. Among them, the software system of the online visual inspection system based on automatic modeling and autonomous learning mechanism provides an interface for viewing the template view, which is convenient for employees to check whether the template is abnormal. If the template is abnormal, manual error correction can be performed. Manual error correction can be specifically to manually check whether the acquired picture is a wind blade picture. If it is not a wind blade picture, it will be deleted and the picture of the wind blade picture will be left. Subsequently, the correct wind blade picture can be used as the wind blade template picture to match with the subsequently acquired pictures to filter out objects that are not wind blades, such as Figure 13 If NF appears in the image, it means that the subsequent image is a non-wind blade image.

[0228] The solution of the present invention can detect whether the nuts on the air conditioner outdoor unit's fan blades are fastened, and can also detect whether other products have similar defects. The process for detecting whether other products have similar defects is similar to that for detecting whether the nuts on the air conditioner outdoor unit's fan blades are fastened. In both cases, features are first extracted from the image of the product to be inspected, then matched with the corresponding product template image, and finally a result is obtained to determine whether the image of the product to be inspected and the corresponding product template image are similar.

[0229] Since the processing and functions implemented by the device of this embodiment basically correspond to the embodiments, principles and examples of the aforementioned method, for any details not fully described in this embodiment, please refer to the relevant descriptions in the aforementioned embodiments and will not be repeated here.

[0230] By adopting the technical solution of the present invention, a picture of the object to be detected is obtained by taking a picture of the object to be detected that enters the detection area; a first feature to be detected in the picture of the object to be detected is identified by using a micro-convolutional deep learning model; when the first feature to be detected is identified in the picture of the object to be detected, a second feature to be detected in the picture of the object to be detected that contains the first feature to be detected is identified by using a template image of the object to be detected pre-stored in a template library; when the second feature to be detected is also identified in the picture of the object to be detected, the object to be detected is considered to be a target detection object; when the second feature to be detected is not identified in the picture of the object to be detected, the object to be detected is considered to be a new detection object, and the picture of the object to be detected is stored in the template library to realize automatic modeling of the template image of the new detection object. Replacing the model has no impact on the device, and the device can automatically model and generate templates, with high compatibility to achieve sustainable operation.

[0231] According to an embodiment of the present invention, a terminal corresponding to the visual detection device is also provided. The terminal may include: the visual detection device described above.

[0232] Since the processing and functions implemented by the terminal of this embodiment basically correspond to the embodiments, principles and examples of the aforementioned devices, for any details not fully described in this embodiment, please refer to the relevant descriptions in the aforementioned embodiments and will not be repeated here.

[0233] By adopting the technical solution of the present invention, a picture of the object to be detected is obtained by taking a picture of the object to be detected that enters the detection area; a first feature to be detected in the picture of the object to be detected is identified by using a micro-convolutional deep learning model; when the first feature to be detected is identified in the picture of the object to be detected, a second feature to be detected in the picture of the object to be detected that contains the first feature to be detected is identified by using a template image of the object to be detected pre-stored in a template library; when the second feature to be detected is also identified in the picture of the object to be detected, the object to be detected is considered to be a target detection object; when the second feature to be detected is not identified in the picture of the object to be detected, the object to be detected is considered to be a new detection object, and the picture of the object to be detected is stored in the template library to realize automatic modeling of the template image of the new detection object. When a new type of target to be detected is added, autonomous learning and autonomous modeling can be achieved without human intervention, and the detection efficiency is high.

[0234] According to an embodiment of the present invention, a storage medium corresponding to the visual detection method is also provided, wherein the storage medium includes a stored program, wherein when the program is run, the device where the storage medium is located is controlled to execute the above-mentioned visual detection method.

[0235] Since the processing and functions implemented by the storage medium of this embodiment basically correspond to the embodiments, principles and examples of the aforementioned method, for any details not fully described in this embodiment, please refer to the relevant descriptions in the aforementioned embodiments and will not be repeated here.

[0236] By adopting the technical solution of the present invention, a picture of the object to be detected is obtained by taking a picture of the object to be detected that enters the detection area; a first feature to be detected in the picture of the object to be detected is identified by using a micro-convolutional deep learning model; when the first feature to be detected is identified in the picture of the object to be detected, a second feature to be detected in the picture of the object to be detected that contains the first feature to be detected is identified by using a template image of the object to be detected pre-stored in a template library; when the second feature to be detected is also identified in the picture of the object to be detected, the object to be detected is considered to be a target detection object; when the second feature to be detected is not identified in the picture of the object to be detected, the object to be detected is considered to be a new detection object, and the picture of the object to be detected is stored in the template library to realize automatic modeling of the template image of the new detection object, which can reduce staff and increase efficiency without the need for employee intervention. After the equipment is turned on, it can automatically model and perform quality inspection.

[0237] In summary, it is easy for those skilled in the art to understand that, under the premise of no conflict, the above-mentioned advantageous methods can be freely combined and superimposed.

[0238] The foregoing description is merely an embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of the claims.

Claims

1. A visual inspection method, characterized in that: The visual inspection method is applied to perform visual defect detection on an object to be inspected; the visual inspection method comprises: Obtain an image of the object to be detected, recorded as a picture of the object to be detected; Based on the image of the object to be detected, using a pre-trained convolutional deep learning model, determining whether the object to be detected has a first feature to be detected; If it is determined that the first feature to be detected exists in the object to be detected, based on the picture of the object to be detected, using the template library of the target detection object obtained by pre-modeling, identify whether the second feature to be detected exists in the object to be detected, including: expanding the area after the second set pixel from the position of the picture of the first feature to be detected in the picture of the object to be detected to the periphery as the picture of the second feature to be detected; comparing the picture of the second feature to be detected with the reference picture in the template library of the second feature to be detected in the target detection object to obtain a comparison result; the comparison result is a similarity value between the picture of the second feature to be detected and at least one reference picture in the template library of the second feature to be detected; determining whether the similarity value between the picture of the second feature to be detected and at least one reference picture in the template library of the second feature to be detected is greater than a first set similarity threshold; if the similarity value between the picture of the second feature to be detected and at least one reference picture in the template library of the second feature to be detected is greater than the first set similarity threshold, then determining that the second feature to be detected exists in the object to be detected; When it is determined that the second feature to be detected also exists in the object to be detected, it is determined that the object to be detected is a target detection object, and it is determined that the object to be detected has no assembly defects.

2. The visual inspection method according to claim 1, wherein: Based on the image of the object to be detected, using a pre-trained convolutional deep learning model, determining whether the object to be detected has a first feature to be detected, including: Performing a window search on the image of the object to be detected with a window size of a first set pixel to obtain a first set number of window images; Determine, using the convolutional deep learning model, whether there is a reference image of the first feature to be detected in each window image of the first set number of window images; If it is determined that there is a reference picture of the first feature to be detected in at least one window picture among the first set number of window pictures, it is determined that the first feature to be detected exists in the object to be detected.

3. The visual inspection method according to claim 2, characterized in that: Determining, by using the convolutional deep learning model, whether a reference picture of the first feature to be detected exists in each of the first set number of window pictures, comprising: Performing feature extraction on each of the first set number of window images through the convolution layer in the convolutional deep learning model to obtain a feature map corresponding to each of the window images; obtaining a first set number of feature maps for the first set number of window images; Performing dimensionality reduction sampling on each of the first set number of feature maps through the pooling layer in the convolutional deep learning model to obtain a reduced-dimensionality feature map corresponding to each of the feature maps; and obtaining a first set number of reduced-dimensionality feature maps for the first set number of feature maps; Classifying each of the first set number of reduced dimensionality feature maps by a classification layer in the convolutional deep learning model to determine a corresponding category of each of the reduced dimensionality feature maps within a reference category range; If the corresponding category of at least one of the first set number of reduced dimensionality feature maps belongs to the category of the reference image of the first feature to be detected, it is determined that there is a reference image of the first feature to be detected in at least one window image among the first set number of window images.

4. The visual inspection method according to claim 1, wherein: in, Comparing the image of the second feature to be detected with a reference image in a template library of the second feature to be detected in the target detection object to obtain a comparison result, including: Performing edge detection on the image of the second feature to be detected to obtain a gradient grayscale image of the image of the second feature to be detected; and performing edge detection on each reference image in the template library of the second feature to be detected in the target detection object to obtain a gradient grayscale image of each reference image of the second feature to be detected; Performing Euclidean distance calculation on the gradient grayscale image of the picture of the second feature to be detected and the gradient grayscale image of each reference picture of the second feature to be detected to obtain a corresponding second set number of similarity values; comparing each similarity value of the second set number of similarity values ​​with a first set similarity threshold, so as to determine that a second feature to be detected exists in the object to be detected if at least one similarity value among the second set number of similarity values ​​is greater than the first set similarity threshold; and / or, Based on the image of the object to be detected, using a template library of target detection objects obtained by pre-modeling, identifying whether the object to be detected has a second feature to be detected, further comprising: If the similarity value between the image of the second feature to be detected and at least one reference image in the template library of the second feature to be detected is less than or equal to a first set similarity threshold, determining whether the similarity value between the image of the second feature to be detected and at least one reference image in the template library of the second feature to be detected is greater than a second set similarity threshold; the first set similarity threshold is greater than the second set similarity threshold; If the similarity value between the image of the second feature to be detected and at least one reference image in the template library of the second feature to be detected is greater than the second set similarity threshold, then the second feature to be detected in the object to be detected is determined to be a new second feature to be detected, and the image of the new second feature to be detected is stored in the template library of the second feature to be detected as a reference image of the new second feature to be detected in the template library of the second feature to be detected; and, it is determined that the object to be detected including the first object to be detected and the new second feature to be detected is a target detection object, and it is determined that the object to be detected has no assembly defects; If the similarity value between the picture of the second feature to be detected and at least one reference picture in the template library of the second feature to be detected is less than or equal to the second set similarity threshold, it is determined that the object to be detected is not the target detection object.

5. The visual inspection method according to any one of claims 1 to 4, characterized in that: Also includes: If it is determined that the first feature to be detected does not exist in the object to be detected, determine whether the second feature to be detected exists in the object to be detected based on the image of the object to be detected and using a pre-trained micro-convolutional deep learning model; In the case where it is determined that the second feature to be detected exists in the object to be detected, it is determined that the object to be detected is a target detection object, and it is determined that an assembly defect exists in the object to be detected.

6. A visual inspection device, characterized in that: The visual inspection device is used to perform visual defect detection on an object to be inspected; the visual inspection device includes: an acquisition unit, configured to acquire an image of the object to be detected, recorded as a picture of the object to be detected; A control unit is configured to determine whether a first feature to be detected exists in the object to be detected based on the image of the object to be detected and using a pre-trained convolutional deep learning model; The control unit is further configured to, if it is determined that the first feature to be detected exists in the object to be detected, identify whether the second feature to be detected exists in the object to be detected based on the picture of the object to be detected and using the template library of the target detection object obtained by pre-modeling, including: expanding the area after the second set pixel from the position of the picture of the first feature to be detected in the picture of the object to be detected to the periphery as the picture of the second feature to be detected; comparing the picture of the second feature to be detected with the reference picture in the template library of the second feature to be detected in the target detection object to obtain a comparison result; the comparison result is a similarity value between the picture of the second feature to be detected and at least one reference picture in the template library of the second feature to be detected; determining whether the similarity value between the picture of the second feature to be detected and at least one reference picture in the template library of the second feature to be detected is greater than a first set similarity threshold; if the similarity value between the picture of the second feature to be detected and at least one reference picture in the template library of the second feature to be detected is greater than the first set similarity threshold, determining that the second feature to be detected exists in the object to be detected; The control unit is further configured to, when it is determined that the second feature to be detected also exists in the object to be detected, determine that the object to be detected is a target detection object and determine that the object to be detected has no assembly defects.

7. The visual inspection device according to claim 6, characterized in that: The control unit, based on the image of the object to be detected, uses a pre-trained convolutional deep learning model to determine whether a first feature to be detected exists in the object to be detected, including: Performing a window search on the image of the object to be detected with a window size of a first set pixel to obtain a first set number of window images; Determine, using the convolutional deep learning model, whether there is a reference image of the first feature to be detected in each window image of the first set number of window images; If it is determined that there is a reference picture of the first feature to be detected in at least one window picture among the first set number of window pictures, it is determined that the first feature to be detected exists in the object to be detected.

8. A terminal, characterized in that: include: The visual inspection device according to any one of claims 6 to 7.

9. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is run, the device where the storage medium is located is controlled to execute the visual inspection method according to any one of claims 1 to 5.

Images