Defect detection network construction method, anomaly detection method and system, and storage medium
By building a defect detection network of the convolutional neural network module, using open data sets and standard reference images for training, the problems of unstable feature extraction and poor cross-domain migration capabilities in object surface anomaly detection are solved, and higher detection is achieved Accuracy and stability.
Patent Information
- Application Number
- CN202010849101.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-08-21
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2040-08-21
Smart Images

Figure CN112001903B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a defect detection network construction method, an anomaly detection method and device, and a storage medium. Background Art
[0002] In the process of industrial product quality control, it is better to have over-inspection than to allow missed inspection. The surface defect detection method in industrial applications can often be regarded as the application of image-based object surface anomaly detection technology, which can be understood as "multi-classification problem of unbalanced data sets under unsupervised or weak supervision". Unlike balanced learning, anomaly detection is generally unsupervised. It seems to be binary classification but it is actually multi-classification (the causes of anomalies are different). In the case of not knowing how many categories there are in fact and without real labels, the effect of anomaly detection is often unsatisfactory. At present, the deployment of object surface anomaly detection in industry still has the following problems: it is easier to obtain good samples in industry, but it is difficult to obtain defective samples in the production process, resulting in unbalanced sample data with more good samples and fewer defective samples; surface scratches and other defects are related to the shooting angle during the shooting process, and the defects of the object can only be reproduced in a specific direction; although the causes of sample defects are different, they are relatively similar in the image presentation, making it difficult to distinguish which category of defects they belong to. In order to better judge the abnormal defects in the image, it is necessary to select a stable feature extraction method.
[0003] Traditional methods for detecting anomalies on the surface of objects often measure similarities such as distance and angle. These methods perform similarly in low dimensions because the core assumption is that "the representation of anomalies is different from that of normal points and they are in the minority." However, most similar methods face the curse of dimensionality, that is, common similarity metrics often fail on high-dimensional data. For high-dimensional data, the goal is to find a good space / representation, and then finding anomalies becomes a simple matter of measuring similarity. Another problem brought about by high-dimensional data is scalability. It is well known that the system computational overhead of measuring similarity is large. For example, the complexity of distance measurement is above O(n). At this time, it is necessary to use data structure optimization or dynamic programming to reduce complexity, which brings difficulties to controlling data dimensions.
[0004] The currently used surface anomaly detection methods for objects still have the following shortcomings: (1) The feature extraction network is unstable and cannot achieve good cross-domain migration; (2) In order to extract better features, the network structure design is relatively complex, resulting in long algorithm operation time and unable to meet the requirements of real-time detection and distributed detection; (3) When using traditional methods for anomaly detection, feature selection depends on the engineer's algorithm design experience, which requires a high level of human experience and is prone to manual detection failure; (4) The feature representation is simplistic and the defect detection stability is poor. Summary of the invention
[0005] The main technical problem solved by the present invention is how to improve the feature extraction performance of the existing feature extraction network and improve the stability and accuracy of surface defect detection of objects. In order to solve the above technical problems, the present application provides a defect detection network construction method, anomaly detection method and system, and storage medium.
[0006] According to the first aspect, an embodiment provides a method for constructing a defect detection network, which includes: obtaining a sample image in a preset open data set and a reference image of a standard product of an object to be detected; configuring a convolutional neural network module or multiple convolutional neural network modules of different scales to form a defect detection network, wherein each of the convolutional neural network modules includes a main feature extraction model and a slave feature extraction model; using the sample image to train the main feature extraction model in each of the convolutional neural network modules to learn and obtain corresponding network parameters; inputting the reference image into the main feature extraction model and the slave feature extraction model in each of the convolutional neural network modules to obtain corresponding first feature vectors and second feature vectors respectively; constructing a loss function of the slave feature extraction model according to the first feature vector and the second feature vector, and obtaining the network parameters of the slave feature extraction model through a back propagation algorithm; configuring the defect detection network according to the network parameters of the main feature extraction model and the network parameters of the slave feature extraction model in each of the convolutional neural networks.
[0007] The main feature extraction model in each convolutional neural network module is trained using the sample image to obtain corresponding network parameters, including: for each main feature extraction model in the convolutional neural network module, a feature extraction network and a pre-training network constituting the main feature extraction model are established; part of the sample images in the open data set are used as normal samples, other sample images randomly selected outside the normal samples are used as negative samples, and Gaussian noise is added to the normal samples to obtain corresponding positive samples; the normal samples, the positive samples and the negative samples are input into the feature extraction network and the pre-training network together, and the feature extraction network is guided by the pre-training network for training, so as to obtain the network parameters corresponding to the main feature extraction model.
[0008] The normal sample, the positive sample and the negative sample are input into the feature extraction network and the pre-trained network, and the feature extraction network is trained by the pre-trained network to obtain the network parameters corresponding to the main feature extraction model, including: constructing the loss function of the main feature extraction model, and expressed as
[0009] L(M)=w1L1+w2L2+w3L3;
[0010] Wherein, M is the identifier of the main feature extraction model, w1≥0, w2≥0, w3≥0 and are all weight coefficients; L1 is used to characterize the distance between the normal sample, the positive sample and the negative sample, and L1=max(d1-d2,0), and the distance between the normal sample and the positive sample is d1=||uu + ||2, the distance between the normal sample, the positive sample and the negative sample is d2=(||uu - ||2+||u + -u - ||2)*0.5,u,u + 、u - They represent the feature vectors output by the normal sample, the positive sample, and the negative sample after passing through the feature extraction network, respectively. max() represents the maximum value operation, and |||| represents the norm operation. L2 is used to characterize the similarity of feature vectors between different samples. For the case where there are m normal samples as input, the feature vector output by the feature extraction network is n-dimensional, and each output feature vector is recorded as v i (i=1,2,…,m), the eigenvectors form a matrix A n×m =[v1,v2,…,v m ], where the similarity between feature vectors is C n×n =AA T , and the element is represented by c ij , then i and j are both indexes of feature vectors; L3 is used to characterize the difference between feature vectors, and L3 = ||xy||2, x is the feature vector extracted by feature vector u through the fully connected layer, and y is the feature vector extracted by the normal sample through the pre-trained network; through the loss function L(M), the pre-trained network guides the feature extraction network to train, and obtains the network parameters corresponding to the main feature extraction model.
[0011] Inputting the reference image into the main feature extraction model and the sub-feature extraction model in each of the convolutional neural network modules to obtain the corresponding first feature vector and the second feature vector respectively, including: for each main feature extraction model in the convolutional neural network module, inputting the reference image into the feature extraction network in the main feature extraction model to obtain the corresponding first feature vector Where M is the identifier of the main feature extraction model, r and c represent the row and column pixel coordinates of the image respectively; for each secondary feature extraction model in the convolutional neural network module, the reference image is input into the secondary feature extraction model to obtain the corresponding second feature vector Where S is the identifier from the feature extraction model.
[0012] The loss function of the defect detection network is constructed according to the first feature vector and the second feature vector, and the network parameters of the feature extraction model are obtained by back propagation algorithm learning, including: and the second eigenvector The loss function of the feature extraction model is constructed and expressed as
[0013]
[0014] in, represents the vector mean of the feature vector output after the reference image passes through the feature extraction network, σ represents the vector variance of the feature vector output after the reference image passes through the feature extraction network, w and h are the width and height of the reference image, respectively, and diag() represents the diagonal matrix construction function; according to the loss function L(S i ) executes a back propagation algorithm to learn and obtain network parameters of the defect detection network.
[0015] According to the second aspect, an embodiment provides a method for detecting anomalies on the surface of an object, which includes: obtaining a detection image of the object to be detected; inputting the detection image into a defect detection network obtained by the construction method described in the first aspect above to obtain surface features of the object to be detected; calculating the confidence of the surface features of the object to be detected and comparing them with a preset threshold, and determining the surface abnormality features of the object to be detected based on the comparison results; if the defect detection network is configured with multiple convolutional neural network modules of different scales, then configuring different receptive fields for each of the convolutional neural network modules as input parameters of the corresponding convolutional neural network module to adapt to surface abnormality features of different sizes on the object to be detected.
[0016] The confidence of the surface feature of the object to be detected is calculated and compared with a preset threshold, and the surface abnormal feature of the object to be detected is determined according to the comparison result, including: calculating a first similarity score and a second similarity score of the output surface feature according to the defect detection network; the first similarity score is the distribution result of the mean of the feature vector output from the feature extraction model in the trained q convolutional neural network modules around the feature vector output from the main feature extraction model, and is expressed by the formula:
[0017]
[0018] The second similarity score is the variance distribution result of the feature vector output from the feature extraction model in the trained q convolutional neural network modules, and is expressed as
[0019]
[0020] Wherein, each of the convolutional neural network modules includes a feature extraction network and a pre-training network constituting the main feature extraction model; represents the second feature vector output from the feature extraction model in the kth convolutional neural network module, represents the mean of the feature vectors output from the feature extraction model in the q convolutional neural network modules, represents the first feature vector output by the feature extraction network in the main feature extraction model, represents the vector mean of the feature vector output by the feature extraction network, σ represents the vector variance of the feature vector output by the feature extraction network, r and c represent the row and column pixel coordinates of the image respectively, and |||| represents the norm operation; the confidence of the surface feature of the object to be detected is calculated according to the first similarity score and the second similarity score, and is expressed as
[0021] s (r,c) =(s1+s2)*0.5;
[0022] The confidence level of the surface feature of the object to be detected is compared with a preset threshold value, and when the confidence level is greater than the threshold value, the surface feature is confirmed to be an abnormal surface feature of the object to be detected.
[0023] According to the third aspect, an embodiment provides a defect detection system, which includes at least one user terminal, each of which is used to perform defect detection on an object to be detected passing through a conveying channel; the user terminal includes a camera, a processor and a display; the camera is used to capture a detection image of the object to be detected and / or a reference image of a standard product of the object to be detected; the processor is connected to the camera to execute the abnormality detection method as described in the second aspect above to obtain the surface abnormality features of the object to be detected; the display is connected to the processor to display the detection image, reference image and / or surface abnormality features of the object to be detected.
[0024] The defect detection system also includes a construction unit for constructing a defect detection network, the construction unit is connected to the processor and the processor calls the defect detection network; the construction unit includes an acquisition module, a configuration module, a first training module and a second training module; wherein the acquisition module is used to acquire sample images in a preset open data set and a reference image of a standard product of the object to be detected; the configuration module is connected to the acquisition module and is used to configure a convolutional neural network module or multiple convolutional neural network modules of different scales to form a defect detection network; wherein each of the convolutional neural network modules includes a main feature extraction model and a secondary feature extraction model; the first training module is connected to the acquisition module and the configuration module; the first training module is used to use the sample image to extract the main feature of each convolutional neural network module. The main feature extraction model is trained to obtain corresponding network parameters; the configuration module is also used to input the reference image into the feature extraction network of the main feature extraction model in each convolutional neural network module to obtain the corresponding first feature vector, and input the reference image into the slave feature extraction model in each convolutional neural network module to obtain the corresponding second feature vector; the second training module is connected to the acquisition module and the configuration module, and is used to construct the loss function of the slave feature extraction model according to the first feature vector and the second feature vector, and obtain the network parameters of the slave feature extraction model through back propagation algorithm learning; the configuration module is also used to configure the defect detection network according to the network parameters of the main feature extraction model and the network parameters of the slave feature extraction model in each convolutional neural network.
[0025] According to the fourth aspect, an embodiment provides a computer-readable storage medium, including a program, which can be executed by a processor to implement the construction method as described in the first aspect above, and / or implement the anomaly detection method as described in the second aspect above.
[0026] The beneficial effects of this application are:
[0027] According to the defect detection network construction method, anomaly detection method and system, and storage medium of the above-mentioned embodiments, the defect detection network construction method includes: obtaining a sample image in a preset open data set and a reference image of a standard product corresponding to the object to be detected; configuring a convolutional neural network module or multiple convolutional neural network modules of different scales to form a defect detection network, wherein each of the convolutional neural network modules includes a main feature extraction model and a slave feature extraction model; using the sample image to train the main feature extraction model in each convolutional neural network module to obtain corresponding network parameters; inputting the reference image into the main feature extraction model and the slave feature extraction model in each convolutional neural network module to obtain the corresponding first feature vector and second feature vector respectively; constructing the loss function of the defect detection network according to the first feature vector and the second feature vector, and obtaining the network parameters of the slave feature extraction model through back propagation algorithm learning; forming a defect detection network according to the network parameters of the main feature extraction model and the network parameters of the slave feature extraction model in each convolutional neural network. First, since both training samples are obtained from open data sets and reference images of standard products are used as training samples, cross-domain migration training can be achieved to enrich and improve the performance of the defect detection network. Second, since each convolutional neural network module involved in the defect detection network includes a main feature extraction model and a slave feature extraction model, it is convenient to use sample images to train the main feature extraction model, input reference images and use the main feature extraction model to guide the training of the slave feature extraction model, so that the defect detection network with completed network parameter configuration has better feature expression and abstraction capabilities. Third, since the loss function of the slave feature extraction model is constructed with the help of the first eigenvector and the second eigenvector and the network parameters are learned, the advantages of unsupervised training are fully utilized to solve the problem of sample imbalance. Fourth, since the main feature extraction model is trained with sample images, The main feature extraction model has good feature extraction ability, ensuring that the feature vector extracted by the main feature extraction model is as close as possible to the feature vector extracted by the pre-trained network, thereby enhancing the feature expression ability of the main feature extraction model; fifthly, since different receptive fields can be configured for each convolutional neural network module in the defect detection network as input parameters of the corresponding convolutional neural network module, this can meet the requirements of different sample training, avoid overfitting of feature extraction, and improve the overall expression ability of the network; sixthly, the technical solution of the present application can complete the automatic extraction of features according to the input image information, effectively reduce the dependence on the experience of staff in the process of product defect detection, and has practical value; seventhly, the constructed defect detection network adopts a lightweight network structure design as a whole, which can not only extract stable feature representations, but also meet certain external environment changes, thereby enhancing the adaptability of the algorithm to the environment. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 This is a schematic diagram of the structure of the defect detection system in Example 1 of the present application;
[0029] Figure 2 is a schematic diagram of the structure of a user terminal;
[0030] Figure 3 This is a schematic diagram of the structure of the building unit in Example 2 of the present application;
[0031] Figure 4 The schematic diagram of the structure of the defect detection network constructed;
[0032] Figure 5 This is a flowchart of a method for constructing a defect detection network in Embodiment 3 of the present application;
[0033] Figure 6 To obtain the flow chart of network parameters corresponding to each main feature extraction model;
[0034] Figure 7 Flow chart for obtaining the first eigenvector and the second eigenvector;
[0035] Figure 8 To obtain the flow chart of network parameters corresponding to each feature extraction model;
[0036] Fig. 9 Schematic diagram of the principle of training the main feature extraction model;
[0037] Fig.10 A schematic diagram showing the principle of obtaining the first eigenvector and the second eigenvector using a reference image;
[0038] Fig.11 This is a flow chart of the anomaly detection method in Embodiment 4 of the present application;
[0039] Fig.12 A flow chart for determining the surface abnormality characteristics of an object to be detected;
[0040] Fig.13 It is a schematic diagram of the principle of obtaining surface abnormality features through detection;
[0041] Fig.14 This is a schematic diagram of the structure of the user terminal in Example 5 of the present application. DETAILED DESCRIPTION
[0042] The present invention is further described in detail below by specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments adopt associated similar element numbers. In the following embodiments, many detailed descriptions are for making the present application better understood. However, those skilled in the art can easily recognize that some features can be omitted in different situations, or can be replaced by other elements, materials, methods. In some cases, some operations related to the present application are not shown or described in the specification, this is to avoid the core part of the present application being overwhelmed by too much description, and for those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations according to the description in the specification and the general technical knowledge in the art.
[0043] In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various implementations. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a required sequence, unless otherwise specified that a certain sequence must be followed.
[0044] The serial numbers of the components in this document, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning. The "connection" and "coupling" mentioned in this application, unless otherwise specified, include direct and indirect connections (couplings).
[0045] The inventive concept of the technical solution of this application is to build a defect detection network with lightweight network characteristics, use open data sets and trained feature extraction networks as knowledge guides, and improve the representation ability and rapid training ability of the defect detection network. In the defect detection network, a multi-layer convolutional neural network module is used for feature comparison, and a loss function is established to measure the expression ability of features from multiple levels and scales, thereby enhancing the ability to extract abnormal features on the surface of objects, avoiding overfitting, and thus improving the stability and accuracy of surface defect detection.
[0046] Embodiment 1
[0047] Please refer to Figure 1 This embodiment discloses a defect detection system, which includes at least one user terminal 11. Each user terminal 11 is used to perform defect detection on an object to be detected passing through a transmission channel. Here, the user terminal 11 includes a camera 111, a processor 112, and a display 113.
[0048] In this embodiment, the user terminal 11 can be a detection device fixed or mounted on a product conveying channel, and can perform defect detection on each product passing through the detection area. In some cases, the defect detection system can include multiple user terminals 11, so as to be set on different conveying channels of the same type of products, or on conveying channels of different products. In addition, the user can set the configuration parameters of each user terminal 11 to adapt to the working mode of different products, and can also view the results of defect detection.
[0049] Among them, the camera 111 can be an imaging component used in imaging devices such as cameras and video cameras. The camera 111 is mainly used to capture the detection image of the object to be detected and / or the reference image of the standard product corresponding to the object to be detected. For the detection image, the processor 112 can drive the camera 111 to capture each object to be detected (such as a product); for the reference image, it is necessary to use the camera 111 to capture the standard product corresponding to the object to be detected multiple times, so as to obtain the reference image obtained by capturing different numbers of standard products of the same type in the external environment (such as different shooting angles, different lighting modes).
[0050] The processor 112 is connected to the camera 111 via a data bus, and performs defect detection processing on the detection image acquired by the camera 111, thereby obtaining the surface abnormality features of the object to be detected. The abnormality detection method used by the processor 112 during the defect detection process can refer to the content disclosed in the third embodiment of the present application, and will not be described in detail here.
[0051] The display 113 is connected to the processor 112 via a data bus, and the display 113 is mainly used to display the detection image, reference image and / or surface abnormality features of the object to be detected. It should be noted that the display 113 can be a display screen of any type and size, as long as it can display the detection image and / or defect features of the object to be detected under the driving action of the processor 112, and there is no specific limitation on the arrangement of the display interface and the image color.
[0052] In the present embodiment, the object to be detected can be a product on an industrial assembly line, a mechanical part in an object box, a tool on an operating table, etc., without specific limitation. Then, the detection image of the object to be detected is the image obtained by imaging equipment such as a camera / camcorder to the object to be detected. For example, for products on an assembly line, a detection image corresponding to each product can be formed by imaging with a camera. In addition, the reference image of the standard product corresponding to the object to be detected is the image obtained by imaging equipment such as a camera / camcorder to the standard product (an object that is considered to have no surface defect features and can be used as a standard part). For example, when defect detection is performed on products of the same class or batch, it is necessary to obtain the reference image obtained by imaging different numbers of standard products of the same class in an external environment (such as different shooting angles, different lighting modes).
[0053] In this embodiment, see Figure 1 The defect detection system 1 further includes a server 12, on which a construction unit 121 for constructing a defect detection network is configured. The construction unit 121 can be connected to the user terminal 11 through a gateway 13 for communication, so that the processor 112 in the user terminal 11 can retrieve the constructed defect detection network from the construction unit in the server 12. The defect detection network is used to process the detection image of the object to be detected, so as to output the surface abnormality features of the object to be detected.
[0054] In this embodiment, the construction unit 121 in the server 12 constructs a neural network structure related to the defect detection network, and needs to learn the network parameters through model training, so as to configure the obtained network parameters to each user terminal, so that the user terminal can normally perform defect detection on a certain type of object to be detected. In a specific embodiment, the construction unit 121 can train and learn the defect detection network through the following steps: obtain sample images in a preset open data set and a reference image of a standard product corresponding to the object to be detected; configure a convolutional neural network module or multiple convolutional neural network modules of different scales to form a defect detection network, wherein each convolutional neural network module includes a main feature extraction model and a slave feature extraction model; use the sample image to train the main feature extraction model in each convolutional neural network module to learn and obtain the corresponding network parameters; input the reference image into the main feature extraction model (specifically input into the feature extraction network in the main feature extraction model) and the slave feature extraction model in each convolutional neural network module to obtain the corresponding first feature vector and second feature vector respectively; construct the loss function of the slave feature extraction model based on the first feature vector and the second feature vector, and learn the network parameters of the slave feature extraction model through the back propagation algorithm; configure the defect detection network according to the network parameters of the main feature extraction model and the network parameters of the slave feature extraction model in each convolutional neural network.
[0055] The functions of the construction unit 121 may be referred to the contents in the second embodiment, and will not be described in detail here.
[0056] It should be noted that, since the network parameters include the weight information of the network nodes, they can be used to configure and update the defect detection network preset on the user terminal. In this way, the server 12 sends the obtained network parameters to the processor 112 to configure and update the defect detection network preset on the user terminal 11. It can be understood that the server 12 can also assist each user terminal in learning the network model in a distributed manner, allowing decentralized processing of data types of the defect detection network.
[0057] See also Figure 1 In order to meet the needs of smooth communication between multiple user terminals with the same configuration (such as user terminal 11, user terminal N1, user terminal N2, etc.) and the server 12, the defect detection system 1 is also provided with a gateway 13. The gateway 13 is arranged between the server 12 and each user terminal to realize network interconnection and communication between different communication protocols and different systems.
[0058] It should be noted that the defect detection system of the present application implements the network training and object detection processes on the server and user terminal respectively, which is conducive to the unified training processing of the server and obtaining the network parameters of the objective function, thereby configuring and updating the preset defect detection network on each user terminal, thereby enhancing the versatility of the defect detection algorithm, and does not require customers to select and design features according to different detection objects, as long as the server performs unified design. In addition, in the process of training the neural network structure, the back propagation algorithm is used to update the weight information, which is conducive to obtaining appropriate segmentation weights according to the learning data of each category, and learning similarity measurement and corresponding features from the data.
[0059] In another embodiment, see Figure 2 The user terminal 11 includes not only a camera 111, a processor 112 and a display 113, but also a video adapter 114, a network adapter 115, a memory 116 and a device interface 117, which are described as follows.
[0060] The video adapter 114 is provided between the camera 111 and the processor 112, and communicates with the processor 112 through a data bus. The video adapter 114 is used to perform image adaptation processing on the detection image captured by the camera according to the video frame to meet the image reception requirements of the processor 112. It can be understood that since the processor 112 performs image processing according to a preset defect detection network, and the defect detection network has strict requirements on the input scale of the image (such as pixel size), it is necessary to set the video adapter 114 to perform scale conversion on each frame of the captured photo or video to meet the image reception requirements of the processor 112 according to the input scale of the defect detection network.
[0061] The network adapter 115 is connected to the processor 112 through a data bus, and is also connected to the external gateway 13. Figure 1 and Figure 2 The network adapter 115 is used to adapt the data communication needs between the processor 112 and the server 12. It can upload the reference image of the standard product corresponding to the object to be detected to the server 12, and can also send the network parameters processed by the server 12 to the processor 112.
[0062] The memory 116 is connected to the processor 112 via a data bus, and is used to store the detection image and the reference image received by the processor 112, and / or store the surface abnormality features of the object to be detected obtained by the processor 112. In addition, the memory 116 can also store the program corresponding to the preset defect detection network, and provide it to the processor 112 for execution.
[0063] The device interface 117 is connected to the processor 112 via a data bus. The device interface 117 is used to connect a keyboard or mouse of an external device, and to perform user interaction control with the processor 112 with the help of the external device, such as switching the display mode on the display 113, viewing the storage content in the memory 116, manually changing the network parameters of the defect detection network in the memory 116, and performing other interactive operations with the display 113.
[0064] It should be noted that Figure 1 The user terminal N1 and the user terminal N2 may have the same hardware structure and software functions as the user terminal 11, and may also be freely configured according to actual needs of the user, which is not specifically limited here.
[0065] Embodiment 2
[0066] Please refer to Figure 1 and Figure 3 The construction unit 121 in the server 12 may specifically include an acquisition module 1211, a configuration module 1212, a first training module 1213 and a second training module 1214, which are described below respectively.
[0067] The acquisition module 1211 is used to obtain sample images in a preset open data set and reference images of standard products corresponding to the object to be detected. The open data set can be a data set such as ImageNet, COCO, PASCAL VOC, etc., preferably the ImageNet data set; the open data set can be set on the cloud or a specific server, and the sample images therein can be retrieved only by accessing the network. The reference images of the standard products corresponding to the object to be detected can be pre-stored in the user terminal or server, and the stored reference image data can be directly read.
[0068] The configuration module 1212 is connected to the acquisition module 1211, and is used to configure a convolutional neural network module or multiple convolutional neural network modules of different scales to form a defect detection network; each convolutional neural network module includes a main feature extraction model and a secondary feature extraction model. It should be noted that if the configuration module 1212 configures multiple convolutional neural network modules of different scales for the defect detection network, it is also necessary to configure different receptive fields for each convolutional neural network module as the input parameter of the corresponding convolutional neural network module to adapt to the surface abnormal features of different sizes on the object to be detected. Figure 3 , the first training module 1213 is connected to the acquisition module 1211 and the configuration module 1212. The first training module 1213 is mainly used to train the main feature extraction model in each convolutional neural network module using sample images to obtain corresponding network parameters. In the process of training to obtain the network parameters of the main feature extraction model, the first training module 1213 establishes a feature extraction network and a pre-training network constituting the main feature extraction model for the main feature extraction model in each convolutional neural network module; uses part of the sample images in the open data set as normal samples, uses other sample images randomly selected outside the normal samples as negative samples, and adds Gaussian noise to the normal samples to obtain corresponding positive samples; inputs the normal samples, positive samples and negative samples into the feature extraction network and the pre-training network together, and guides the feature extraction network to be trained through the pre-training network, thereby obtaining the network parameters corresponding to the main feature extraction model.
[0069] See also Figure 3, the second training module 1214 is connected to the acquisition module 1211 and the configuration module 1212. The main function of the second training module 1214 is to use the main feature extraction model to guide the training of the slave feature extraction model. In order to realize the training process of the slave feature extraction model guided by the main feature extraction model, the configuration module 1212 is also required to cooperate here. The configuration module 1212 inputs the reference image into the main feature extraction model and the slave feature extraction model in each convolutional neural network module, and obtains the corresponding first feature vector and second feature vector respectively; then, the second training module 1214 constructs the loss function of the slave feature extraction model according to the first feature vector and the second feature vector, and obtains the network parameters of the slave feature extraction model through the back propagation algorithm.
[0070] In this embodiment, see Figure 3 , the first training module 1213 can train to obtain the network parameters of the main feature extraction model, and the second training module 1214 can train to obtain the network parameters of the secondary feature extraction model. Then the configuration module 1212 can configure the network parameters of the main feature extraction model and the secondary feature extraction model in each convolutional neural network to form a defect detection network, and at this time, it can be considered that the defect detection network is constructed.
[0071] In a specific embodiment, the first training module 1213 trains and learns to obtain corresponding network parameters by constructing a loss function of the main feature extraction model. The loss function of the main feature extraction model is expressed as
[0072] L(M)=w1L1+w2L2+w3L3;
[0073] Among them, M is the identifier of the main feature extraction model, w1≥0, w2≥0, w3≥0 and they are all weight coefficients.
[0074] L1 is used to represent the distance between the normal sample, positive sample and negative sample corresponding to the sample image, and L1 = max(d1-d2,0). The distance between the normal sample and the positive sample is d1 = ||uu + ||2, the distance between normal samples, positive samples and negative samples is d2=(||uu - ||2+||u + -u - ||2)*0.5;u,u + 、u - They represent the feature vectors output by normal samples, positive samples, and negative samples after passing through the feature extraction network, respectively. max() represents the maximum value operation, and |||| represents the norm operation.
[0075] L2 is used to characterize the similarity of feature vectors between different samples. For the case where there are m normal samples as input, the feature vector output by the feature extraction network is n-dimensional, and each output feature vector is recorded as v i (i=1,2,…,m), the eigenvectors form a matrix A n×m =[v1,v2,…,v m ], where the similarity between feature vectors is C n×n =AA T , and the element is represented by c ij , then i and j are the indices of the feature vector.
[0076] L3 is used to characterize the difference between feature vectors, and L3 = ||xy|| 2. x is the feature vector extracted from feature vector u through the fully connected layer, and y is the feature vector extracted from the normal sample through the pre-trained network.
[0077] It can be understood that after obtaining the loss function L(M), the first training module 1213 can use the loss function L(M) to allow the pre-trained network to guide the feature extraction network for training, and obtain the network parameters corresponding to the main feature extraction model.
[0078] In a specific embodiment, the second training module 1214 can train and learn to obtain the corresponding network parameters by constructing a loss function of the slave feature extraction model. If the configuration module 1212 inputs the reference image into the feature extraction network of the main feature extraction model in each convolutional neural network module, and into the slave feature extraction model in each convolutional neural network module, the corresponding first feature vectors can be obtained respectively. and the second eigenvector Then the loss function of the feature extraction model constructed by the second training module 1214 is expressed as
[0079]
[0080] Among them, M is the identifier of the main feature extraction model, and S is the identifier of the slave feature extraction model. It represents the vector mean of the feature vector output by the reference image after passing through the feature extraction network, σ represents the vector variance of the feature vector output by the reference image after passing through the feature extraction network, k is the traversal number of the convolutional neural network module, r and c represent the row and column pixel coordinates of the image respectively, |||| represents the norm operation, and diag() represents the diagonal matrix construction function.
[0081] After obtaining the loss function L(S i ) then the second training module 1214 can be trained according to the loss function L(S i) performs the back propagation algorithm to learn the network parameters of the feature extraction model.
[0082] It should be noted that the back propagation algorithm, also known as the BP algorithm, is a learning algorithm suitable for multi-layer neural networks. The input-output relationship of the BP network is essentially a mapping relationship: the function performed by a BP neural network with n inputs and m outputs is a continuous mapping from n-dimensional Euclidean space to a finite field in m-dimensional Euclidean space, and this mapping is highly nonlinear; the information processing ability of the relevant algorithm comes from the multiple compounding of simple nonlinear functions, so it has a strong function reproduction ability. The back propagation algorithm mainly consists of two links (excitation propagation and weight update) that are repeatedly iterated until the network's response to the input reaches the predetermined target range. For the excitation propagation link, each iteration contains two steps: the forward propagation stage in which the training input is sent into the network to obtain the excitation response, and the back propagation stage in which the excitation response is subtracted from the target output corresponding to the training input to obtain the response error of the hidden layer and the output layer. For the weight update link, the weight on each synapse can be updated as follows: multiply the input stimulus and the response error to obtain the gradient of the weight, multiply this gradient by a ratio, negate it, and add it to the weight. Since this ratio will affect the speed and effect of the training process, it is called the "training factor", and the direction of the gradient indicates the direction of error expansion, so it needs to be negated when updating the weight to reduce the error caused by the weight.
[0083] In a specific embodiment, see Figure 4 , three convolutional neural network modules of different scales (such as CNN_1, CNN_2, and CNN_3) can be configured, and each convolutional neural network module is configured with a different receptive field as the input parameter of the corresponding convolutional neural network module to adapt to the surface abnormal features of different sizes on the object to be detected. It can be understood that in order to ensure the defect detection effect of different sizes, a three-scale network structure design is adopted, in which each scale uses a different receptive field as the input parameter of the control network structure; then, large defective objects are highlighted on a large scale, and small defective objects are highlighted on a small scale. Based on this integrated training, the comprehensive judgment of the defect detection results is completed.
[0084] It should be noted that the receptive field can be considered as a local area of the image. It is like a window on the input layer. The hidden layer neurons of the convolutional neural network module learn to analyze the features within the image field of view. In the CNN learning and training process, instead of processing the entire image at once, the image is first divided into multiple small blocks. The kernel only looks at one block of the image at a time. This small block is called a patch (i.e. the size of the receptive field), and then moves to another patch of the image, and so on.
[0085] Embodiment 3
[0086] In order to clearly understand the construction process of the defect detection network, a construction method of a defect detection network will be disclosed in this embodiment. The construction method can be applied to a server or a user terminal and constitute a functional module of the server or the user terminal (such as the construction unit 121 in Embodiment 1). It can be understood that the functional module corresponding to the construction method needs to be executed by the processor to realize the corresponding function.
[0087] Please refer to Figure 5 The method for constructing a defect detection network for which protection is sought includes steps S110-S160, which are described below respectively.
[0088] Step S110 , obtaining a sample image in a preset open data set and a reference image of a standard product corresponding to the object to be detected.
[0089] It should be noted that the open datasets can be ImageNet, COCO, PASCAL VOC and other datasets, preferably ImageNet datasets; the open datasets can be set up in the cloud or on a specific server, and sample images can be retrieved by simply accessing the network. The reference images of the standard products corresponding to the objects to be detected can be pre-stored on the user terminal or server, and the stored reference image data can be directly read.
[0090] Step S120, configuring a convolutional neural network module or multiple convolutional neural network modules of different scales to form a defect detection network, wherein each convolutional neural network module includes a main feature extraction model and a secondary feature extraction model.
[0091] In this embodiment, see Figure 4 ,For CNN network structures of different scales, they all include a main feature extraction network model and a slave feature extraction model, where the network structures of the main feature extraction model and the slave feature extraction model can be consistent or inconsistent. For example, the main feature extraction model and the slave feature extraction model can generate multiple groups of different network structures according to different settings of the receptive field, forming integrated training and judgment, thereby improving the adaptability and stability to defects of different sizes.
[0092] Step S130, using the sample image to train the main feature extraction model in each convolutional neural network module to obtain corresponding network parameters.
[0093] In this embodiment, the feature extraction network and the training network are configured to form a main feature extraction model. The main feature extraction model is trained by using the trained feature extraction network backbone as a knowledge guide. The network inputs three different types of data, normal sample X, positive sample X + and negative samples X -, where normal samples X are some sample images in the open dataset, positive samples X + Image after adding Gaussian noise to normal sample, negative sample X - The images are randomly selected from other sample images other than normal samples. The pre-trained network is a network trained with a large data set. It can be a part of a classification network or a target detection network. The pre-trained feature extraction data can be intercepted from these networks as the input of the pre-trained network.
[0094] Step S140, input the reference image into the main feature extraction model and the secondary feature extraction model in each convolutional neural network module to obtain the corresponding first feature vector and second feature vector respectively.
[0095] Step S150, constructing a loss function of a feature extraction model according to the first feature vector and the second feature vector, and obtaining network parameters of the feature extraction model through back propagation learning.
[0096] In order to determine whether there are surface abnormal features of defect types on the surface of the object to be inspected, it is necessary to use a slave feature extraction model to extract the features of the object to be inspected. This part of the training is trained using defect-free samples of the object to be inspected (i.e., reference images). During the training process, the main feature extraction model is used to guide the training of the slave feature extraction model. Since the main feature extraction model can already express the feature information well, the features output by the slave feature extraction model should be as close as possible to the features output by the main feature extraction model. Therefore, the distance between the output features can be used as the loss function.
[0097] Step S160, forming a defect detection network according to the network parameters of the main feature extraction model and the network parameters of the slave feature extraction model in each convolutional neural network.
[0098] In this embodiment, the above step S130 mainly involves the process of using the sample image to train the main feature extraction model in each convolutional neural network module and obtain the corresponding network parameters, so please refer to Figure 6 , the step S130 may specifically include steps S131-S133, which are described as follows respectively.
[0099] Step S131, for the main feature extraction model in each convolutional neural network module, establish a feature extraction network and a pre-training network that constitute the main feature extraction model.
[0100] See also Fig. 9 For the main feature extraction model shown in the figure, the feature extraction network and pre-training network are respectively used F means.
[0101] In a specific embodiment, the feature extraction network It may include multiple convolutional layers (Conv), multiple maximum pooling layers (MaxPool) and fully connected layers. The specific parameters of each layer can be found in Table 1.
[0102] Table 1 Parameters of feature extraction network
[0103]
[0104] Step S132: taking part of the sample images in the open data set as normal samples, taking other sample images randomly selected from the normal samples as negative samples, and adding Gaussian noise to the normal samples to obtain corresponding positive samples. Fig. 9 For example, taking a capsule as a reference object, the normal sample and positive sample of the capsule are denoted by X and X respectively. + The negative samples of objects outside the capsule (such as the captured image of the car) are represented by the symbol X - To express.
[0105] In step S133, normal samples, positive samples and negative samples are input into the feature extraction network and the pre-training network, and the feature extraction network is trained by guiding the pre-training network, so as to obtain the network parameters corresponding to the main feature extraction model.
[0106] It should be noted that see Fig. 9 , the network trained based on the open dataset already has good feature extraction and expression capabilities, so the feature extraction network The extracted feature vector x is as close as possible to the feature vector y extracted by the pre-trained network F, so that the main feature extraction model has a good feature extraction ability. In addition, in order to ensure that the main feature extraction model can meet the requirements of small sample training and avoid overfitting of feature extraction, the feature vector x is output by the previous layer. This can improve the network's expressiveness. Fig. 9 Three loss functions can be introduced for evaluation, and the loss functions are represented by L1, L2, and L3 respectively.
[0107] In a specific embodiment, please refer to Fig. 9 , normal sample X, positive sample X + and negative samples X - Common input feature extraction network and pre-trained network F, and guide the feature extraction network through the pre-trained network F Train to obtain the network parameters corresponding to the main feature extraction model.
[0108] See also Fig. 9, we can use the loss function L1 to characterize the normal sample X and the positive sample X + and negative samples X - The distance between the normal sample X and the positive sample X is L1 = max(d1-d2,0). + The distance between them is d1=||uu + ||2, normal sample X, positive sample X + With negative samples X - The distance between them is d2=(||uu - ||2+||u + -u - ||2)*0.5,u,u + 、u - Represent normal samples X and positive samples X respectively. + , negative sample X - After feature extraction network The output feature vector is max(), which represents the maximum value operation, and |||| represents the norm operation.
[0109] It should be noted that the loss function L1 is used to characterize normal samples X and positive samples X + and negative samples X - The distance between the normal sample X and the positive sample X + They should be close to each other, because slight environmental changes or the introduction of noise will not affect the stability of feature extraction; normal sample X, positive sample X + and negative samples X respectively - The greater the difference in the features obtained, the better, which means that the feature extraction network has good expression and resolution capabilities.
[0110] See also Fig. 9 The loss function L2 is used to characterize the similarity of feature vectors between different samples. For the case where there are m normal samples as input, after the feature extraction network The output feature vector is n-dimensional, and each output feature vector is recorded as v i (i=1,2,…,m), the matrix composed of eigenvectors is represented as A n×m =[v1,v2,…,v m ], where the similarity between feature vectors is C n×n =AA T , and the element is represented by c ij , then i and j are both indices of feature vectors (such as the index of the feature vector of each image patch generated in the current batch sample image).
[0111] It should be noted that the loss function L2 is mainly used to measure the similarity of feature vectors between different images, that is, after the images are extracted, the feature extraction should be different, that is, the correlation between them should be relatively small. i (i=1,2,…,n) as input, we can select a batch of samples to calculate the similarity between their features. Suppose that after the feature extraction network The dimension of the feature vector obtained is n, and the number of batches is m. Therefore, the correlation of the m n-dimensional feature vectors can be calculated, and the components of the corresponding dimensions of the m n-dimensional feature vectors can be reconstructed. The obtained feature vector can be recorded as v i (i=1,2,…,m), where the similarity of the feature vector can be expressed as C n×n =AA T .
[0112] See also Fig. 9 , the loss function L3 is used to characterize the difference between feature vectors, and L3 = ||xy||2, x is the feature vector extracted by the feature vector u through the fully connected layer, and y is the feature vector extracted by the normal sample X through the pre-trained network F.
[0113] When the loss functions L1, L2, and L3 are constructed, the loss function of the main feature extraction model is constructed and expressed as
[0114] L(M)=w1L1+w2L2+w3L3.
[0115] Among them, M is the identifier of the main feature extraction model, w1≥0, w2≥0, w3≥0 and they are all weight coefficients, and the weight coefficients can be freely set by the user according to actual needs.
[0116] In summary, the loss function L(M) is used to make the pre-trained network guide the feature extraction network to train, so as to obtain the network parameters of the main feature extraction model. The network parameters here include the weight information of each network node in the main feature extraction model, which is used to configure the update Figure 3 The main feature extraction model in the configuration module 1212.
[0117] It should be noted that in this embodiment, the main feature extraction model is trained using sample images, so that the main feature extraction model has good feature extraction capabilities, ensuring that the feature vector extracted by the main feature extraction model is as close as possible to the feature vector extracted by the pre-trained network, thereby enhancing the feature expression capability of the main feature extraction model.
[0118] In this embodiment, the above step S140 mainly involves the process of inputting the reference image into the main feature extraction model and the secondary feature extraction model in each convolutional neural network module and obtaining the corresponding first feature vector and second feature vector respectively. Figure 7 , the step S140 may specifically include steps S141-S142, which are described as follows respectively.
[0119] Step S141: for each main feature extraction model in the convolutional neural network module, the reference image is input into the feature extraction network of the main feature extraction model to obtain the corresponding first feature vector Where M is the identifier of the main feature extraction model, r and c represent the row and column pixel coordinates of the image respectively.
[0120] Step S142: for each convolutional neural network module, the reference image is input into the feature extraction model to obtain a corresponding second feature vector Where S is the identifier from the feature extraction model, r and c represent the row and column pixel coordinates of the image respectively.
[0121] It should be noted that step S141 and step S142 are merely separate explanations for obtaining the first eigenvector and the second eigenvector, and there is no order of processing. The second eigenvector may be obtained first and then the first eigenvector, or the first eigenvector and the second eigenvector may be obtained simultaneously by parallel processing.
[0122] It should be noted that in this embodiment, training samples are obtained from open data sets, and reference images of standard products are also used as training samples, so that cross-domain migration training can be achieved, enriching and improving the performance of the defect detection network; in addition, since each convolutional neural network module involved in the defect detection network includes a main feature extraction model and a secondary feature extraction model, it is convenient to use sample images to train the main feature extraction model and use reference images to train the secondary feature extraction model, so that the defect detection network with completed network parameter configuration has better feature expression and abstraction capabilities.
[0123] In this embodiment, the above step S150 mainly involves the process of constructing a loss function of the feature extraction model and learning the network parameters of the feature extraction model through the back propagation algorithm. Figure 8 , the step S150 may specifically include steps S151-S152, which are described as follows respectively.
[0124] Step S151, through the first feature vector and the second eigenvector Construct the loss function from the feature extraction model and express it as
[0125]
[0126] in, It means that the reference image is passed through the feature extraction network (such as Fig. 9 Feature extraction network in ), σ represents the vector variance of the feature vector output after the reference image passes through the feature extraction network, w and h are the width and height of the reference image, respectively, and diag() represents the diagonal matrix construction function. Since the sample is input to the feature extraction network in the form of an image patch, It can also be considered as the vector mean of the feature vectors output by different image patches after passing through the feature extraction network.
[0127] It should be noted that since the main feature extraction model can already express the feature information well, the features output from the feature extraction model should be as close as possible to the features output by the main feature extraction model. Therefore, the distance between the output features can be used as the loss function.
[0128] Step S152: According to the loss function L(S i ) executes the back propagation algorithm to learn the network parameters of the feature extraction model, where the network parameters include the weight information of each network node in the feature extraction model.
[0129] It should be noted that the back propagation algorithm is mainly composed of two links (excitation propagation and weight update) that are repeatedly iterated until the network's response to the input reaches the predetermined target range. For the excitation propagation link, each iteration contains two steps: the forward propagation stage in which the training input is sent to the network to obtain the excitation response, and the back propagation stage in which the excitation response is subtracted from the target output corresponding to the training input to obtain the response error of the hidden layer and the output layer. For the weight update link, the weight on each synapse can be updated as follows: multiply the input excitation and the response error to obtain the gradient of the weight, multiply this gradient by a ratio and negate it and add it to the weight, and the direction of the gradient indicates the direction of error expansion, so it needs to be negated when updating the weight to reduce the error caused by the weight.
[0130] Step S160, the defect detection network is formed according to the network parameters of the main feature extraction model and the network parameters of the slave feature extraction model in each convolutional neural network. It can be understood that the defect detection network is configured with a convolutional neural network module or multiple convolutional neural network modules of different scales, and each convolutional neural network module includes a main feature extraction model and a slave feature extraction model. Then, after obtaining the network parameters of the main feature extraction model and the slave feature extraction model in each convolutional neural network, it is equivalent to obtaining the network parameters of each convolutional neural network module. In this case, it is easy to configure and form a defect detection network.
[0131] In another embodiment, after the defect detection network is configured, it also includes a feature determination step, which is mainly used to calculate the confidence of the surface features of the object to be detected and compare it with a preset threshold. For the feature determination step, please refer to the relevant content in Example 4.
[0132] It should be noted that see Fig.10 In order to train the slave feature extraction model, the reference image can be used as input and transmitted to the main feature extraction model (specifically, the feature extraction network input into the main feature extraction model) and the slave feature extraction model in the form of an input image, so as to obtain the first feature vector and the second feature vector respectively. The corresponding network parameters are obtained by training the slave feature extraction model, and then the defect detection network can be formed in the form of network configuration. After the defect detection network is configured, the confidence of the surface features of the object to be detected is calculated and compared with the preset threshold to determine whether the surface features belong to the surface abnormality features. Generally, the farther the features of the abnormal data and the features of the normal data are, the better. Since the main feature extraction model can express the features well, it is hoped that the features of the normal data extracted from the feature extraction model are as close to the main feature extraction model as possible, and the features of the abnormal data extracted from the feature extraction model are as far away from the main feature extraction model as possible. This comprehensive judgment process can be achieved by comparing the confidence and the threshold.
[0133] In this embodiment, see Figure 4For the constructed defect detection network, the main feature extraction model and the slave feature extraction model corresponding to each convolutional neural network module can generate multiple groups of different network structures according to the different settings of the receptive field, form integrated training and judgment, so as to improve the adaptability and stability of defects of different sizes. In order to improve the feature expression ability of the main feature extraction model and reduce the capacity of the network, this embodiment uses a pre-trained network as a priori knowledge to guide the feature extraction network to be trained. At the same time, in order to improve the network expression ability, feature representation loss functions are added to other layers of the network to achieve the application effect of multiple loss functions working together. In some cases, the main feature extraction model and the slave feature extraction model can use the same logical structure, but the weights are not shared. In order to complete the detection of defects of different sizes, different receptive fields are used to train multiple groups of network structures to achieve integrated training and comprehensive judgment. The input image blocks are used to represent the receptive field sizes of different sizes. The main feature extraction model is trained using an open data set, and the slave feature extraction model is trained using the reference image of the object to be detected.
[0134] Those skilled in the art can understand that this embodiment uses the first eigenvector and the second eigenvector to construct the loss function of the defect detection network and learn the network parameters, making full use of the advantages of unsupervised training and learning, and can solve the problem of sample imbalance. In addition, the technical solution of this application can complete the automatic extraction of features based on the input image information, effectively reducing the dependence on the experience of staff in the process of product defect detection, and has practical value; moreover, the constructed defect detection network adopts a lightweight network structure design as a whole, which can not only extract stable feature representations, but also meet certain external environment changes, thereby enhancing the adaptability of the algorithm to the environment.
[0135] Embodiment 4:
[0136] Based on the defect detection network constructed in the second embodiment, the surface of the object to be detected can be detected with the help of the defect detection network, so that the surface abnormality characteristics of the object to be detected can be obtained. Therefore, this embodiment discloses a method for detecting abnormalities on the surface of an object, which is mainly used on the user terminal in the first embodiment and is executed by the processor in the user terminal to realize the abnormality detection function.
[0137] In this embodiment, please refer to Fig.11 The method for detecting anomalies on the surface of an object for which protection is requested includes steps S210-S230, which are described below respectively.
[0138] Step S210, obtaining a detection image of the object to be detected.
[0139] The objects to be inspected can be products on an industrial assembly line, mechanical parts in an object box, tools on an operating table, etc., without specific limitation. Then, the inspection image of the object to be inspected is the image obtained by an imaging device such as a camera / camcorder when taking an image of the object to be inspected. For example, for products on an assembly line, the inspection image corresponding to each product can be formed by taking an image with a camera. Figure 1 In the process, the processor 112 can drive the camera 111 to capture an image of the object to be detected passing through the transmission channel, thereby obtaining a detection image of the object to be detected. The detection image can be acquired by the processor 112 and subjected to detection processing.
[0140] Step S220: Input the detection image into a preset defect detection network to process and obtain the surface features of the object to be detected. The defect detection network here can be constructed by the construction unit in the second embodiment. Fig.13 After the inspection image is input into the preset defect detection network, the feature extraction capability of the defect detection network is used to process the image and output the surface features of the object to be inspected. It should be noted that the surface features here are the embodiment of the surface texture, convexity and concavity and other information of the object to be inspected, but whether it is a texture defect or a convexity and concave defect or other abnormal features needs to be further determined through the feature determination step.
[0141] In this embodiment, if the defect detection network is configured with multiple convolutional neural network modules of different scales, a different receptive field is configured for each convolutional neural network module as an input parameter of the corresponding convolutional neural network module to adapt to surface abnormality features of different sizes on the object to be detected.
[0142] In a specific embodiment, the preset defect detection network can be used as follows Figure 4 The network structure is shown in . Three convolutional neural network modules of different scales are configured: CNN_1, CNN_2, and CNN_3. Each convolutional neural network module is configured with a different receptive field as the input parameter of the corresponding convolutional neural network module. Then it can be understood that after the detection image is input into the defect detection network, large defective objects can be highlighted on a large scale, and small defective objects can be highlighted on a small scale. Based on this integrated training, the comprehensive judgment of the defect detection results can be well completed.
[0143] Step S230, calculating the confidence of the surface feature of the object to be detected and comparing it with a preset threshold, and determining the abnormal surface feature of the object to be detected according to the comparison result.
[0144] In a specific embodiment, see Fig.12 , step S230 may specifically include steps S231-S233, which are described as follows respectively.
[0145] Step S231 , calculating a first similarity score and a second similarity score of the output surface feature according to the defect detection network.
[0146] The first similarity score is the distribution result of the mean of the feature vectors output from the feature extraction model in the trained q convolutional neural network modules around the feature vector output from the main feature extraction model, and is expressed as
[0147]
[0148] The second similarity score is the variance distribution result of the feature vector output from the feature extraction model in the trained q convolutional neural network modules, and is expressed as
[0149]
[0150] Among them, each convolutional neural network module includes a feature extraction network and a pre-training network that constitute the main feature extraction model; in the formula represents the second feature vector output from the feature extraction model in the kth convolutional neural network module, represents the mean of the feature vectors output from the feature extraction model in the q convolutional neural network modules, represents the first feature vector output by the feature extraction network in the main feature extraction model, represents the vector mean of the feature vector output by the feature extraction network, σ represents the vector variance of the feature vector output by the feature extraction network, r and c represent the row and column pixel coordinates of the image respectively, and || || represents the norm operation.
[0151] Step S232, the confidence of the surface feature of the object to be detected is calculated according to the first similarity score and the second similarity score, and is expressed as
[0152] s (r,c) =(s1+s2)*0.5.
[0153] Step S233, compare the confidence of the surface feature of the object to be detected with a preset threshold value, and confirm that the surface feature is a surface abnormality feature of the object to be detected when the confidence is greater than the threshold value. It should be noted that the preset threshold value can be a data close to 100%, such as 90%, and when the confidence of the surface feature is greater than 90%, the surface feature is considered to be a surface abnormality feature.
[0154] Embodiment 5
[0155] Based on the method for constructing a defect detection network disclosed in the third embodiment and the method for detecting anomalies on the surface of an object disclosed in the fourth embodiment, this embodiment discloses a user terminal, see Fig.14The user terminal 3 specifically includes a memory 31 and a processor 32.
[0156] The memory 31 stores programs.
[0157] The processor 32 is used to implement a method for constructing a defect detection network and / or a method for detecting anomalies on the surface of an object by executing a program stored in the memory 31 .
[0158] In this embodiment, the memory 31 can be regarded as a computer-readable storage medium, which includes a program that can be executed by a processor to implement the defect detection network construction method disclosed in Example 3, and / or implement the object surface abnormality detection method disclosed in the above-mentioned Example 4.
[0159] In a specific embodiment, when the processor 32 executes the program to implement the method for constructing a defect detection network, the execution process can be described as: (a) obtaining a sample image in a preset open data set and a reference image of a standard product corresponding to the object to be detected; (b) configuring a convolutional neural network module or multiple convolutional neural network modules of different scales and forming a defect detection network, wherein each convolutional neural network module includes a main feature extraction model and a secondary feature extraction model; (c) using the sample image to train the main feature extraction model in each convolutional neural network module to learn and obtain the corresponding network parameters; (d) inputting the reference image into the main feature extraction model (specifically inputting into the feature extraction network of the main feature extraction model) and the secondary feature extraction model in each convolutional neural network module to obtain the corresponding first feature vector and second feature vector respectively; (e) constructing the loss function of the secondary feature extraction model according to the first feature vector and the second feature vector, and obtaining the network parameters of the secondary feature extraction model through the back propagation algorithm; (f) configuring the defect detection network according to the network parameters of the main feature extraction model and the network parameters of the secondary feature extraction model in each convolutional neural network. For details, please refer to steps S110-S160 in Example 3, which will not be repeated here.
[0160] In a specific embodiment, when the processor 32 executes the relevant program to implement the method for detecting anomalies on the surface of an object, the execution process can be described as: (a) obtaining a detection image of the object to be detected; (b) inputting the detection image into a preset defect detection network to obtain the surface features of the object to be detected; (c) calculating the confidence of the surface features of the object to be detected and comparing it with a preset threshold, and determining the surface abnormal features of the object to be detected according to the comparison result. The specific process can be referred to steps S210-S230 in the fourth embodiment, which will not be repeated here.
[0161] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above-mentioned embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above-mentioned functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above-mentioned functions can be implemented. In addition, when all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and can be downloaded or copied and saved in the memory of the local device, or the system of the local device is updated, and when the program in the memory is executed by the processor, all or part of the functions in the above-mentioned embodiments can be implemented.
[0162] The above specific examples are used to illustrate the present invention, which is only used to help understand the present invention and is not intended to limit the present invention. For those skilled in the art, according to the concept of the present invention, some simple deductions, modifications or substitutions can be made.
Claims
1. A method for constructing a defect detection network, characterized in that: include: Obtain sample images from a preset open data set and reference images of standard products of the object to be detected; Configure a convolutional neural network module or multiple convolutional neural network modules of different scales to form a defect detection network, wherein each of the convolutional neural network modules includes a main feature extraction model and a secondary feature extraction model; Using the sample image to train the main feature extraction model in each convolutional neural network module, and learning to obtain corresponding network parameters; Inputting the reference image into the main feature extraction model and the secondary feature extraction model in each of the convolutional neural network modules to obtain corresponding first feature vectors and second feature vectors respectively; Constructing a loss function of the feature extraction model according to the first feature vector and the second feature vector, and obtaining network parameters of the feature extraction model through back propagation algorithm learning; The defect detection network is formed according to the network parameters of the main feature extraction model and the network parameters of the slave feature extraction model in each of the convolutional neural networks.
2. The construction method according to claim 1, characterized in that: The main feature extraction model in each convolutional neural network module is trained using the sample image to learn and obtain corresponding network parameters, including: For each main feature extraction model in the convolutional neural network module, a feature extraction network and a pre-training network constituting the main feature extraction model are established; Some sample images in the open data set are used as normal samples, other sample images randomly selected from the normal samples are used as negative samples, and Gaussian noise is added to the normal samples to obtain corresponding positive samples; The normal sample, the positive sample and the negative sample are input into the feature extraction network and the pre-trained network, and the feature extraction network is guided by the pre-trained network for training, so as to obtain the network parameters corresponding to the main feature extraction model.
3. The construction method according to claim 2, characterized in that: The normal sample, the positive sample and the negative sample are input into the feature extraction network and the pre-trained network, and the feature extraction network is trained by guiding the pre-trained network to obtain network parameters corresponding to the main feature extraction model, including: The loss function of the main feature extraction model is constructed and expressed as L(M)=w1L1+w2L2+w3L3; Among them, M is the identifier of the main feature extraction model, w1≥0, w2≥0, w3≥0 and they are all weight coefficients; L1 is used to characterize the distance between the normal sample, the positive sample and the negative sample, and L1 = max(d1-d2,0), the distance between the normal sample and the positive sample is d1 = ||uu + ||2, the distance between the normal sample, the positive sample and the negative sample is d2=(||uu - ||2+||u + -u - ||2)*0.5,u,u + 、u - respectively represent the feature vectors outputted by the normal sample, the positive sample, and the negative sample after passing through the feature extraction network, max() represents the maximum value operation, and || || represents the norm operation; L2 is used to characterize the similarity of feature vectors between different samples. For the case where there are m normal samples as input, the feature vector output by the feature extraction network is n-dimensional, and each output feature vector is recorded as v i (i=1,2,…,m), the matrix composed of eigenvectors is represented as A n×m =[v1,v2,…,v m ], where the similarity between feature vectors is C n×n =AA T , and the element is represented by c ij , then i and j are the indices of the feature vector; L3 is used to characterize the difference between feature vectors, and L3=||xy||2, where x is the feature vector extracted by the feature vector u through the fully connected layer, and y is the feature vector extracted by the normal sample through the pre-trained network; The loss function L(M) is used to allow the pre-trained network to guide the feature extraction network for training, thereby obtaining network parameters corresponding to the main feature extraction model.
4. The construction method according to claim 1 or 2, characterized in that: Inputting the reference image into the main feature extraction model and the secondary feature extraction model in each of the convolutional neural network modules to obtain the corresponding first feature vector and second feature vector respectively, including: For each main feature extraction model in the convolutional neural network module, the reference image is input into the feature extraction network in the main feature extraction model to obtain the corresponding first feature vector Where M is the identifier of the main feature extraction model, r and c represent the row and column pixel coordinates of the image respectively; For each secondary feature extraction model in the convolutional neural network module, the reference image is input into the secondary feature extraction model to obtain a corresponding second feature vector Where S is the identifier from the feature extraction model.
5. The construction method according to claim 4, characterized in that: Constructing a loss function of the feature extraction model according to the first feature vector and the second feature vector, and obtaining network parameters of the feature extraction model through back propagation algorithm learning, including: By the first eigenvector and the second eigenvector The loss function of the feature extraction model is constructed and expressed as in, represents the vector mean of the feature vector output by the reference image after passing through the feature extraction network of the main feature extraction model, σ represents the vector variance of the feature vector output by the reference image after passing through the feature extraction network of the main feature extraction model, w and h are the width and height of the reference image respectively, and diag() represents a diagonal matrix construction function; According to the loss function L(S i ) executes the back propagation algorithm to learn and obtain the network parameters of the feature extraction model.
6. A method for detecting anomalies on the surface of an object, characterized in that: include: Obtaining a detection image of the object to be detected; Inputting the detection image into a defect detection network obtained by the construction method according to any one of claims 1 to 5 to obtain the surface features of the object to be detected; Calculating the confidence of the surface feature of the object to be detected and comparing it with a preset threshold, and determining the abnormal surface feature of the object to be detected according to the comparison result; If the defect detection network is configured with multiple convolutional neural network modules of different scales, a different receptive field is configured for each convolutional neural network module as an input parameter of the corresponding convolutional neural network module to adapt to surface abnormality features of different sizes on the object to be detected.
7. The abnormality detection method according to claim 6, characterized in that: The calculating the confidence of the surface feature of the object to be detected and comparing it with a preset threshold, and determining the abnormal surface feature of the object to be detected according to the comparison result, comprises: calculating a first similarity score and a second similarity score of the output surface features according to the defect detection network; The first similarity score is the distribution result of the feature vectors output from the feature extraction model in the trained q convolutional neural network modules around the feature vector output by the main feature extraction model, and is expressed by the formula: The second similarity score is the variance distribution result of the feature vector output from the feature extraction model in the trained q convolutional neural network modules, and is expressed as Wherein, each of the convolutional neural network modules includes a feature extraction network and a pre-training network constituting the main feature extraction model; represents the second feature vector output from the feature extraction model in the kth convolutional neural network module, represents the mean of the feature vectors output from the feature extraction model in the q convolutional neural network modules, represents the first feature vector output by the feature extraction network in the main feature extraction model, represents the vector mean of the feature vector output by the feature extraction network of the main feature extraction model, σ represents the vector variance of the feature vector output by the feature extraction network of the main feature extraction model, r and c represent the row and column pixel coordinates of the image respectively, and || || represents the norm operation; The confidence of the surface feature of the object to be detected is calculated according to the first similarity score and the second similarity score, and is expressed as s (r,c) =(s1+s2)*0.5; The confidence level of the surface feature of the object to be detected is compared with a preset threshold value, and when the confidence level is greater than the threshold value, the surface feature is confirmed to be an abnormal surface feature of the object to be detected.
8. A defect detection system, characterized in that: It includes at least one user terminal, each of which is used to perform defect detection on an object to be detected passing through the transmission channel; the user terminal includes a camera, a processor and a display; The camera is used to capture a detection image of the object to be detected and / or a reference image of a standard product of the object to be detected; The processor is connected to the camera and is used to execute the anomaly detection method according to any one of claims 6 to 7 to obtain the surface abnormality features of the object to be detected; The display is connected to the processor and is used to display the detection image, the reference image and / or the surface abnormality features of the object to be detected.
9. The defect detection system according to claim 8, characterized in that: It also includes a construction unit for constructing a defect detection network, the construction unit is connected to the processor and the processor calls the defect detection network; the construction unit includes: An acquisition module, used to acquire sample images in a preset open data set and reference images of standard products of the object to be detected; A configuration module, connected to the acquisition module, configured to configure a convolutional neural network module or a plurality of convolutional neural network modules of different scales to form a defect detection network; wherein each of the convolutional neural network modules includes a main feature extraction model and a secondary feature extraction model; A first training module connected to the acquisition module and the configuration module; the first training module is used to train the main feature extraction model in each convolutional neural network module using the sample image to obtain corresponding network parameters; The configuration module is also used to input the reference image into the feature extraction network of the main feature extraction model in each of the convolutional neural network modules to obtain a corresponding first feature vector, and input the reference image into the slave feature extraction model in each of the convolutional neural network modules to obtain a corresponding second feature vector; A second training module, connected to the acquisition module and the configuration module, is used to construct a loss function of the feature extraction model according to the first feature vector and the second feature vector, and obtain network parameters of the feature extraction model through back propagation algorithm learning; The configuration module is also used to form the defect detection network according to the network parameters of the main feature extraction model and the network parameters of the slave feature extraction model in each of the convolutional neural networks.
10. A computer-readable storage medium, characterized in that: The method comprises a program which can be executed by a processor to implement the construction method as claimed in any one of claims 1 to 5, and / or to implement the abnormality detection method as claimed in any one of claims 6 to 7.
Citation Information
Patent Citations
Image searching method and device, storage medium and electronic device
CN109815355A
Casting defect identification method based on convolutional neural network
CN110222681A
