Target detection method, device, electronic device, storage medium and application thereof

Abstract

A target detection method, device, electronic device, storage medium and application thereof, the method comprising: obtaining a support sample x s and query sample x q Two image samples as input; support sample x s is a normal sample without defects, and the query sample x q is the sample to be tested; two backbone networks with the same structure and shared weights respectively support the sample x s and query sample x q Perform feature extraction to obtain the corresponding feature map G w (x s ) and G w (x q );The feature map G w (x s ) and G w (x q ) Input the feature enhancement network to obtain the enhanced and / or suppressed feature maps v s and v q ; Based on the feature matching network, the enhanced and / or suppressed feature maps v are respectively s and v q Perform similarity measurement and output the measurement result H(v s , v q ); the measurement result H(v s , v q ) Input the YOLO layer module for regression calculation, and predict the query sample x based on the regression calculation results q The method provided by the present invention amplifies small sample training data, improves generalization performance, enhances real-time detection speed and can better perform position detection.

Description

Technical Field

[0001] The present invention belongs to the field of industrial visual inspection technology, and in particular relates to a target detection method, a target detection device, an electronic device for target detection, a computer-readable storage medium, and an application of the target detection method in texture surface defect detection. Background Art

[0002] Machine vision is the AI ​​technology most closely integrated with industrial applications. Machine vision analyzes image data captured by sensors to perform tasks such as image classification and object location, and then feeds the results back to the appropriate equipment for subsequent operations. Surface defect detection plays a crucial role in the field of machine vision. The rapid development of deep learning, exemplified by convolutional neural networks (CNNs), has led to an increasing number of deep learning models being applied to surface defect detection.

[0003] However, in the aforementioned industrial scenarios, texture-based surface defect detection faces numerous challenges, such as ever-changing lighting environments, a wide variety of defect types, large variations in defect sizes, image noise caused by camera shake, and interference from the image background, which can significantly impact texture-based surface defect detection. At the same time, training data samples are scarce, and the cost of producing defective samples is too high. Consequently, the number of defective samples is relatively small, making it difficult for general deep learning models to learn the characteristics of defective images from a small number of samples. The generalization performance of deep learning-based surface defect detection methods is low. Once the style or category of the object to be inspected changes and differs significantly from the training samples, it is difficult for the model to guarantee effective detection. Deep learning models have a slow detection speed, making them difficult to apply in scenarios where real-time detection is highly demanding. Furthermore, existing methods can only classify whether an image contains defects, but cannot identify the specific location of the defects.

[0004] Therefore, it is still necessary and urgent to develop and design a target detection method that can ensure single-sample learning of texture surface defects, and apply it to amplified training samples to improve the generalization performance of deep learning surface defect detection methods, and to be able to classify whether an image has defects in real time while also identifying the specific location of texture surface defects. Summary of the Invention

[0005] The purpose of the present invention is to provide a target detection method, device, electronic device, storage medium and application thereof, so as to solve the technical problems of small number of training samples, low generalization performance of surface defect detection method, poor real-time detection and inaccurate defect detection location when detecting texture surface defects.

[0006] The purpose of the present invention and the solution to the technical problem are achieved by adopting the following technical solutions.

[0007] The first aspect of the present invention proposes a target detection method, comprising the following steps: obtaining a support sample x s and query sample x q As two image samples of input; wherein, the support sample x s is a normal sample without defects, the query sample x q is the sample to be tested; two backbone networks with the same structure and shared weights respectively support the sample x s and query sample x q Perform feature extraction to obtain the corresponding feature map G w (x s ) and G w (x q );The feature map G w (x s ) and G w (x q ) Input the feature enhancement network to obtain the enhanced and / or suppressed feature maps v s and v q ; Based on the feature matching network, the enhanced and / or suppressed feature maps v are respectively s and v q Perform similarity measurement and output the measurement result H(v s ,,v q ); the measurement result H(v s , v q ) Input the YOLO layer module to perform regression calculation, and predict the query sample x based on the regression calculation result q Defect location and / or confidence in .

[0008] Preferably, the two backbone networks with the same structure and shared weights respectively support the sample x s and query sample x q Perform feature extraction to obtain the corresponding feature map G w (x s ) and G w (x q ) includes: forming the backbone network by YOLO-fastest and denoted as Gw; transforming the support sample x s and query sample x q Input the input end of the backbone network respectively for feature extraction; obtain the corresponding feature map G w (x s ) and G w (x q ).

[0009] Preferably, the feature map G w (x s ) and G w (x q ) Input feature enhancement network to obtain the enhanced and / or suppressed feature maps v s and v q Including: constructing the feature enhancement network based on the improved Non-Local attention mechanism; combining the two feature maps G w (x s ) and G w (x q Input the input end of the feature enhancement network; enhance the features with strong correlation and suppress the features with weak similarity; output the enhanced and / or suppressed feature maps v respectively through the feature enhancement network s and v q ; Wherein, the feature graph G w (x s ) has a dimension of w s *h s *c,G w (x q ) has a dimension of w q *h q *c,h s , h q Represents the feature map G w (x s ) and G w (x q ) height dimension, w s , w q Should represent the feature map G w (x s ) and G w (x q ) is the width size, c is the feature map G w (x s ) and G w (x q ) channels.

[0010] Preferably, the method of mutually enhancing the features with strong correlation and suppressing the features with weak similarity comprises: w (x s ) are convolved through two point-by-point convolutional networks to make the feature map G w (x s ) space size remains unchanged, the number of channels is reduced by half, and the results are recorded as g(G w (x s ))and The feature map G w (x q) are convolved through two point-by-point convolutional networks to make the feature map G w (x q ) space size remains unchanged, the number of channels is reduced by half, and the results are recorded as g(G w (x q )) and θ(G w (x q ));Will and θ(G w (x q )) are reconstructed into two two-dimensional matrices respectively; the two two-dimensional matrices are matrix multiplied to obtain a dimension of w q h q *w s h s The matrix of dimension w q h q *w s h s The network layer composed of the matrix input softmax function completes the similarity calculation; the output dimension is w q h q *w s h s The matrix of g(G w (x s )) and g(G w (x q )) is reconstructed into a matrix; the matrix is ​​matrix-multiplied with the output of the softmax function, and the dimensions are and Two matrices of and The two matrices are reconstructed into dimensions and Two feature maps of and The two feature maps of are respectively input into two point-by-point convolutional networks for channel dimension increase; the results of channel dimension increase are respectively compared with G w (x s ) and G w (x q ) add them together to get v s and v q .

[0011]

[0012]

[0013] in, W θ , and These are all linear transformation coefficients, i, j represent the characteristics of the map The i-th or W θ G W The jth element, Represents the similarity calculation function of the above feature graph, v s The dimension is w s *h s *c,v q The dimension is w q *h q *c.

[0014] Preferably, the The similarity calculation function uses the radial basis function to calculate the similarity between two vectors, as shown below:

[0015] Among them, W θ G W (x s ) i , Represent two column vectors respectively.

[0016] Preferably, the feature matching network is used to respectively perform the enhanced and / or suppressed feature maps v s and v q Perform similarity measurement and output the measurement result H(v s , v q ) includes: the dimensions are w s *h s *c's respective enhanced and / or suppressed feature maps v s and w q *h q *c's respective enhanced and / or suppressed feature maps v q Input feature matching network; for v s w s *h s vectors of dimension c*1 and v q w q *h q Vectors with a dimension of c*1 are combined with each other and the similarity is calculated according to the similarity calculation formula; the dimension is w q *h q *(w s h s c) similarity feature graph; the similarity calculation formula is:

[0017] similarity(v s，i , v q，j )=(v s，i -v q，j ) 2; Wherein, subscripts i and j represent v s The i-th vector and v q The jth vector of q *h q *(w s h s c) The similarity feature map is grouped and convolved to obtain a dimension of w q *h q *c similarity feature map; wherein the convolution kernel size of the group convolution is 1*1, the step size is 1, and the number of groups is c; the dimension is w q *h q * The similarity feature map of c and the respective enhanced and / or suppressed feature maps v q Along the w q *h q *c dimensions are spliced; the final output dimension of the feature matching network is w q *h q *(2c)The measurement result H(v s , v q ).

[0018] Preferably, the measurement result H(v s , v q ) Input the YOLO layer for regression calculation, and predict the query sample x based on the regression calculation result q The defect location and / or confidence in the measurement result H(v s , v q ) input into the YOLO layer for regression calculation; based on the loss function CIOU, a first error loss is obtained between the prediction result of the defect position and the true value label, and the query sample x is predicted based on the first error loss q and / or when predicting the confidence, obtaining a second error loss between the predicted confidence of the defect position and the true value label based on the ternary loss function in the twin network, and predicting the query sample x based on the second error loss q The confidence of the defect position in ; the ternary loss function is:

[0019] TripleLoss=∑max((1-y)y′+y(my′),0)

[0020] Where y is the label, 0 indicates no defect, and 1 indicates defect; y′ represents the confidence of the output of the YOLO layer, and the confidence value range is [0, 1]; m represents the edge margin, and m = 1.

[0021] Preferably, when predicting the defect position, the detection method further comprises: using a non-main maximum suppression (NMS) algorithm to filter out overlapping recognition results for the defect position, and outputting the position and confidence of the defect in the query sample xq.

[0022] The second aspect of the present invention provides a target detection device, which includes the following modules: a sample acquisition module for acquiring a support sample x s and query sample x q As two image samples of input; wherein, the support sample x s is a normal sample without defects, the query sample x q is the sample to be tested; the backbone network module is composed of two backbone networks with the same structure and shared weights, and the backbone networks are used to support the sample x s and query sample x q Perform feature extraction to obtain the corresponding feature map G W (x s ) and G w (x q ); Feature enhancement module, the feature map G w (x s ) and G w (x q ) Input the feature enhancement module to obtain the enhanced and / or suppressed feature maps v s and v q ; Feature matching module, based on the feature matching module, respectively, the enhanced and / or suppressed feature map v s and v q Perform similarity measurement and output the measurement result H(v s , v q ); YOLO layer module, the measurement result H(v s , v q ) Input the YOLO layer module to perform regression calculation, and predict the query sample x based on the regression calculation result q Defect location and / or confidence in .

[0023] The third aspect of the present invention proposes an electronic device, which includes: a memory for storing non-temporary computer-readable instructions; and a processor for running the computer-readable instructions, so that when the computer-readable instructions are executed by the processor, the target detection method described in any one of claims 1 to 8 is implemented.

[0024] A fourth aspect of the present invention provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on a device, the device executes the target detection method described above.

[0025] A fifth aspect of the present invention proposes an application of the above-mentioned target detection method in texture surface defect detection.

[0026] Compared with the prior art, the present invention has obvious advantages and beneficial effects. By means of the above technical solution, the present invention has at least the following advantages and beneficial effects:

[0027] First, the present invention employs a twin network structure, ensuring that the input to the target detection method is not a single image sample, but rather a pair of two image samples. This structure allows for a larger number of training samples, thereby addressing the difficulty faced by deep learning models in learning target defect image features from a small number of samples, due to the high cost of producing defective samples and the limited number of defective samples.

[0028] Second, the present invention uses a metric learning method in a twin network to improve the generalization performance of the model by learning the similarities and differences between the corresponding features of input samples. This metric learning method based on the twin network of the present invention improves the generalization performance of the target detection method based on single-sample learning in texture-based surface defect detection.

[0029] Third, this invention integrates and transforms the YOLOv3 target detection method and model, and uses regression on deep feature maps to predict the location and confidence of target defects. This invention proposes a designed feature enhancement network and feature matching network, which can better integrate the YOLOv3 detection method and backbone network module under the framework of the twin network. This can not only classify whether an image has defects, but also identify the specific location of the target defect, and has a good application effect in texture surface defect detection.

[0030] Fourth, on the one hand, the present invention adopts a lightweight network YOLO-fastest from the open source community as the backbone network in the present invention. Compared with the classic YOLOv3 based on the DarkNet backbone network, the backbone network used in the present invention has fewer parameters, reduces the complexity of calculation, and is more real-time. On the other hand, the present invention realizes a single-stage single-sample learning target detection method and model by integrating and transforming the YOLOv3 model. Compared with the existing two-stage single-sample learning target detection method and model, the present invention has a simpler process in model training and reasoning, faster calculation speed, and enhanced real-time performance of target detection.

[0031] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following preferred embodiments are specifically cited and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 1 is a schematic diagram of the process structure of the single-sample learning target detection method according to an embodiment of the present invention;

[0033] Figure 2 is a schematic diagram of a data set used in an embodiment of the present invention;

[0034] Figure 3 is a schematic diagram of a method for constructing a data set in an embodiment of the present invention;

[0035] Figure 4 This is a technical framework flow chart of a single-sample learning target detection model according to an embodiment of the present invention;

[0036] Figure 5 Schematic diagram of the internal structure of the feature enhancement network in an embodiment of the present invention;

[0037] Figure 6 Schematic diagram of the internal structure of the feature matching network in an embodiment of the present invention;

[0038] Figure 7 This is a comparison of the detection effects of the embodiment of the present invention and the classic YOL0v3-yolofastest on the test set samples;

[0039] Figure 8 2 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the specific implementation methods, structures, features and effects proposed in accordance with the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0041] In industrial scenarios, texture-based surface defect detection based on deep learning faces many difficulties. (1) Small training data samples: The cost of manufacturing defective samples is too high, so the number of defective samples is small, which makes it difficult for general deep learning models to learn the characteristics of defective images from a small number of samples. (2) Low generalization performance: The surface defect detection method based on deep learning has limited generalization performance. Once the style or category of the object to be detected changes and differs greatly from the training samples, it is difficult for the model to ensure effective detection. (3) Poor real-time performance: The detection speed of deep learning models is slow, and it is difficult to apply to some scenarios with strong requirements for real-time detection. (4) Inaccurate position detection: Although there are currently related methods that can solve the above three problems, the existing methods can only classify whether the image has defects, but cannot identify the specific location of the defects.

[0042] With the rapid development of deep learning, represented by convolutional neural networks (CNNs), an increasing number of deep learning models are being applied to the field of surface defect detection for precision workpieces. Because deep learning learns from large amounts of data, it far outperforms traditional feature extraction solutions based on sample-based hand-crafted design when dealing with complex scenarios, such as small training data samples, low generalization performance, poor real-time performance, and poor position detection.

[0043] In order to solve the problems existing in the prior art in this field, the present invention applies the concept, device, electronic device and storage medium of the single-sample learning target detection method to the field of texture surface defect detection. The first aspect of the present invention proposes a single-sample learning target detection method that can detect defects on texture surfaces. The method includes the following steps: obtaining a support sample x s and query sample x q As two image samples of input; wherein, the support sample x s is a normal sample without defects, the query sample x q is the sample to be tested; two backbone networks with the same structure and shared weights respectively support the sample x s and query sample x q Perform feature extraction to obtain the corresponding feature map G w (x3) and G w (x q );The feature map G w (x s ) and G w (x q ) Input the feature enhancement network to obtain the enhanced and / or suppressed feature maps v s and v q ; Based on the feature matching network, the enhanced and / or suppressed feature maps v are respectively s and v q Perform similarity measurement and output the measurement result H(v s , v q ); the measurement result H(v s , v q ) Input the YOLO layer module to perform regression calculation, and predict the query sample x based on the regression calculation result q The target detection method of the present invention not only achieves efficient and rapid defect location detection, but also solves to some extent the problems of "small training data samples", "low generalization performance", "poor real-time performance" and "poor position detection effect" existing in the field of texture surface defect detection.

[0044] In view of the problems existing in the above technologies, the preferred embodiment of the present invention aims to propose a single-sample learning target detection method for defect detection on textured surfaces. The overall framework of the technical solution of the preferred embodiment of the present invention is to integrate and improve the target detection method model structure of the twin network and YOLOv3. Its framework structure and detection process, such as Figure 1 and Figure 4 The technical solution of the present invention specifically includes the following steps:

[0045] Step S1: Get support sample x s and query sample x q As two image samples of input; wherein, the support sample x s is a normal sample without defects, the query sample x q is the sample to be detected. Specifically, the target detection method model is a single-sample learning target detection method model, whose input is a sample pair consisting of two image samples. When performing texture surface defect detection, one input sample is a normal sample without defects, which is called the support sample and is denoted by x. s , the other sample is the sample to be tested, which is called the query sample and is recorded as x q After the two samples are respectively passed through the single-sample learning target detection method model, the location and confidence level of the defect in the sample to be detected are output. If the texture style of the sample to be detected changes, it is only necessary to replace the normal sample at the input with a normal sample with the same texture style.

[0046] Before performing single-sample learning for texture-based surface defect detection, a dataset for the target detection method of single-sample learning must first be constructed. The steps of constructing the dataset include S11 to S14:

[0047] S11: A dataset suitable for the model involved in the preferred embodiment of the present invention is constructed based on the German DAGM2007 dataset. The data in this dataset is an open source dataset created in 2007 for an industrial image processing competition. This dataset is artificially generated and contains 10 categories of texture styles, such as Figure 2 As shown in Figure 2, each dataset consists of 1,000 defect-free images and 150 defective images. Each defective image has a corresponding mask image to mark the defect location.

[0048] S12: Since the image size of the original dataset is relatively large, each image and its mask map are first divided into four equal parts, and then the coco format annotation of the corresponding defect image is regenerated based on the mask map. The coco format target frame after the four equal parts is processed by using the normalized center point horizontal coordinate, normalized center point vertical coordinate, normalized width and normalized height, as shown in the following example: Figure 3 shown.

[0049] S13: The defect-free samples and defective samples of each texture style are stored in two folders respectively, and then randomly paired with each other to construct sample pairs that meet the conditions.

[0050] S14: When creating sample pairs, this dataset randomly selects sample pairs consisting of 7 categories of samples as training sets and validation sets, and the remaining 3 categories of samples as test sets. During the pairing process, this dataset randomly matches each defective sample with a non-defective sample of the same type as a sample pair. When all defective samples are matched, it is said that one round of matching is completed. After dataset preprocessing, a total of 3032 defective samples were obtained from 10 categories of samples. For the training set, 70% of the defective samples were randomly selected, and 5 rounds of matching were performed for each category, resulting in a total of 90,120 sample pairs; for the validation set, the remaining 30% of the defective samples were used, and 1 round of matching was performed for each category, resulting in a total of 2,242 sample pairs; for the test set, 1 round of matching was performed for each category, resulting in a total of 1,119 sample pairs.

[0051] The above-mentioned design division of the training set and the validation set is to prevent samples in the validation set from appearing in the training set, so as to ensure that the validation set does not mix with the training set samples, thereby ensuring the credibility and effectiveness of the evaluation results. The preferred embodiment of the present invention can be implemented in the following three ways:

[0052] 1. Hold-out validation: When evaluating a method model, divide the data into a training set, a validation set, and a test set. (Suitable for large datasets)

[0053] 2. k-fold validation: Divide the training data into k partitions of equal size (suitable for small data sets, not adopted in the preferred embodiment of the present invention).

[0054] 3. (k-fold cross validation) Repeated k-fold validation with disrupted rhythm (small amount of available data set).

[0055] In a preferred embodiment of the present invention, in order to prevent overfitting, the following method is adopted:

[0056] (1) Reduce the method model specifications, that is, reduce the number of parameters learned by the method model (determined by the number of layers and the number of neurons in each layer); evaluate on the validation set to find the method model specifications corresponding to the best target detection method.

[0057] (2) Add weight regularization, that is, let the model weight corresponding to the target detection method take only a small value, thereby limiting the complexity of the method model corresponding to the target detection method; L1 regularization, L2 regularization and other methods can be used.

[0058] (3) Add random inactivation (dropout) regularization. Use dropout for a certain layer, that is, randomly discard some output features of the layer (i.e., set to 0) during training. The dropout ratio is the proportion of features set to 0, usually between 0.2 and 0.5.

[0059] The process of constructing the above dataset obtains a single-sample graphic dataset for defect detection on textured surfaces. The preferred embodiment of the present invention, on the one hand, constructs sample pairs, which can expand the training dataset and thus solve the problem of "small sample training data" in textured surface defect detection; on the other hand, by adopting the idea of ​​twin networks, the target detection method model with one image sample as input is improved to a sample pair consisting of two different image samples as input. By using multiple image samples at the input of the improved twin network, the number of training samples can also be expanded, thereby solving to a certain extent the problem that the target detection method model of the above deep learning is difficult to learn the target defect image features from a small number of samples.

[0060] S2, two backbone networks with the same structure and shared weights respectively support the sample x s and query sample x q Perform feature extraction to obtain the corresponding feature map G w (x s ) and G w (x q ). That is, the backbone network is composed of YOLO-fastest and is recorded as Gw; the support sample x s and query sample x q Input the input end of the backbone network respectively for feature extraction; obtain the corresponding feature map G w (x s ) and G w (x q ).

[0061] Specifically, the backbone network uses the classic YOLO, and further YOLO-fastest can be used, which is recorded as Gw. Here, just to abstract the network model into a function mapping, the variable can be omitted or written as G w(·). The network model YOLO-fastest is an open source network model that makes extensive use of depthwise separable convolutions. Compared with the classic YOLOv3 backbone network based on DarketNet, it greatly reduces the number of parameters in the backbone network and reduces the computational complexity. YOLO-fastest focuses on single-core real-time inference performance, with low CPU usage while meeting real-time conditions. It can not only achieve real-time performance on mobile phones, but also meet certain real-time requirements on RK3399, Raspberry Pi 4, and various Cortex-A53 low-cost and low-power devices. The above embedded devices are much weaker than mobile phones, which makes YOLO-fastest more widely used. And because it is backward compatible with low-spec devices, it improves computing speed and is more cost-effective. The above S2 includes steps S21 to S22 as follows.

[0062] S21, set the model hyperparameters of the backbone network: set the image size to 256*256, the batch size (batchsize) to 32, the number of model weight updates in each epoch (epoch) to 40 times, and the backbone network model uses the adaptive moment estimation Adam optimizer; the adaptive moment estimation Adam optimizer combines the advantages of the AdaGrad and RMSProp optimization algorithms, comprehensively considering the first moment estimation (i.e., the mean value of the gradient) and the second moment estimation (i.e., the uncentered variance of the gradient) to calculate the update step size. The formula of the adaptive moment estimation Adam optimizer is:

[0063]

[0064] Where θ represents the parameter vector, represents the bias-corrected first-order moment estimate of the exponential shift calculation of the gradient mean, The exponential shift of the squared gradient computes the bias-corrected second raw moment estimate, θ t-1 represents the update step size at time t-1, θ t represents the update step size at time t, α represents the default learning rate, and α=0.001, ε=10 -8 Avoid divisors becoming 0. From the above expression, it can be seen that the updated step size calculation can be adaptively adjusted from two perspectives: the gradient mean and the gradient square, rather than being directly determined by the current gradient.

[0065] The adaptive moment estimation Adam optimizer described above has parameter updates that are unaffected by scaling of the gradient; the update step size can be limited to approximately the range of the initial learning rate; the hyperparameters are well interpretable and typically require little or no fine-tuning, and the learning rate can be automatically adjusted to implement a step-size annealing process. This makes it suitable for unstable objective functions with sparse gradients or those with significant gradient noise, and is particularly well-suited for single-sample learning objectives such as defect detection on textured surfaces with large-scale data and parameters. This adaptive moment estimation Adam optimizer is simple and convenient to implement, is computationally efficient when calculating the adaptive learning rate for each parameter, and requires little memory. The amount of error utilized in each parameter update is controlled by a parameter: the learning rate, also known as the step size. The learning rate (step size) is set to 0.001.

[0066] S22, input the sample pairs in the training set into the YOLO-fastest backbone network. s and x q Perform feature extraction and obtain the corresponding feature map G w (x s ) and G w (x q ). The backbone network adopts the idea of ​​weight sharing in the twin network, that is, the two samples x at the input s and x q , respectively, through two backbone networks with the same structure, s and x q Perform feature extraction and obtain the corresponding feature map G w (x s ) and G w (x q The weight sharing of the backbone network means that the two samples at the input end pass through two backbone networks with the same structure, and the weights of these two backbone networks are exactly the same.

[0067] In the main framework of the preferred embodiment of the present invention, as shown in the attached Figure 4 As shown in the figure, the idea of ​​the twin network is combined with the idea of ​​YOLOv3. The backbone network is a replaceable part, which can also include but is not limited to the backbone networks of various target detection method models of the YOLO series, such as MobileNet, ResNet, ShuffleNet and other image classification backbone network models, which will not be repeated here.

[0068] S3, the feature map G w (x s and G w (x q ) Input feature enhancement network to obtain the enhanced and / or suppressed feature maps vs and v q Including: constructing the feature enhancement network based on the improved Non-Local attention mechanism; combining the two feature maps G w (x s ) and G w (x q ) input into the input end of the feature enhancement network; mutually enhance the features with strong correlation, and suppress the features with weak similarity; output the enhanced and / or suppressed feature maps v respectively through the feature enhancement network s and v q ; Wherein, the feature graph G w (x s ) has a dimension of w s *h s *c,G w (x q The dimension is w q *h q *c;h s , h q Represents the feature map G w (x s ) and G w (x q ) height dimension, w s , w q Should represent the feature map G w (x s ) and G w (x q ) is the width size, c is the feature map G w (x s ) and G w (x q ) channels.

[0069] Specifically, the feature enhancement network employs an improved non-local attention mechanism. The so-called non-local attention mechanism stems from the fact that dependencies within and between frames are crucial in the field of computer vision, especially in dynamic video sequences. In particular, for behavior classification tasks, understanding global context and the connections between different frames play a significant role in guiding classification results. A common approach currently employed is to improve understanding of global context by increasing the receptive field through recurrent convolutional networks or deeper networks. Despite this, this approach remains relatively local, not considering temporal direction or spatial location. The major drawbacks of this approach are the inability to transfer long-distance information back and forth. Furthermore, the Ldeeper network is computationally intensive, inefficient, and difficult to optimize for gradients. Therefore, to address the issue of long-distance information transfer and improve long-range dependencies, a preferred embodiment of the present invention, starting with the traditional non-local mean filtering method, proposes a non-local approach within convolutional networks. Specifically, the response at a pixel is the sum of the feature weights at all other pixels, associating each pixel with all others, thus implementing the non-local concept.

[0070] Before the feature enhancement network adopts the improved Non-Local attention mechanism, it is first defined that the target to be detected in the query sample is a target belonging to the same category as the support sample. If there is a target to be detected in the query sample, the correlation between the features of the query sample feature map belonging to the target to be detected and the support sample feature map will be relatively strong; while the correlation between the features of the query sample feature map belonging to the background features or other non-targets to be detected and the support sample feature map will be relatively weak. The feature enhancement network is equivalent to considering the features with weaker correlation as noise and the features with stronger correlation as the target features of the target to be detected. The target features of the feature enhancement network are equivalent to classifying the background features with weaker correlation as noise and performing a suppression operation, while classifying the features of the target to be detected with stronger correlation as enhanced features to be detected and performing an enhancement operation. The above-mentioned improved Non-Local attention mechanism includes the following steps S31 to S35.

[0071] S31, such as Figure 5 As shown, according to the feature map G output by the backbone network output w (x s ) and G w (x q ), feature maps G are used at the two input ends of the feature matching network layer w (x s ) and G w (x q ), the dimensions are w s *h s *c and w q *h q*c, where w s , h s , w q , h q is the spatial size of the feature map, and c represents the number of channels of the feature map.

[0072] S32, said enhancing the features with strong correlation and suppressing the features with weak similarity comprises: w (x s ) are convolved through two point-by-point convolutional networks to make the feature map G w (x s ) space size remains unchanged, the number of channels is reduced by half, and the results are recorded as g(G w (x s ))and The feature map Gw(xq) is convolved through two point-by-point convolutional networks respectively, so that the feature map G w (x q ) space size remains unchanged, the number of channels is reduced by half, and the results are recorded as g(G w (x q )) and θ(G w (x q )).

[0073] Specifically, such as Figure 5 As shown, the attention mechanism uses the feature map G w (x s ) and G w (x q ) as input, PW represents point-by-point convolution network, that is, the convolution kernel is 1*1 and the step size is 1. G w (x s ) are respectively subjected to two PW convolutions to keep the spatial size of the feature map unchanged and reduce the number of channels by half. The two calculation results are Denoted as g(G w (x s ))and Similarly, G w (x q ) also undergoes the above two PW convolutions, making the spatial size of the feature map unchanged and the number of channels reduced by half. The two calculation results are Denoted as g(G w (x q )) and θ(G w (x q )). Since point-by-point convolution is equivalent to matrix operation on feature maps, the feature maps v that are enhanced and / or suppressed are s and v q As shown below:

[0074]

[0075]

[0076] In formulas (1) and (2), W θ , and These are all linear transformation coefficients, i, j represent the characteristics of the map The i-th or W θ G w The jth element, Represents the similarity calculation function of the above feature graph. s The dimension is w s *h s *c,v q The dimension is w q *h q *c. g(G w (x s )) is equivalent to the above formula Likewise, Equivalent to g(G w (x q )) is equivalent to θ(G w (x q )) is equivalent to W θ G w (x s ) i .

[0077] S33: In formulas (1) and (2) of step S32, it is necessary to calculate the inner product and similarity of each vector of the feature map. Figure 5 In the convolutional network, the vector inner product is completed by matrix multiplication. and θ(G w (x q )) are reconstructed into two two-dimensional matrices respectively; the two two-dimensional matrices are matrix multiplied to obtain a dimension of w q h q *w s h s The matrix of dimension w q h q *w s h s The network layer composed of the matrix input softmax function completes the similarity calculation; the output dimension is w q h q *w s h s The matrix of .

[0078] Specifically, such as Figure 5 As shown, and θ(G w (x q )) are input into the reconstruction function (reshape function), and the reshape function is used to and θ(G w (x q )) is converted into two two-dimensional matrices; and θ(G w (x q )) The two two-dimensional matrices converted are multiplied to obtain a dimension of w q h q *w s h s The matrix is ​​input into the softmax network layer for similarity calculation to complete the similarity calculation.

[0079] The calculation of the above correlation and the enhancement or suppression of different features based on the correlation are all calculated once in the formula. q *h q *w s *h s The real value vector (a1, a2, a3, a4...ai) of the feature matrix is ​​mapped into a (b1, b2, b3, b4...bi), where bi is a constant from 0 to 1. Then, it can be sorted according to the size of bi. For example, the average value bm of the above bi can be used as a limit. When the value of bi is greater than bm, the multi-dimensional bi with a weight greater than bm is taken to perform the enhancement task; at the same time, when the value of bi is less than bm, the one-dimensional bi with the smallest weight is taken to perform the weakening task. Through the above-mentioned enhancement and weakening tasks based on the weight of the feature vector, G w (x s ) for small targets in G w (x q ) is enhanced and G is weakened w (x s ) for small targets in the relevant features G w (x q ) in the noise; G w (x s ) for small targets in G w (x s ) is enhanced and G is weakened w (x s ) in the sample noise. The Softmax function does not change the dimension of the input matrix, so the matrix dimension of its output is still w q h q*w s h s .

[0080] S34: g(G w (x s )) and g(G w (x q )) is reconstructed into a matrix; the matrix is ​​matrix-multiplied with the output of the softmax function, and the dimensions are and Two matrices of and The two matrices are reconstructed into dimensions and Two feature maps.

[0081] Specifically, such as Figure 4 As shown, g(G w (x s )) and g(G w (x q )) are input into the reshape function respectively, and the reshape function is used to transform g(G w (x s )) and g(G w (x q )) is converted into a matrix and then respectively combined with the output w of the above softmax q *h q *w s *h s Perform matrix multiplication and get the dimensions as and Input these two matrices into the reshape function respectively, and use the reshape function to convert and Matrix conversion to dimension and Two feature maps.

[0082] The preferred embodiment of the present invention is to make the feature map G w (x s ) and G w (x q ) generate attention for each other and enhance the support sample x s The obtained feature map G w (x s ) for the query sample x q The generated feature map G w (x s ) in the query sample with strong correlation about the characteristics of the small target, by adopting the feature map G w (xq ) performs a dimensionality reduction transformation and transforms the reduced-dimensional feature map G w (x s ) for query samples with strong correlation about small targets.

[0083] S35: Note that the above formula uses a summation operation, while Figure 5 In the network, the summation is completed by matrix multiplication. Specifically, if Figure 5 As shown, the two feature maps obtained in the previous step are and Input two PW convolutions respectively to increase the number of channels, and increase the number of low-dimensional dimensions to Know In the two feature map matrices The number of channels is increased to c channels. In the preferred embodiment of the present invention, in order to make the feature map G w (x s ) and G w (x q ) generate attention for each other and enhance the support sample x s The obtained feature map G w (x s ) for the query sample x q The generated feature map G w (x s ) in the query sample with strong correlation about the characteristics of the small target, by adopting the feature map G w (x q ) performs a dimensionality-enhancing transformation and transforms the feature map G after dimensionality-enhancing w (x q ) in the supporting samples with weak correlation weaken the noise characteristics of small targets. Finally, the above-mentioned dimension-upgraded w s *h s *c and w q *h q *c, respectively with G w (x s ) and G w (x q ) add them together to get v s and v q .

[0084] Specifically, by transforming the dimension-upgraded w s *h s *c feature map and G w (x s ) feature maps are added so that G w (x s ) In the feature graph, about the dimension w s *h s*c (by removing the noise features in the support samples and query samples, the relevant corresponding features in the support samples and query samples are screened out), and by adding the relevant corresponding features screened out above to G w (x s ) feature map, so that G w (x s ) The feature map is enhanced by the feature enhancement network, which enables the features with strong correlation to be w (x s ) feature map, thereby further weakening the noise features with weak correlation. In order to further improve the signal-to-noise ratio of the input feature map between the support sample and the query sample, and input the enhanced feature map, respectively denoted as v s and v q .

[0085] In a preferred embodiment of the present invention, the The similarity calculation function uses the radial basis function to calculate the similarity between two vectors, as shown below:

[0086]

[0087] In formula (4), W θ G w (x s ) i , Represent two column vectors respectively. Since the linear operators in the above calculation process are all matrix operations, they can be replaced by convolutional networks, and the similarity function calculation process can be replaced by softmax networks. Finally, the calculation process of the above improved Non-Local attention mechanism can be replaced by the attached Figure 5 The neural network alternative shown.

[0088] In a preferred embodiment of the present invention, the above-mentioned feature enhancement network adopts an improved Non-Local attention mechanism and can also set up multiple groups, which are cascaded and connected respectively, to perform multi-level classification tasks based on the feature vector weights for sample noise and small target features, thereby further enhancing sample features and weakening sample noise. Finally, the appropriately enhanced and / or weakened feature map obtained after processing is output, and similar feature vector calculations are not repeated here. At the same time, the feature enhancement network of the preferred embodiment of the present invention is improved based on the Non-Local attention mechanism, in which the similarity metric function is a replaceable part. In the prior art, distance metric functions including but not limited to Euclidean norm and cosine distance can also be used to achieve the replacement of the similarity metric function in the preferred embodiment of the present invention.

[0089] The preferred embodiment of the present invention has the effect of designing a feature enhancement network structure based on an improved Non-Local attention mechanism, which can replace the calculation process of enhancing the target features of the target to be detected with basic neural networks and matrix operations, thereby facilitating the training and deployment of the entire model.

[0090] S4, the feature matching network is used to respectively perform the enhanced and / or suppressed feature maps v s and v q Perform similarity measurement and output the measurement result H(v s ,,v q ). Specifically, the feature matching network is a network model for calculating similarity designed in a preferred embodiment of the present invention, which aims to solve the similarity between the supporting sample feature graph and the query sample feature graph. The input layer of the feature matching network is the feature graph v s and v q , the dimensions are w s *h s *c and w q *h q *c, the output is the similarity between the two feature maps, denoted as H(v s , v q ), this step includes the following steps S41 to S43.

[0091] S41, the dimensions are w s *h s *c's respective enhanced and / or suppressed feature maps v s and w q *h q *c's respective enhanced and / or suppressed feature maps v q Input feature matching network. That is, when G w (x s ) The feature map is enhanced by the feature enhancement network, which enables the features with strong correlation to be w (x s ) feature map is further enhanced. Or when G w (x s ) The feature map enhances the network so that the features with weak correlation can be w (x s ) feature map is further suppressed and weakened. Or when G w (x s ) The feature map is enhanced by the feature enhancement network, which enables the features with strong correlation to be w (x s ) feature map is further enhanced, and when G w (x s ) The feature map enhances the network so that the features with weak correlation can bew (x s ) feature map is further suppressed and weakened. And finally the dimensions are w s *h s *c's respective enhanced and / or suppressed feature maps v s and w q *h q *c's respective enhanced and / or suppressed feature maps v q Input feature matching network.

[0092] S42, for v s w s *h s vectors of dimension c*1 and v q w q *h q Vectors with a dimension of c*1 are combined with each other and the similarity is calculated according to the similarity calculation formula; the dimension is w q *h q *(w s h s c) similarity feature graph; the similarity calculation formula is:

[0093] similarity(v s，i , v q，j )=(v s，i -v q，j ) 2 (5)

[0094] Wherein, the subscripts i and j represent v s The i-th vector and v q The jth vector of .

[0095] Specifically, such as Figure 5 As shown, put v s Split into or as w s *h s A vector of dimension c*1, similarly, v q Split into or as w q *h q A vector of dimension c*1. s The vector and v q The vectors are combined in pairs and the similarity between them is calculated one by one according to formula (5).

[0096] The calculation result of the similarity is still a vector. Specifically, Figure 5 As shown, the similarity vectors are finally arranged into a dimension of w q *h q *(w s h sc) Similarity feature map.

[0097] S43, the dimension w q *h q *(w s h s c) The similarity feature map is grouped and convolved to obtain a dimension of w q *h q *c similarity feature map; wherein, the convolution kernel size of the grouped convolution is 1*1, the step size is 1, and the number of groups is c.

[0098] Specifically, such as Figure 5 As shown in the figure, the similarity feature map is grouped and convolved to further extract and compress the features in the similarity feature map. The convolution kernel size is 1*1, the step size is 1, and the number of groups is c. The dimension of the feature map after convolution is w q *h q *c. The result obtained after convolution mainly contains the similarity between the target to be detected in the query sample and the support sample, and the position information of the target to be detected may be lost.

[0099] S43, sequentially transform the dimension w q *h q * The similarity feature map of c and the respective enhanced and / or suppressed feature maps v q Along the w q *h q *c dimensions are spliced; the final output dimension of the feature matching network is w q *h q *(2c)The measurement result H(v s , v q ). Specifically, Figure 5 As shown, the feature map obtained after the above convolution is combined with v q (w q *h q *c) The concat function is used to concatenate along the dimension of the number of channels to obtain the final output of the feature matching network, namely H(v s , v q ), whose dimension is w q *h q *(2c) Feature map.

[0100] In a preferred embodiment of the present invention, the structure of the network is as shown in the attached Figure 6 As shown, attached Figure 6 The similarity feature map is the result of similarity calculation, H(v s , v q ) refers to the final output result of the feature matching network, and is attached Figure 5The ws*hs*(wq*hq*c) in the formula refers to the dimension of the similarity feature map. In the feature matching network of the preferred embodiment of the present invention, the similarity measurement function is a replaceable part. For example, the technical solution of the preferred embodiment of the present invention includes but is not limited to distance measurement functions such as Euclidean norm and cosine distance; the convolutional network used therein is a replaceable part, which is not limited to convolutional networks or fully connected networks with different numbers of layers and different convolution kernel sizes. Generally, the metric learning method based on the twin network belongs to the category of single-sample learning method. Therefore, the preferred embodiment of the present invention improves the generalization performance of the single-sample learning target detection method model by learning the similarities and differences between the corresponding features of the input samples through the concept of the improved twin network metric learning.

[0101] S5, the measurement result H(v s , v q ) Input the YOLO layer module to perform regression calculation, and predict the query sample x based on the regression calculation result q Specifically, the regression network adopts the regression design of the YOLO layer, which is based on the design of the YOLO layer in YOLOv3, to calculate the defect position and / or confidence of H(v s , v q ) performs regression calculations, thereby predicting the defect location and confidence level in the query sample using the convolutional network in the YOLO layer. When predicting the defect location in the query sample, CIOU is used as the loss function. Using CIOU achieves a mAP of 49.21%, an increase of 1.5 percentage points compared to GIOU. CIOU(D) refers to the mAP evaluation of the validation model, replacing IOU with DIOU, which offers room for improvement. S5 includes steps S51 to S52.

[0102] S51, the normal sample without defects x s , the sample to be tested x q The above detection model is input, and the position and confidence of the sample to be detected output by the detection model are overlapped and filtered to obtain the recognition result. The above steps include the following steps S511 to S512.

[0103] S511, when performing texture surface defect detection, let x s is a normal sample without defects, let x q is the sample to be tested. After the model is applied, the location and confidence level of the defect in the sample to be tested are output.

[0104] The measurement result H(v s , v q) input into the YOLO layer for regression calculation; based on the loss function CIOU, a first error loss is obtained between the prediction result of the defect position and the true value label, and the query sample x is predicted based on the first error loss q The defect position in . When predicting the defect position, CIOU in YOLOv5 is used as the loss function to solve the first error loss between the prediction result and the true value label. The CIOU target box loss function used in the preferred embodiment of the present invention is completely borrowed from YOLOv5. As a replaceable part, it can be but not limited to other target box loss functions such as mean square error loss function, IOU loss function, smooth-L1, etc. to achieve the prediction of the defect position, but CIOU in YOLOv5 is used as the loss function. The CIOU loss function is more comprehensive in design. DIOU takes into account the center distance between the two detection frames. CIOU takes into account three geometric factors, namely: (1) overlapping area; (2) center point distance; (3) aspect ratio. Through comparative analysis, it can be seen that CIOU adds an aspect ratio information parameter compared with DIOU. Therefore, CIOU can add a penalty term for the aspect ratio, and the penalty term for the aspect ratio is a positive number used to measure the consistency of the aspect ratio of the feature map to be tested (v measures the consistency of aspect ratio).

[0105] If the width and height of the true box and the predicted box are similar, then the penalty term is 0, and the penalty term has no effect. Therefore, intuitively, the role of this penalty term is to control the width and height of the predicted box to be as close to the width and height of the true box as possible. Therefore, the CIOU loss function can control the width and height of the predicted box to make its loss as small as possible so that it can be as close to the width and height of the true box as quickly as possible, thereby quickly reducing the number of width and height selections of the predicted box, quickly determining the width and height selections of the predicted box, simplifying the amount of calculation, and further improving the calculation speed.

[0106] In actual detection, CIOU is more effective than GIOU in selecting objects. CIOU uses the GIOU loss function to find a more appropriate selection position. When a target is outside the selection frame, the CIOU loss function can accurately mark the target's position. Similarly, while GIOU completely marks the target, it cannot accurately select the target's outline. Therefore, using CIOU as the loss function is more appropriate for the selection position and accuracy of the detection frame than GIOU, and can better meet the needs of industrial high-precision detection.

[0107] When predicting the confidence, a second error loss between the predicted confidence of the defect position and the true value label is obtained based on the ternary loss function in the twin network, and the query sample x is predicted based on the second error loss. q The confidence of the defect position in ; the ternary loss function is:

[0108] TripleLoss=∑max((1-y)y′+y(my′),0)

[0109] Where y is the label, 0 indicates no defect, and 1 indicates defect; y′ represents the confidence of the output of the YOLO layer, and the confidence value range is [0, 1]; m represents the edge margin, and m = 1.

[0110] In the field of object detection, the cross-entropy loss function is used to determine the closeness between the actual output and the expected output. It measures the difference between two probability distributions for the same random variable. In machine learning, it is expressed as the difference between the true and predicted probability distributions. Smaller cross-entropy values ​​indicate better model predictions. The cross-entropy loss function addresses the slow gradient optimization problem associated with using the Mean Sequential Error (MSE) in logistic regression. However, for positive samples, the greater the output probability, the smaller the loss; for negative samples, the smaller the output probability, the smaller the loss. This loss function is slow when selecting a large number of simple samples and may not achieve satisfactory optimization results.

[0111] The Focal-Loss loss function adds an adjustment factor and a focusable parameter to the balanced cross-entropy loss, allowing the loss function to focus on training difficult samples. This function can reduce the weight of easily classified samples, allowing the model to focus more on quasi-classified samples during training, thereby adjusting the imbalance between positive and negative samples. Currently, there are two approaches to addressing the imbalance in the number of positive and negative samples: designing a sampling strategy, which generally involves resampling a small number of samples; and designing a Loss Function, which generally assigns weights to samples of different categories to address the serious imbalance in the ratio of positive and negative samples and the difficulty and easy samples in target detection. This loss function reduces the weight of a large number of simple negative samples in training, which has the advantage of mining difficult samples, but is insufficient for training and predicting simple samples.

[0112] The ternary loss function, TripletIoss, minimizes the distance between the anchor and positive samples with the same identity, and minimizes the distance between the anchor and negative samples with different identities. The goal of Tripletloss is to keep features of the same label as close as possible in space, while keeping features of different labels as far apart as possible. Furthermore, to prevent sample features from converging into a very small space, it requires that for two positive examples and one negative example of the same class, the negative example should be at least margin further away from the positive example. It can be seen that after learning with Tripletloss, the distance between positive examples of the same class and the anchor becomes closer, while the distance between negative examples of different classes and the anchor becomes farther away.

[0113] In a preferred embodiment of the present invention, after learning through the ternary loss function Tripletloss, the distance between similar Positive samples and Anchors becomes closer and closer, while the distance between different Negative samples and Anchors becomes farther and farther. It not only solves the problem that the cross entropy loss function will lead to deviations in the optimization direction of the model when the sample categories are unbalanced, but also overcomes the problem that the Focal-Loss loss function focuses on quasi-classified samples, has the advantage of mining difficult samples, but is insufficient in training and prediction of simple samples. At the same time, the preferred embodiment of the present invention adopts the ternary loss function as the confidence loss function, which avoids the dispersion of features of the same label in spatial positions and the aggregation of features of different labels in spatial positions, and at the same time prevents the features of the samples from being aggregated into a very small space.

[0114] In a preferred embodiment of the present invention, a ternary loss function is used as a confidence loss function. As a replaceable part, it can be but not limited to the cross entropy loss function, Focal-Loss loss function and other commonly used confidence loss functions in target detection or image classification can also be used to achieve confidence prediction, which will not be repeated here.

[0115] S512: When predicting the defect location, the detection method further includes: using a non-primary maximum suppression algorithm (NMS) to filter out overlapping recognition results for the defect location, and outputting the query sample x q The non-primary maximum suppression algorithm (NMS) is a widely used suppression method in the field of target detection.

[0116] In the YOLO layer structure of the preferred embodiment of the present invention, the design method and concept of the YOLO layer in YOLOv3 are adopted, and the position and confidence of the target are predicted by regressing the deep feature map. The feature enhancement network and feature matching network designed and proposed in the preferred embodiment of the present invention can better integrate the YOLO layer of YOLOv3 under the framework of the twin network. Compared with the currently replaceable technical solutions, it also includes but is not limited to the YOLO layer of various target detection models in the YOLO series. In the YOLO layer of the preferred embodiment of the present invention, the position and confidence of the target are predicted by regressing the deep feature map. The feature enhancement network and feature matching network proposed and designed in the preferred embodiment of the present invention predict the position and confidence of the target by regressing the deep feature map after processing. The prediction method and efficiency are simpler and more efficient, and the calculation method and concept of the YOLO layer in YOLOv3 can be better integrated under the framework of the twin network.

[0117] The preferred embodiment of the present invention also provides a target detection device for single sample learning, which includes the following modules: a sample acquisition module for acquiring a support sample x s and query sample x q As two image samples of input; wherein, the support sample x s is a normal sample without defects, the query sample x q is the sample to be tested. The backbone network module is composed of two backbone networks with the same structure and shared weights. The backbone networks support the sample x s and query sample x q Perform feature extraction to obtain the corresponding feature map G w (x s ) and G w (x q ). Feature enhancement module, the feature map G w (x s ) and G w (x q ) Input the feature enhancement module to obtain the enhanced and / or suppressed feature maps v s and v q The feature matching module is based on the feature matching module to respectively perform the enhanced and / or suppressed feature map v s and v q Perform similarity measurement and output the measurement result H(v s ,,v q ). YOLO layer module, the measurement result H(v s , v q ) Input the YOLO layer module to perform regression calculation, and predict the query sample x based on the regression calculation resultq Defect location and / or confidence in .

[0118] The backbone network module in a single-sample learning target detection device provided by a preferred embodiment of the present invention combines the twin network and the backbone network of YOLOv3; the design of the YOLO layer of the preferred embodiment of the present invention combines and integrates the design method of YOLOv3, but does not use the backbone network module of YOLOv3. Instead, a more lightweight open source backbone network module yolofastest is adopted. The yolofastest backbone network module of the preferred embodiment of the present invention is combined with the YOLOv3 target detection method model. The preferred embodiment of the present invention can use a more efficient target detection algorithm to detect the location of defects, achieving the technical effect of real-time detection of the location of surface defects.

[0119] The preferred embodiment of the present invention involves a target detection device for single-sample learning of defect detection. By adopting a designed feature enhancement module and feature matching module, and performing "feature enhancement" and "feature matching" on the target features under the framework of the twin network, it is more effectively combined with YOLOv3 and integrated into the YOLO layer for regression calculation. By combining the twin network structure, the input end of the target detection device for single-sample learning of defect detection is no longer a single image sample, but a sample pair consisting of two image samples. In this way, the preferred embodiment of the present invention can expand the number of training samples, thereby solving the problem of "small training data samples" in surface defect detection to a certain extent.

[0120] The preferred embodiment of the present invention combines the concept of metric learning in the improved twin network structure, and improves the generalization performance of the model by learning the similarities and differences between the corresponding features of the input samples. Furthermore, the preferred embodiment of the present invention provides a single-sample learning target detection device for defect detection, which adds a feature enhancement network designed based on the improved Non-Local attention mechanism. The feature enhancement network can simultaneously process the two feature maps at the input end, mutually enhance features with strong correlation (or similarity) while suppressing features with weaker similarity, thereby solving the problem of "low generalization performance" to a certain extent and achieving the technical effect of further enhancing the generalization performance of the model.

[0121] like Figure 8 As shown, a preferred embodiment of the present invention also provides an electronic device, the target detection electronic device 800 includes: a memory 801 for storing non-temporary computer-readable instructions; and a processor 802 for running the computer-readable instructions, so that when the computer-readable instructions are executed by the processor, the above-mentioned single-sample learning target detection method is implemented.

[0122] A preferred embodiment of the present invention further provides a computer-readable storage medium having executable code stored thereon. When the executable code is executed by a processor, the processor implements the target detection method of single-sample learning described above.

[0123] Those skilled in the art will appreciate that, in order to address the technical issue of achieving a good user experience, this embodiment may also include well-known structures such as a communication bus and an interface, and these well-known structures should also be included within the scope of protection of the preferred embodiments of the present invention. For detailed descriptions and technical effects of this embodiment, please refer to the corresponding descriptions of the aforementioned embodiments and will not be repeated here.

[0124] The single-sample learning target detection electronic device provided by the preferred embodiment of the present invention, on the one hand, replaces the backbone network module in the standard YOLOv3 solution by using a lightweight YOLO-fastest network design. Compared with the classic YOLOv3 based on the DarkNet backbone network, the backbone network used in the preferred embodiment of the present invention has fewer parameters, lower computational complexity, and stronger real-time performance. On the other hand, the preferred embodiment of the present invention uses the target detection method in YOLOv3 and adopts YOLO-fastest to more effectively implement a single-stage single-sample learning target detection method model. Compared with the existing two-stage single-sample learning target detection method model, the complexity of the single-sample learning target detection method model is simplified and the computational complexity of the single-sample learning target detection method model is reduced. The preferred embodiment of the present invention makes the entire training and reasoning process more concise and efficient in model training and reasoning, and the calculation speed is faster, solving the problem of "poor real-time performance" in surface defect detection. The single-sample learning target detection electronic device provided by the preferred embodiment of the present invention improves the model's ability to perform metric learning by adding a feature matching module, thereby achieving the technical effect of further enhancing the generalization performance of the method model.

[0125] A preferred embodiment of the present invention further provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on a device, the device executes the target detection method of single-sample learning described above.

[0126] Among them, the devices, equipment, computer-readable storage media and computer program products or chips provided in the preferred embodiments of the present invention are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0127] The preferred embodiment of the present invention also provides the application of a single-sample learning target detection method to texture-based surface defect detection. When applied to texture-based surface defect detection, the target detection method model described above first undergoes deep learning using a single-sample learning target detection method model similar to texture-based surface defect detection. Based on this learning, texture-based surface defects are then identified. The identification results are then compared and analyzed, and the target detection method model of the preferred embodiment of the present invention is evaluated. The results are as follows.

[0128] This example compares the classic YOLO with the preferred embodiment of the present invention. When both use YOLO-fastest as the backbone network, the detection performance of the two is compared as shown in the attached figure. Figure 2 、 3 As shown in Table 1, "Seen Classes" indicates that these categories appeared in the training dataset and belong to the validation set; "Unseen Classes" indicates that these categories did not appear in the training dataset and belong to the test set. "Seen" here represents texture categories that appeared in the training set, namely 2, 3, 4, 6, 7, 8, and 9. "Used" represents texture categories that did not appear in the training set, namely 1, 5, and 10. Since these 10 categories were randomly assigned to "Seen" and "Unsed" in a 7:3 ratio, the order of the category numbers was shuffled.

[0129] Table 1

[0130]

[0131] In the preferred embodiment of the present invention, YOLOv3-yolofastest has a total of 3032 training sample images; while the preferred embodiment of the present invention has a total of 90,120 pairs of training sample pairs after multiple pairings. Judging from the corresponding detection results of Seen, the data results of the target detection performance of the preferred embodiment of the present invention in the seven randomly acquired categories are correspondingly improved compared with YOLOv3-yolofastest. This result shows that the detection method provided by the preferred embodiment of the present invention has, to a certain extent, solved the difficulty of "small sample training data" in the field of texture surface defect detection compared to OLOv3-yolofastest.

[0132] Unssen's results show that the preferred embodiment of the present invention has stronger generalization capabilities than the classic YOLOv3; Seen's results show that the preferred embodiment of the present invention has stronger learning capabilities for problems with small sample training datasets. In other words, the method proposed in the preferred embodiment of the present invention outperforms the classic YOLOv3 on both the validation set and the test set. Figure 7The detection effects of the classic YOLOv3 and the preferred embodiment of the present invention on the test set samples are demonstrated. It can be seen that the preferred embodiment of the present invention can also effectively detect defects in texture categories that do not appear in the training set, while the classic YOLOv3 has missed detections, false detections, etc. in the defect detection process, and the corresponding true value accuracy cannot meet the requirements of high-precision industrial target detection. The result data of the above detection verifies that the preferred embodiment of the present invention can effectively solve the two problems of "small sample training data" and "low generalization performance" existing in this technical background while realizing surface defect position detection. As shown in Table 1, the preferred embodiment of the present invention improves the generalization performance in the target detection process. The results shown in Table 1 just verify that by improving the generalization performance, missed detection and false detection are effectively avoided.

[0133] This embodiment can detect a defect sample in approximately 20ms on a GTX1660S, while the classic DarkNet-based YOLOv3 requires approximately 200ms. This result verifies that the preferred embodiment of the present invention can effectively solve the problem of "poor real-time performance" in this technical background.

[0134] The preferred embodiment of the present invention is different from the prior art solution.

[0135] (1) Existing technical solutions for texture-based surface defect detection can detect the location of defects, and the calculation speed may be fast enough. However, these solutions have low generalization performance and cannot detect surface defects on textures of different types and styles.

[0136] (2) Existing technical solutions for texture-based surface defect detection can classify whether an image has defects and can effectively learn with a small number of defective image samples, with strong generalization performance and high real-time performance. For example, there are improved classification models based on twin networks (VGG16, ResNet50, MobileNetv3, etc.). However, these solutions can only classify images and cannot obtain the specific location of defects, or use relatively backward and computationally complex methods such as sliding windows to obtain the location of defects.

[0137] The above description is merely a preferred embodiment of the preferred embodiment of the present invention, and does not constitute any form of limitation on the preferred embodiment of the present invention. Although the preferred embodiment of the present invention has been disclosed as above in the form of a preferred embodiment, it is not intended to limit the preferred embodiment of the present invention. Any technician familiar with this profession can make some changes or modifications to equivalent embodiments of the present invention using the technical content disclosed above without departing from the scope of the preferred embodiment of the present invention. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the preferred embodiment of the present invention without departing from the content of the preferred embodiment of the present invention are still within the scope of the preferred embodiment of the present invention.

Claims

1. A target detection method, characterized in that: The steps include: Get support sample x s and query sample x q As two image samples of input; wherein, the support sample x s is a normal sample without defects, the query sample x q is the sample to be tested; Two backbone networks with the same structure and shared weights are used to support the sample x s and query sample x q Perform feature extraction to obtain the corresponding feature map G w (x s ) and G w (x q ); The feature map G w (x s ) and G w (x q ) Input the feature enhancement network to obtain the enhanced and / or suppressed feature maps v s and v q ; Based on the feature matching network, the enhanced and / or suppressed feature maps v are respectively s and v q Perform similarity measurement and output the measurement result H(v s , v q ); The measurement result H(v s , v q ) Input the YOLO layer module to perform regression calculation, and predict the query sample x based on the regression calculation result q Defect location and / or confidence in .

2. The target detection method according to claim 1, wherein: The two backbone networks with the same structure and shared weights support the sample x s and query sample x q Perform feature extraction to obtain the corresponding feature map G w (x s ) and G w (x q )include: The backbone network is composed of YOLO-fastest and is denoted as Gw; The support sample x s and query sample x q Inputting the input ends of the backbone network respectively for feature extraction; Get the corresponding feature map G w (x s ) and G w (x q ).

3. The target detection method according to claim 1, wherein: The feature map G w (x s ) and G w (x q ) Input feature enhancement network to obtain the enhanced and / or suppressed feature maps v s and v q include: Constructing the feature enhancement network based on the improved Non-Local attention mechanism; The two feature maps G w (x s ) and G w (x q ) input to the input end of the feature enhancement network; Features with strong correlations enhance each other, while features with weaker similarities suppress each other; The feature enhancement network outputs the enhanced and / or suppressed feature maps v respectively. s and v q ; Among them, the feature map G w (x s ) has a dimension of w s *h s *c,G w (x q ) has a dimension of w q *h q *c;h s , h q Represents the feature map G w (x s ) and G w (x q ) height dimension, w s , w q Should represent the feature map G w (x s ) and G w (x q ) is the width size, c is the feature map G w (x s ) and G w (x q ) channels.

4. The target detection method according to claim 3, wherein: The method of mutually enhancing features with strong correlation and suppressing features with weak similarity includes: The feature map G w (x s ) are convolved through two point-by-point convolutional networks to make the feature map G w (x s ) space size remains unchanged, the number of channels is reduced by half, and the results are recorded as g(G w (x s ))and The feature map G w (x q ) are convolved through two point-by-point convolutional networks to make the feature map G w (x q ) space size remains unchanged, the number of channels is reduced by half, and the results are recorded as g(G w (x q )) and θ(G w (x q )); Will and θ(G w (x q )) are reconstructed into two two-dimensional matrices respectively; the two two-dimensional matrices are matrix multiplied to obtain a dimension of w q h q *w s h s The matrix of dimension w q h q *w s h s The network layer composed of the matrix input softmax function completes the similarity calculation; the output dimension is w q h q *w s h s Matrix of g(G w (x s )) and g(G w (x q )) is reconstructed into a matrix; the matrix is ​​matrix-multiplied with the output of the softmax function, and the dimensions are and Two matrices of and The two matrices are reconstructed into dimensions and Two feature maps of ; The dimension and The two feature maps of are respectively input into two point-by-point convolutional networks for channel dimension increase; the results of channel dimension increase are respectively compared with G w (x s ) and G w (x q ) add them together to get v s and v q ; in, W θ , and These are all linear transformation coefficients, i, j represent the characteristics of the map The i-th or W θ G W The jth element, Represents the similarity calculation function of the above feature graph; v s The dimension is w s *h s *c,v q The dimension is w q *h q *c.

5. The target detection method according to claim 4, characterized in that: described The similarity calculation function uses the radial basis function to calculate the similarity between two vectors, as shown below: Among them, W θ G w (x s ) i , Represent two column vectors respectively.

6. The target detection method according to claim 5, characterized in that: The feature matching network is based on the enhanced and / or suppressed feature map v s and v q Perform similarity measurement and output the measurement result H(v s , v q )include: The dimensions are w s *h s *c's respective enhanced and / or suppressed feature maps v s and w q *h q *c's respective enhanced and / or suppressed feature maps v q Input feature matching network; v s w s *h s vectors of dimension c*1 and v q w q *h q Vectors with a dimension of c*1 are combined with each other and the similarity is calculated according to the similarity calculation formula; the dimension is w q *h q *(w s h s c) similarity feature graph; the similarity calculation formula is: similarity(v s，i ,v q，j )=(v s，i -v q，j ) 2 ; Wherein, the subscripts i and j represent v s The i-th vector and v q The jth vector of ; Let dimension be w q *h q *(w s h s c) The similarity feature map is grouped and convolved to obtain a dimension of w q *h q *c similarity feature map; wherein the convolution kernel size of the grouped convolution is 1*1 and the step size is 1; The dimension w is q *h q * The similarity feature map of c and the respective enhanced and / or suppressed feature maps v q Along the w q *h q *c dimensions are spliced; The final output dimension of the feature matching network is w q *h q *(2c)The measurement result H(v s , v q ).

7. The target detection method according to claim 6, characterized in that: The measurement result H(v s , v q ) Input the YOLO layer for regression calculation, and predict the query sample x based on the regression calculation result q The defect location and / or confidence level in include: The measurement result H(v s , v q ) Input the YOLO layer for regression calculation; Based on the loss function CIOU, a first error loss is obtained between the prediction result of the defect position and the true value label, and the query sample x is predicted based on the first error loss. q location of defects in; and / or When predicting the confidence, a second error loss between the predicted confidence of the defect position and the true value label is obtained based on the ternary loss function in the twin network, and the query sample x is predicted based on the second error loss. q Confidence of defect location in ; The ternary loss function is: TripleLoss=∑max((1-y)y′+y(my′),0) Where y is the label, 0 indicates no defect, and 1 indicates defect; y′ represents the confidence of the output of the YOLO layer, and the confidence value range is [0, 1]; m represents the edge margin, and m = 1.

8. The target detection method according to claim 7, wherein: When predicting the defect location, the detection method further includes: using a non-primary maximum suppression algorithm (NMS) to filter out the overlapping recognition results of the defect location, and outputting the query sample x q The location and confidence of the defect in the .

9. A target detection device, characterized in that: Includes the following modules: Sample acquisition module, used to obtain support sample x s and query sample x q As two image samples of input; wherein, the support sample x s is a normal sample without defects, the query sample x q is the sample to be tested; The backbone network module is composed of two backbone networks with the same structure and shared weights. The backbone networks support the sample x s and query sample x q Perform feature extraction to obtain the corresponding feature map G w (x s ) and G w (x q ); Feature enhancement module, the feature map G w (x s ) and G w (x q ) Input the feature enhancement module to obtain the enhanced and / or suppressed feature maps v s and v q ; The feature matching module is based on the feature matching module to respectively perform the enhanced and / or suppressed feature map v s and v q Perform similarity measurement and output the measurement result H(v s , v q ); YOLO layer module, the measurement result H(v s , v q ) Input the YOLO layer module to perform regression calculation, and predict the query sample x based on the regression calculation result q Defect location and / or confidence in .

10. An electronic device, characterized in that: include: a memory for storing non-transitory computer-readable instructions; as well as A processor is configured to run the computer-readable instructions, so that when the computer-readable instructions are executed by the processor, the target detection method according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium, characterized in that The method comprises computer instructions, which, when executed on a device, enable the device to perform the target detection method according to any one of claims 1 to 8.

12. Application of the target detection method according to any one of claims 1 to 8 in texture surface defect detection.

Images