Irregular defect detection method based on dynamic vision sensor and Yolox

Abstract

The invention discloses an irregular defect detection method based on a dynamic visual sensor and Yolox, which comprises the following steps of: acquiring an image sequence, and inputting a pre-trained defect detection model to execute the following steps of: extracting a double down-sampling feature map by utilizing a Focus structure for each event frame, performing deformable convolution on the double down-sampling feature map through a DCNV3 structure, inputting the double down-sampling feature map into a CSPDarknet53 network, and performing data processing on the CSPDarknet53 network; obtaining a first feature map, and inputting the first feature map into the PAFPN to generate second feature maps of different scales; obtaining second feature maps of the same scale corresponding to all event frames, inputting the second feature maps to a spatial-temporal feature aggregation module, and calculating to obtain a first two-three similarity matrix under each scale; and according to the first two-three similarity matrix under each scale and the second feature maps of the same scale corresponding to all the event frames, calculating to obtain a plurality of time sequence fusion feature maps, and inputting the time sequence fusion feature maps into a decoupling head to obtain a defect detection result of each event frame. According to the invention, a good detection effect is realized on defects with different scales and irregular shapes.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to an irregular defect detection method based on a dynamic vision sensor and Yolox. Background Art

[0002] Defect detection is a key step in ensuring product and process quality. By promptly discovering and correcting defects, we can prevent defective products and improve product quality standards. Existing visual defect detection methods usually use traditional CCD / CMOS cameras for defect imaging, and then use back-end detection algorithms for further processing. They have been widely used in industries such as textiles and ceramics. However, traditional CCD / CMOS cameras use energy integration to image the foreground and background with equal weights. They face problems such as slow imaging speed, small dynamic range, large interference from environmental background, unclear defect imaging, and difficult feature extraction, which poses great challenges to the detection of weak defects in complex dynamic environments.

[0003] Yolox is a target detection algorithm based on deep learning. Its core concept is to achieve faster detection speed while maintaining high accuracy. The algorithm is based on a single-shot target detection framework and achieves a balance between detection accuracy and speed by adopting advanced neural network structures and optimization strategies. However, due to the asynchronous output characteristics of dynamic vision sensors and the irregular shapes of defects, directly applying Yolox to defect detection does not produce good results.

[0004] also, et al. proposed a method for surface defect detection using a segmentation-based end-to-end two-stage network. The specific steps are: use a semantic segmentation fully convolutional network to train each pixel in the input image as a sample, perform deconvolution to obtain pixel-level prediction results; combine the segmentation output and the features of the segmentation network for classification to obtain the final classification output result; train in two stages, perform inference after training, and finally determine whether the surface has defects by probability. The network used in this method is simple and can only classify relatively obvious surface defects, so its application has great limitations; at the same time, for situations with severe interference such as reflective surfaces or noise, the network has difficulty distinguishing between weak defects and interference. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides an irregular defect detection method based on a dynamic vision sensor and Yolox. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] The present invention provides an irregular defect detection method based on a dynamic vision sensor and Yolox, comprising:

[0007] Acquire an image sequence and a pre-trained defect detection model, wherein the image sequence includes a plurality of continuous event frames, and the pre-trained defect detection model includes: a Focus structure, a DCNV3 structure, a CSPDarknet53 network, a path aggregation feature pyramid PAFPN, a spatiotemporal feature aggregation module, and a decoupling head;

[0008] The image sequence is input into a pre-trained defect detection model, which performs the following steps:

[0009] For each event frame, a double downsampled feature map of the event frame is extracted using the Focus structure. The double downsampled feature map is subjected to deformable convolution by the DCNV3 structure and then input into the CSPDarknet53 network for feature extraction to obtain a first feature map. The first feature map is input into the path aggregation feature pyramid PAFPN to generate second feature maps of different scales.

[0010] Obtain the second feature maps of the same scale corresponding to all event frames and input them into the spatiotemporal feature aggregation module to calculate the first similarity matrix, the second similarity matrix and the third similarity matrix at each scale;

[0011] According to the first similarity matrix, the second similarity matrix and the third similarity matrix at each scale and the second feature maps of the same scale corresponding to all event frames, multiple time series fusion feature maps are calculated;

[0012] The multiple time series fusion feature maps are input into the decoupling head to obtain the defect detection results of each event frame.

[0013] In one embodiment of the present invention, the image sequence includes three consecutive event frames: a first event frame, a second event frame and a third event frame, and the three event frames are all obtained based on a dynamic vision sensor.

[0014] In one embodiment of the present invention, the step of inputting the first feature map into a path aggregation feature pyramid PAFPN to generate second feature maps of different scales includes:

[0015] The first feature map is input into the path aggregation feature pyramid PAFPN to generate a second feature map of a first scale, a second feature map of a second scale, and a second feature map of a third scale, wherein the first scale is 80×80×256, the second scale is 40×40×512, and the third scale is 20×20×1024.

[0016] In one embodiment of the present invention, the steps of obtaining the second feature maps of the same scale corresponding to all event frames and inputting them into the spatiotemporal feature aggregation module, and calculating the first similarity matrix, the second similarity matrix and the third similarity matrix at each scale include:

[0017] Respectively obtain the second feature maps of the first scale corresponding to the first event frame, the second event frame, and the third event frame, and input them into the spatiotemporal feature aggregation module to calculate the first similarity matrix at the first scale;

[0018] Respectively obtain the second feature maps of the second scale corresponding to the first event frame, the second event frame, and the third event frame, and input them into the spatiotemporal feature aggregation module to calculate the second similarity matrix at the second scale;

[0019] The second feature maps of the third scale corresponding to the first event frame, the second event frame and the third event frame are respectively obtained and input into the spatiotemporal feature aggregation module to calculate the third similarity matrix at the third scale.

[0020] In one embodiment of the present invention, the first similarity matrix at the first scale includes: a first similarity matrix A, a first similarity matrix B and a first similarity matrix C;

[0021] The steps of respectively obtaining second feature maps of the first scale corresponding to the first event frame, the second event frame, and the third event frame, and inputting the second feature maps into the spatiotemporal feature aggregation module to calculate a first similarity matrix at the first scale include:

[0022] Perform a dot product operation on the second feature map of the first scale corresponding to the first event frame and the second feature map of the first scale corresponding to the second event frame after convolution respectively, to obtain a first similarity matrix A;

[0023] Perform a dot product operation on the second feature map of the first scale corresponding to the second event frame after convolution with itself to obtain a first similarity matrix B;

[0024] A dot product operation is performed on the second feature map of the first scale corresponding to the second event frame and the second feature map of the first scale corresponding to the third event frame after convolution to obtain a first similarity matrix C.

[0025] In one embodiment of the present invention, the second similarity matrix at the second scale includes: a second similarity matrix A, a second similarity matrix B, and a second similarity matrix C, and the third similarity matrix at the third scale includes: a third similarity matrix A, a third similarity matrix B, and a third similarity matrix C;

[0026] The step of calculating a plurality of temporal fusion feature maps according to the first similarity matrix, the second similarity matrix and the third similarity matrix at each scale and the second feature maps of the same scale corresponding to all event frames includes:

[0027] A first temporal fusion feature map is calculated according to the first similarity matrix A, the first similarity matrix B, the first similarity matrix C at the first scale and the second feature maps of the first scale corresponding to the first event frame, the second event frame, and the third event frame;

[0028] A second temporal fusion feature map is calculated based on the second similarity matrix A, the second similarity matrix B, the second similarity matrix C at the second scale and the second feature maps of the second scale corresponding to the first event frame, the second event frame, and the third event frame;

[0029] A third temporal fusion feature map is calculated according to the third similarity matrix A, the third similarity matrix B, the third similarity matrix C at the third scale and the third feature maps of the third scale corresponding to the first event frame, the second event frame, and the third event frame.

[0030] In one embodiment of the present invention, the step of calculating a first temporal fusion feature map according to a first similarity matrix A, a first similarity matrix B, a first similarity matrix C at a first scale and a second feature map of a first scale corresponding to a first event frame, a second event frame, and a third event frame includes:

[0031] Performing a dot product operation on the first similarity matrix A and the second feature map of the first scale corresponding to the first event frame, performing a dot product operation on the first similarity matrix B and the second feature map of the first scale corresponding to the second event frame, and performing a dot product operation on the first similarity matrix C and the second feature map of the first scale corresponding to the third event frame, respectively, to obtain a first weighted feature, a second weighted feature, and a third weighted feature;

[0032] The first weighted feature, the second weighted feature and the third weighted feature are fused by using a convolution operation, and then fused with the second feature map of the first scale corresponding to the second event frame through a residual structure to obtain a first temporal fusion feature map.

[0033] In one embodiment of the present invention, the step of inputting the multiple time series fusion feature graphs into a decoupling head to obtain defect detection results of each event frame includes:

[0034] Inputting the first time series fusion feature map, the second time series fusion feature map and the third time series fusion feature map into the decoupling head to obtain defect information of each event frame; the defect information at least includes: a predicted frame position containing a defect and a defect category, and the defect category is one of "dot", "scratch" and "stain";

[0035] Determine the defect detection result according to the predicted box positions of the same category defects in the first event frame, the second event frame, and the third event frame.

[0036] In one embodiment of the present invention, the step of determining the defect detection result according to the predicted frame positions of the same type of defects in the first event frame, the second event frame and the third event frame includes:

[0037] Obtain the predicted box positions of defects of the same category in the first event frame and the second event frame and calculate the first intersection-union ratio between the two;

[0038] Obtaining predicted box positions of defects of the same category in the second event frame and the third event frame and calculating a second intersection-over-union ratio between the two;

[0039] The prediction box corresponding to the first intersection-and-union ratio less than the preset threshold in the second event frame is deleted, and the prediction box corresponding to the second intersection-and-union ratio less than the preset threshold in the second event frame is deleted, and the defect detection result is determined according to the remaining prediction boxes in the second event frame.

[0040] In one embodiment of the present invention, the defect detection model is trained based on a preset loss function, and the preset loss function includes a classification loss function, a positioning loss function and a VariFocal loss function.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention provides an irregular defect detection method based on a dynamic vision sensor and Yolox. The pre-trained defect detection model includes: a Focus structure, a DCNV3 structure, a CSPDarknet53 network, a path aggregation feature pyramid PAFPN and a decoupling head. Among them, deformable convolution is embedded in the DCNV3 structure. Combined with the path aggregation feature pyramid network, the present invention can achieve good detection effects on defects of different scales and irregular shapes.

[0043] Since the defect detection model in the present invention is trained based on the defect data set generated by the dynamic vision sensor, the asynchronous output characteristics of the dynamic vision sensor are taken into consideration, and the network structure is adjusted according to the characteristics of the data, making the present invention suitable for scenarios such as dynamic vision sensors.

[0044] In addition, since the present invention utilizes the spatiotemporal attention mechanism to fuse the correlation features between different event frames and fully utilizes the timing information, the present invention further significantly improves the accuracy of defect detection.

[0045] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flow chart of an irregular defect detection method based on a dynamic vision sensor and Yolox provided by an embodiment of the present invention;

[0047] Figure 2 is a schematic diagram of the structure of a defect detection model provided by an embodiment of the present invention;

[0048] Figure 3 is an example diagram of an aluminum substrate with defects provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0050] Figure 1 is a flow chart of an irregular defect detection method based on a dynamic vision sensor and Yolox provided by an embodiment of the present invention. Figure 2 Schematic diagram of the structure of the defect detection model provided by the embodiment of the present invention. Figure 1-2 As shown, an embodiment of the present invention provides an irregular defect detection method based on a dynamic vision sensor and Yolox, comprising:

[0051] S1. Obtain an image sequence and a pre-trained defect detection model. The image sequence includes multiple continuous event frames. The pre-trained defect detection model includes: Focus structure, DCNV3 structure, CSPDarknet53 network, path aggregation feature pyramid PAFPN, spatiotemporal feature aggregation module and decoupling head;

[0052] S2. Input the image sequence into the pre-trained defect detection model and make it perform the following steps:

[0053] S201, for each event frame, using the Focus structure to extract a double downsampled feature map of the event frame, the double downsampled feature map is subjected to deformable convolution by the DCNV3 structure and then input into the CSPDarknet53 network for feature extraction to obtain a first feature map, and the first feature map is input into the path aggregation feature pyramid PAFPN to generate second feature maps of different scales;

[0054] S202, obtaining the second feature graphs of the same scale corresponding to all event frames and inputting them into the spatiotemporal feature aggregation module, and calculating the first similarity matrix, the second similarity matrix and the third similarity matrix at each scale;

[0055] S203, calculating and obtaining a plurality of time series fusion feature maps according to the first similarity matrix, the second similarity matrix and the third similarity matrix at each scale and the second feature maps of the same scale corresponding to all event frames;

[0056] S204, inputting multiple time series fusion feature graphs into the decoupling head to obtain defect detection results of each event frame.

[0057] In this embodiment, multiple continuous event frames are used as inputs of the defect detection model, and these event frames can be obtained based on a dynamic vision sensor. The dynamic vision sensor is a new type of vision sensor, in which each pixel works independently and uses a differential logarithmic photosensitivity method (which only produces a pulse response to changes in light intensity) for imaging. The dynamic vision sensor uses an asynchronous signal triggering method, which gives it the advantages of a large dynamic range, fast imaging speed, low data redundancy, significant imaging features for weak moving targets, and low power consumption. In addition, the large dynamic range enables it to effectively capture moving targets under various lighting conditions such as weak light, strong light, and backlight. The dynamic vision sensor's characteristics of moving target sensitivity, large dynamic range, and low information redundancy make it very suitable for application scenarios of weak defect detection, and it can capture weak defects more clearly and stably than traditional cameras.

[0058] It should be noted that after the event frame is input into the defect detection model, the image size can be adaptively adjusted to 640×640, and the single-channel image can be converted into a three-channel image.

[0059] Optionally, in step S1, the image sequence includes three consecutive event frames: a first event frame, a second event frame and a third event frame.

[0060] In step S201, after each event frame of size 640×640×3 is input into the pre-trained defect detection model, the Focus structure first obtains a double downsampled feature map of size 160×160×128 with no information loss through slicing and Concat operations. Next, the double downsampled feature map is input into the DCNV3 structure for deformable convolution to expand the receptive field to effectively extract irregular and sparse defect features. In this embodiment, the number of group convolutions is set to 8, and the convolution kernel size is 3*3. The deformable convolution allows the position of the convolution kernel to be non-uniformly sampled to adapt to the irregular shape and position of the target or feature, so that the convolution kernel can perform different weight calculations at each position.

[0061] For example, a normal convolution can be expressed as:

[0062]

[0063] Compared with ordinary convolution, deformable convolution adds a learnable offset to the location of the convolution, expressed as:

[0064]

[0065] Among them, p0 represents any point on the double down-sampled feature map, p n Represents the offset of each point in the convolution kernel relative to p0, Δp n Indicates the offset, w(p n) represents the weight of the corresponding position of the convolution kernel, x(p0+p n +Δp n ) represents the times down-sampling feature map p0+p n +Δp n The element value at position y(p0) represents the element value at position p0 on the feature map output by the DCNV3 structure, which is obtained by convolving the convolution kernel with the twice downsampled feature map.

[0066] The feature map of size 160×160×128 output by the DCNV3 structure is further input into the CSPDarknet53 network for feature extraction. Through operations such as convolution and residual superposition, a balance between detection accuracy and speed is achieved.

[0067] Furthermore, the step of inputting the first feature map into the path aggregation feature pyramid PAFPN to generate second feature maps of different scales includes:

[0068] The first feature map is input into the path aggregation feature pyramid PAFPN to generate a second feature map of a first scale, a second feature map of a second scale, and a second feature map of a third scale, where the first scale is 80×80×256, the second scale is 40×40×512, and the third scale is 20×20×1024.

[0069] It should be understood that the path aggregation feature pyramid PAFPN processes the feature map output by the DCNV3 structure to construct a multi-level feature pyramid, which includes second feature maps of different scales from coarse to detailed. Optionally, the second feature map includes three scales: 80×80×256, 40×40×512, and 20×20×1024. This design aims to make the defect detection model better adapt to the different sizes and complexities of various targets, thereby significantly improving the detection performance.

[0070] In step S202, the steps of obtaining the second feature maps of the same scale corresponding to all event frames and inputting them into the spatiotemporal feature aggregation module, and calculating the first similarity matrix, the second similarity matrix and the third similarity matrix at each scale include:

[0071] S2021, respectively obtain second feature maps of the first scale corresponding to the first event frame, the second event frame, and the third event frame, and input them into a spatiotemporal feature aggregation module to calculate a first similarity matrix at the first scale;

[0072] S2022, respectively obtain second feature maps of the second scale corresponding to the first event frame, the second event frame, and the third event frame, and input them into a spatiotemporal feature aggregation module to calculate a second similarity matrix at the second scale;

[0073] S2023, respectively obtain the second feature maps of the third scale corresponding to the first event frame, the second event frame, and the third event frame, and input them into the spatiotemporal feature aggregation module to calculate and obtain a third similarity matrix at the third scale.

[0074] The first similarity matrix at the first scale includes: a first similarity matrix A, a first similarity matrix B and a first similarity matrix C;

[0075] The steps of respectively obtaining second feature maps of the first scale corresponding to the first event frame, the second event frame, and the third event frame, and inputting the second feature maps into the spatiotemporal feature aggregation module to calculate a first similarity matrix at the first scale include:

[0076] Perform a dot product operation on the second feature map of the first scale corresponding to the first event frame and the second feature map of the first scale corresponding to the second event frame after convolution respectively, to obtain a first similarity matrix A;

[0077] Perform a dot product operation on the second feature map of the first scale corresponding to the second event frame after convolution with itself to obtain a first similarity matrix B;

[0078] The second feature map of the first scale corresponding to the second event frame and the second feature map of the first scale corresponding to the third event frame are convolved and then dot-producted to obtain a first similarity matrix C.

[0079] In this embodiment, the second feature maps of the same scale corresponding to the three event frames are input into the spatiotemporal feature aggregation module. After the second feature maps of the same scale corresponding to the first event frame and the third event frame pass through the convolution layer, they are respectively subjected to dot product operations with the second feature maps of the same scale corresponding to the second event to obtain pixel-level similarity matrices, namely: the first similarity matrix A and the first similarity matrix C. At the same time, the second feature map of the first scale corresponding to the second event frame is subjected to dot product operations with itself after convolution to obtain the first similarity matrix B. It should be noted that the sizes of the first similarity matrix A, the first similarity matrix B and the first similarity matrix C are all B×3×C×H×W, where B represents Batchsize, C represents the number of channels, and H and W represent the height and width of the similarity matrix, respectively.

[0080] In addition, the second similarity matrix at the second scale includes: a second similarity matrix A, a second similarity matrix B, and a second similarity matrix C. The third similarity matrix at the third scale includes: a third similarity matrix A, a third similarity matrix B, and a third similarity matrix C. Since the calculation process is the same as that of the first similarity matrix A, the first similarity matrix B, and the first similarity matrix C, it will not be repeated here.

[0081] In step S203, the step of calculating a plurality of temporal fusion feature maps according to the first similarity matrix, the second similarity matrix and the third similarity matrix at each scale and the second feature maps of the same scale corresponding to all event frames includes:

[0082] S2031, calculating a first temporal fusion feature map according to a first similarity matrix A, a first similarity matrix B, a first similarity matrix C at a first scale, and second feature maps of the first scale corresponding to a first event frame, a second event frame, and a third event frame;

[0083] S2032, calculating a second temporal fusion feature map according to a second similarity matrix A, a second similarity matrix B, a second similarity matrix C at a second scale and second feature maps of the second scale corresponding to the first event frame, the second event frame, and the third event frame;

[0084] S2033. Calculate a third temporal fusion feature map according to the third similarity matrix A, the third similarity matrix B, the third similarity matrix C at the third scale and the third feature maps of the third scale corresponding to the first event frame, the second event frame, and the third event frame.

[0085] Among them, the step of calculating the first temporal fusion feature map according to the first similarity matrix A, the first similarity matrix B, the first similarity matrix C at the first scale and the second feature maps of the first scale corresponding to the first event frame, the second event frame, and the third event frame includes:

[0086] Performing a dot product operation on the first similarity matrix A and the second feature map of the first scale corresponding to the first event frame, performing a dot product operation on the first similarity matrix B and the second feature map of the first scale corresponding to the second event frame, and performing a dot product operation on the first similarity matrix C and the second feature map of the first scale corresponding to the third event frame, respectively, to obtain a first weighted feature, a second weighted feature, and a third weighted feature;

[0087] The first weighted feature, the second weighted feature and the third weighted feature are fused by using a convolution operation, and then fused with the second feature map of the first scale corresponding to the second event frame through a residual structure to obtain a first temporal fusion feature map.

[0088] It should be noted that the generation methods of the second time series fusion feature map and the third time series fusion feature map are similar to those of the first time series fusion feature map, so they will not be repeated here.

[0089] In step S204, the step of inputting multiple time series fusion feature maps into the decoupling head to obtain the defect detection results of each event frame includes:

[0090] S2041, inputting the first time series fusion feature map, the second time series fusion feature map and the third time series fusion feature map into the decoupling head to obtain defect information of each event frame; the defect information at least includes: the predicted frame position containing the defect and the defect category, and the defect category is one of "dot", "scratch" and "stain";

[0091] S2042: Determine defect detection results according to predicted box positions of defects of the same category in the first event frame, the second event frame, and the third event frame.

[0092] The step of determining the defect detection result according to the predicted frame positions of the same type of defects in the first event frame, the second event frame and the third event frame includes:

[0093] Obtain the predicted box positions of defects of the same category in the first event frame and the second event frame and calculate the first intersection-union ratio between the two;

[0094] Obtaining predicted box positions of defects of the same category in the second event frame and the third event frame and calculating a second intersection-over-union ratio between the two;

[0095] The prediction box corresponding to the first intersection-and-union ratio less than the preset threshold in the second event frame is deleted, and the prediction box corresponding to the second intersection-and-union ratio less than the preset threshold in the second event frame is deleted, and the defect detection result is determined according to the remaining prediction boxes in the second event frame.

[0096] Optionally, the defect detection model in this embodiment is trained based on a preset loss function, and the preset loss function includes a classification loss function, a positioning loss function and a VariFocal loss function, which are expressed as:

[0097] Loss=loss(box)+VFL(p,q)+loss(class).

[0098] Specifically, in the training process of Yolox, the loss function used is usually composed of a classification loss function loss(class), a positioning loss function loss(box) and a confidence loss function loss(obj). In this embodiment, the confidence loss function loss(obj) is replaced by a VariFocal loss function to adapt to the situation where the proportion of defects in the sample image is extremely small.

[0099] It should be understood that when the above defect detection model is trained using a defect detection dataset based on a dynamic visual sensor, the positive and negative samples in the defect detection dataset are extremely unbalanced, where positive samples are sample images with defects and negative samples are sample images without defects, and the proportion of defects in the entire image is extremely small. The VariFocal loss function reduces the weight of easy-to-classify samples to reduce their contribution to the total loss, while increasing the weight of difficult-to-classify foreground samples to increase attention to these samples.

[0100] The calculation formula of VariFocal loss function is as follows:

[0101]

[0102] Among them, p represents the predicted defect category score, q is the target classification label score, and is a soft label related to IOU. When q>0, VFL has no hyperparameters for positive samples, that is, there is no attenuation; when q=0, VFL has hyperparameters for negative samples, γ will reduce the contribution of negative samples, and α is to prevent over-suppression, which will reduce the contribution of negative samples in general. This makes the defect detection model more focused on foreground samples (defects) that are difficult to classify during training, thereby improving the performance of the target detector in the case of class imbalance.

[0103] Next, the irregular defect detection method based on the dynamic vision sensor and Yolox provided by the present invention is demonstrated through simulation experiments.

[0104] 1. Simulation experiment conditions:

[0105] The hardware platform is: the processor is Intel(R) Xeon(R) CPU E5-2680 v3, the main frequency is 2.90GHz, the memory is 16GB, and the GPU is RTX 2080Ti.

[0106] The software platform is: Windows 10 operating system, python3.8, PyTorch1.7.0 and pycharm2017.

[0107] Specifically, the irregular defect detection method based on dynamic vision sensor and Yolox is used to detect defects on the surface of industrial products. The defect detection data set based on dynamic vision sensor is collected from different aluminum substrates, which contains three types of defects: spots, scratches, and stains. Figure 3 This is an example of an aluminum substrate with defects provided by an embodiment of the present invention, in which a scratch is contained in a rectangular box. The defect detection dataset contains a total of 16,643 images and 9,094 labels, of which 89 sequence images are used as training sets, 12 sequence images are used as validation sets, and 15 sequence images are used as test sets.

[0108] Table 1 shows the distribution of three types of defect and no-object image sequences in the training set, validation set, and test set. Table 2 shows the number of different types of defect labels in the training set, validation set, and test set.

[0109] Table 1

[0110]

[0111]

[0112] Table 2

[0113] category Training set Validation set Test Set Spot 2381 233 251 Scratches 1781 200 178 Stains 3060 637 373 total 7222 1070 802

[0114] From Table 1-2, it can be seen that the data set meets the task requirements of defect detection and the data set distribution is reasonable.

[0115] 2. Simulation content and results analysis:

[0116] This simulation experiment selected existing target detection algorithms for comparison, including SSD-event, Faster-RCNN, YoloV, MEGA, and RDN. In order to evaluate the performance of the model, the map@0.4 (the mean average precision of the model when the IoU threshold is 0.4) indicator is used. The experimental results are shown in Table 3:

[0117] Table 3

[0118] Model Map0.4 point Scratches Stains SSD-event 19.94 5 35 19 YOLOV5 53.90 51.20 75.40 35.00 YOLO V7 59.21 53.07 61.01 63.89 YOLOV 53.7 11.16 62.81 67 Faster-Rcnn 20.20 0 43 18 MEGA 41.22 34.85 40.72 48.08 RDN 53.30 56.61 30.12 73.17 The present invention 72.34 57 74 86

[0119] It can be seen from Table 3 that the present invention achieves the highest detection performance, which intuitively demonstrates the superiority of the present invention in the defect detection task based on dynamic vision sensors.

[0120] The above simulation experiments show that: the present invention embeds deformable convolution and path aggregation pyramid network to construct a multi-scale feature association module to extract irregular defect features, designs a spatiotemporal feature aggregation module to fuse spatiotemporal feature information, replaces the VariFocal loss function, and implements decision-level fusion in the reasoning stage, so that the method can perform efficient defect detection tasks on data based on dynamic vision sensors, solving the problem that traditional image algorithms cannot be effectively applied to data with asynchronous output characteristics based on dynamic vision sensors, cannot effectively extract irregular defect features, and do not make good use of data timing information. Experiments have proved that this method is superior to the existing technology in terms of both interpretability and irregular defect detection task performance, and is an efficient and interpretable irregular defect detection method based on dynamic vision sensors.

[0121] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0122] The description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification.

[0123] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. An irregular defect detection method based on dynamic vision sensor and Yolox, characterized in that: include: Acquire an image sequence and a pre-trained defect detection model, wherein the image sequence includes a plurality of continuous event frames, and the pre-trained defect detection model includes: a Focus structure, a DCNV3 structure, a CSPDarknet53 network, a path aggregation feature pyramid PAFPN, a spatiotemporal feature aggregation module, and a decoupling head; The image sequence is input into a pre-trained defect detection model, which performs the following steps: For each event frame, a double downsampled feature map of the event frame is extracted using the Focus structure. The double downsampled feature map is subjected to deformable convolution by the DCNV3 structure and then input into the CSPDarknet53 network for feature extraction to obtain a first feature map. The first feature map is input into the path aggregation feature pyramid PAFPN to generate second feature maps of different scales. Obtain the second feature maps of the same scale corresponding to all event frames and input them into the spatiotemporal feature aggregation module to calculate the first similarity matrix, the second similarity matrix and the third similarity matrix at each scale; According to the first similarity matrix, the second similarity matrix and the third similarity matrix at each scale and the second feature maps of the same scale corresponding to all event frames, multiple time series fusion feature maps are calculated; The multiple time series fusion feature maps are input into the decoupling head to obtain the defect detection results of each event frame.

2. The irregular defect detection method based on dynamic vision sensor and Yolox according to claim 1 is characterized in that: The image sequence includes three consecutive event frames: a first event frame, a second event frame and a third event frame, and the three event frames are all obtained based on a dynamic vision sensor.

3. The irregular defect detection method based on dynamic vision sensor and Yolox according to claim 1, characterized in that: The step of inputting the first feature map into the path aggregation feature pyramid PAFPN to generate second feature maps of different scales includes: The first feature map is input into the path aggregation feature pyramid PAFPN to generate a second feature map of a first scale, a second feature map of a second scale, and a second feature map of a third scale, wherein the first scale is 80×80×256, the second scale is 40×40×512, and the third scale is 20×20×1024.

4. The irregular defect detection method based on dynamic vision sensor and Yolox according to claim 2 is characterized in that: The steps of obtaining the second feature maps of the same scale corresponding to all event frames and inputting them into the spatiotemporal feature aggregation module, and calculating the first similarity matrix, the second similarity matrix and the third similarity matrix at each scale include: Respectively obtain the second feature maps of the first scale corresponding to the first event frame, the second event frame, and the third event frame, and input them into the spatiotemporal feature aggregation module to calculate the first similarity matrix at the first scale; Respectively obtain the second feature maps of the second scale corresponding to the first event frame, the second event frame, and the third event frame, and input them into the spatiotemporal feature aggregation module to calculate the second similarity matrix at the second scale; The second feature maps of the third scale corresponding to the first event frame, the second event frame and the third event frame are respectively obtained and input into the spatiotemporal feature aggregation module to calculate the third similarity matrix at the third scale.

5. The irregular defect detection method based on dynamic vision sensor and Yolox according to claim 4 is characterized in that: The first similarity matrix at the first scale includes: a first similarity matrix A, a first similarity matrix B and a first similarity matrix C; The steps of respectively obtaining second feature maps of the first scale corresponding to the first event frame, the second event frame, and the third event frame, and inputting the second feature maps into the spatiotemporal feature aggregation module to calculate a first similarity matrix at the first scale include: Perform a dot product operation on the second feature map of the first scale corresponding to the first event frame and the second feature map of the first scale corresponding to the second event frame after convolution respectively, to obtain a first similarity matrix A; Perform a dot product operation on the second feature map of the first scale corresponding to the second event frame after convolution with itself to obtain a first similarity matrix B; A dot product operation is performed on the second feature map of the first scale corresponding to the second event frame and the second feature map of the first scale corresponding to the third event frame after convolution to obtain a first similarity matrix C.

6. The irregular defect detection method based on dynamic vision sensor and Yolox according to claim 5 is characterized in that: The second similarity matrix at the second scale includes: a second similarity matrix A, a second similarity matrix B, and a second similarity matrix C; the third similarity matrix at the third scale includes: a third similarity matrix A, a third similarity matrix B, and a third similarity matrix C; The step of calculating a plurality of temporal fusion feature maps according to the first similarity matrix, the second similarity matrix and the third similarity matrix at each scale and the second feature maps of the same scale corresponding to all event frames includes: A first temporal fusion feature map is calculated according to the first similarity matrix A, the first similarity matrix B, the first similarity matrix C at the first scale and the second feature maps of the first scale corresponding to the first event frame, the second event frame, and the third event frame; A second temporal fusion feature map is calculated based on the second similarity matrix A, the second similarity matrix B, the second similarity matrix C at the second scale and the second feature maps of the second scale corresponding to the first event frame, the second event frame, and the third event frame; A third temporal fusion feature map is calculated according to the third similarity matrix A, the third similarity matrix B, the third similarity matrix C at the third scale and the third feature maps of the third scale corresponding to the first event frame, the second event frame, and the third event frame.

7. The irregular defect detection method based on dynamic vision sensor and Yolox according to claim 6 is characterized in that: The step of calculating a first temporal fusion feature map according to a first similarity matrix A, a first similarity matrix B, a first similarity matrix C at a first scale and second feature maps of the first scale corresponding to a first event frame, a second event frame, and a third event frame includes: Performing a dot product operation on the first similarity matrix A and the second feature map of the first scale corresponding to the first event frame, performing a dot product operation on the first similarity matrix B and the second feature map of the first scale corresponding to the second event frame, and performing a dot product operation on the first similarity matrix C and the second feature map of the first scale corresponding to the third event frame, respectively, to obtain a first weighted feature, a second weighted feature, and a third weighted feature; The first weighted feature, the second weighted feature and the third weighted feature are fused by using a convolution operation, and then fused with the second feature map of the first scale corresponding to the second event frame through a residual structure to obtain a first temporal fusion feature map.

8. The irregular defect detection method based on dynamic vision sensor and Yolox according to claim 6, characterized in that: The step of inputting the multiple time series fusion feature graphs into the decoupling head to obtain the defect detection results of each event frame includes: Input the first temporal fusion feature map, the second temporal fusion feature map and the third temporal fusion feature map into the decoupling head to obtain defect information of each event frame; the defect information at least includes: the predicted frame position containing the defect and the defect category, and the defect category is one of "dot", "scratch" and "stain"; Determine the defect detection result according to the predicted box positions of the same category defects in the first event frame, the second event frame, and the third event frame.

9. The irregular defect detection method based on dynamic vision sensor and Yolox according to claim 8, characterized in that: The step of determining defect detection results according to the predicted frame positions of defects of the same category in the first event frame, the second event frame, and the third event frame comprises: Obtain the predicted box positions of defects of the same category in the first event frame and the second event frame and calculate the first intersection-union ratio between the two; Obtaining predicted box positions of defects of the same category in the second event frame and the third event frame and calculating a second intersection-over-union ratio between the two; The prediction box corresponding to the first intersection-and-union ratio less than the preset threshold in the second event frame is deleted, and the prediction box corresponding to the second intersection-and-union ratio less than the preset threshold in the second event frame is deleted, and the defect detection result is determined according to the remaining prediction boxes in the second event frame.

10. The irregular defect detection method based on dynamic vision sensor and Yolox according to claim 1, characterized in that: The defect detection model is trained based on a preset loss function, and the preset loss function includes a classification loss function, a positioning loss function and a VariFocal loss function.

Images