Irregular defect detection method based on dynamic vision sensor and Yolox

Abstract

This invention discloses a method for detecting irregular defects based on a dynamic vision sensor and Yolox. The method comprises the following steps: obtaining an image sequence and inputting it into a pre-trained defect detection model, which performs the following steps: for each event frame, extracting a double-downsampled feature map using a Focus structure, performing deformable convolution with a DCNV3 structure, and inputting it into a CSPDarknet53 network to obtain a first feature map. This first feature map is then input into a PAFPN network to generate second feature maps of different scales; obtaining the second feature maps of the same scale corresponding to all event frames and inputting them into a spatiotemporal feature aggregation module to calculate the first, second, and third similarity matrices at each scale; and calculating multiple temporal fusion feature maps based on the first, second, and third similarity matrices at each scale and the second feature maps of the same scale corresponding to all event frames. These maps are then input into a decoupling head to obtain defect detection results for each event frame. The method achieves good detection results for defects of different scales and irregular shapes.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to an irregular defect detection method based on a dynamic vision sensor and Yolox. Background Art

[0002] Defect detection is a critical step in ensuring product and process quality. Promptly detecting and correcting defects can prevent substandard products and improve product quality standards. Existing visual defect detection methods typically use traditional CCD / CMOS cameras to image defects, followed by further processing using back-end detection algorithms. These methods have been widely used in industries such as textiles and ceramics. However, traditional CCD / CMOS cameras use energy integration to image weighted foreground and background images. This poses challenges such as slow imaging speed, narrow dynamic range, significant interference from environmental backgrounds, unclear defect imaging, and difficulty in feature extraction. These methods pose significant challenges for detecting small defects in complex, dynamic environments.

[0003] Yolox is a deep learning-based object detection algorithm. Its core concept is to achieve faster detection speeds while maintaining high accuracy. Based on a single-shot object detection framework, the algorithm utilizes advanced neural network structures and optimization strategies to achieve a balance between accuracy and speed. However, due to the asynchronous output of dynamic vision sensors and the irregular shapes of defects, directly applying Yolox to defect detection does not yield good results.

[0004] also, et al. proposed a method for surface defect detection using a segmentation-based, end-to-end, two-stage network. The method involves training each pixel in the input image as a sample through a semantic segmentation fully convolutional network, performing deconvolution to obtain pixel-level predictions. Classification is performed by combining the segmentation output with the features of the segmentation network to obtain the final classification output. Training is performed in two stages, followed by inference after training, and finally a probabilistic judgment of whether the surface is defective. This method uses a simple network that can only classify relatively obvious surface defects, which has significant limitations. Furthermore, in situations with heavy interference, such as reflective surfaces or noise, the network has difficulty distinguishing between subtle defects and interference. Summary of the Invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides an irregular defect detection method based on a dynamic vision sensor and Yolox. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] The present invention provides an irregular defect detection method based on a dynamic vision sensor and Yolox, comprising:

[0007] Obtain an image sequence and a pre-trained defect detection model, wherein the image sequence includes multiple consecutive event frames, and the pre-trained defect detection model includes: a Focus structure, a DCNV3 structure, a CSPDarknet53 network, a path aggregation feature pyramid PAFPN, a spatiotemporal feature aggregation module, and a decoupling head;

[0008] The image sequence is fed into a pre-trained defect detection model, which performs the following steps:

[0009] For each event frame, a double-downsampled feature map of the event frame is extracted using the Focus structure. The double-downsampled feature map is deformably convolved with the DCNV3 structure and then input into the CSPDarknet53 network for feature extraction to obtain a first feature map. The first feature map is then input into the path aggregation feature pyramid PAFPN to generate second feature maps of different scales.

[0010] Obtain the second feature maps of the same scale corresponding to all event frames and input them into the spatiotemporal feature aggregation module to calculate the first similarity matrix, second similarity matrix, and third similarity matrix at each scale;

[0011] Calculate and obtain multiple temporal fusion feature maps based on the first similarity matrix, the second similarity matrix, and the third similarity matrix at each scale and the second feature maps of the same scale corresponding to all event frames;

[0012] The multiple time series fusion feature maps are input into the decoupling head to obtain the defect detection results of each event frame.

[0013] In one embodiment of the present invention, the image sequence includes three consecutive event frames: a first event frame, a second event frame, and a third event frame, and the three event frames are all obtained based on a dynamic vision sensor.

[0014] In one embodiment of the present invention, the step of inputting the first feature map into a path aggregation feature pyramid PAFPN to generate second feature maps of different scales includes:

[0015] The first feature map is input into the path aggregation feature pyramid PAFPN to generate a second feature map of a first scale, a second feature map of a second scale, and a second feature map of a third scale, where the first scale is 80×80×256, the second scale is 40×40×512, and the third scale is 20×20×1024.

[0016] In one embodiment of the present invention, the steps of obtaining second feature maps of the same scale corresponding to all event frames and inputting them into the spatiotemporal feature aggregation module, and calculating the first similarity matrix, the second similarity matrix, and the third similarity matrix at each scale include:

[0017] Obtain the second feature maps of the first scale corresponding to the first event frame, the second event frame, and the third event frame respectively, and input them into the spatiotemporal feature aggregation module to calculate the first similarity matrix at the first scale;

[0018] Obtain the second feature maps of the second scale corresponding to the first event frame, the second event frame, and the third event frame respectively, and input them into the spatiotemporal feature aggregation module to calculate the second similarity matrix at the second scale;

[0019] The second feature maps of the third scale corresponding to the first event frame, the second event frame, and the third event frame are respectively obtained and input into the spatiotemporal feature aggregation module to calculate the third similarity matrix at the third scale.

[0020] In one embodiment of the present invention, the first similarity matrix at the first scale includes: a first similarity matrix A, a first similarity matrix B, and a first similarity matrix C;

[0021] The steps of respectively obtaining the second feature maps of the first scale corresponding to the first event frame, the second event frame, and the third event frame, and inputting the second feature maps into the spatiotemporal feature aggregation module to calculate a first similarity matrix at the first scale include:

[0022] Performing a dot product operation on the second feature map of the first scale corresponding to the first event frame and the second feature map of the first scale corresponding to the second event frame after convolution respectively, to obtain a first similarity matrix A;

[0023] Perform a dot product operation on the second feature map of the first scale corresponding to the second event frame after convolution with itself to obtain a first similarity matrix B;

[0024] A dot product operation is performed on the second feature map of the first scale corresponding to the second event frame and the second feature map of the first scale corresponding to the third event frame after convolution to obtain a first similarity matrix C.

[0025] In one embodiment of the present invention, the second similarity matrix at the second scale includes: a second similarity matrix A, a second similarity matrix B, and a second similarity matrix C; the third similarity matrix at the third scale includes: a third similarity matrix A, a third similarity matrix B, and a third similarity matrix C;

[0026] The step of calculating multiple temporal fusion feature maps according to the first similarity matrix, the second similarity matrix, and the third similarity matrix at each scale and the second feature maps of the same scale corresponding to all event frames includes:

[0027] A first temporal fusion feature map is calculated based on the first similarity matrix A, the first similarity matrix B, the first similarity matrix C at the first scale and the second feature maps of the first scale corresponding to the first event frame, the second event frame, and the third event frame;

[0028] A second temporal fusion feature map is calculated based on the second similarity matrix A, the second similarity matrix B, the second similarity matrix C at the second scale and the second feature maps of the second scale corresponding to the first event frame, the second event frame, and the third event frame;

[0029] A third temporal fusion feature map is calculated based on the third similarity matrix A, the third similarity matrix B, the third similarity matrix C at the third scale and the third feature maps of the third scale corresponding to the first event frame, the second event frame, and the third event frame.

[0030] In one embodiment of the present invention, the step of calculating a first temporal fusion feature map according to a first similarity matrix A, a first similarity matrix B, a first similarity matrix C at a first scale, and second feature maps of the first scale corresponding to a first event frame, a second event frame, and a third event frame includes:

[0031] Performing a dot product operation on the first similarity matrix A and the second feature map of the first scale corresponding to the first event frame, performing a dot product operation on the first similarity matrix B and the second feature map of the first scale corresponding to the second event frame, and performing a dot product operation on the first similarity matrix C and the second feature map of the first scale corresponding to the third event frame, respectively, to obtain a first weighted feature, a second weighted feature, and a third weighted feature;

[0032] The first weighted feature, the second weighted feature, and the third weighted feature are fused using a convolution operation, and then fused with the second feature map of the first scale corresponding to the second event frame through a residual structure to obtain a first temporal fusion feature map.

[0033] In one embodiment of the present invention, the step of inputting the multiple time series fusion feature maps into the decoupling head to obtain the defect detection results of each event frame includes:

[0034] Inputting the first temporal fusion feature map, the second temporal fusion feature map, and the third temporal fusion feature map into a decoupling head to obtain defect information of each event frame; the defect information at least includes: a predicted box position containing a defect and a defect category, where the defect category is one of "spot", "scratch", and "stain";

[0035] Determine the defect detection result according to the predicted box positions of defects of the same category in the first event frame, the second event frame, and the third event frame.

[0036] In one embodiment of the present invention, the step of determining a defect detection result based on the predicted box positions of defects of the same category in the first event frame, the second event frame, and the third event frame includes:

[0037] Obtain the predicted box positions of defects of the same category in the first event frame and the second event frame and calculate the first intersection-over-union ratio between the two;

[0038] Obtain the predicted box positions of defects of the same category in the second event frame and the third event frame and calculate the second intersection-over-union ratio between the two;

[0039] The prediction box corresponding to the first intersection-and-union ratio less than the preset threshold in the second event frame is deleted, and the prediction box corresponding to the second intersection-and-union ratio less than the preset threshold in the second event frame is deleted, and the defect detection result is determined based on the remaining prediction boxes in the second event frame.

[0040] In one embodiment of the present invention, the defect detection model is trained based on a preset loss function, and the preset loss function includes a classification loss function, a positioning loss function, and a VariFocal loss function.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention provides an irregular defect detection method based on a dynamic vision sensor and Yolox. The pre-trained defect detection model includes: a Focus structure, a DCNV3 structure, a CSPDarknet53 network, a path aggregation feature pyramid PAFPN, and a decoupling head. Among them, the DCNV3 structure is embedded with deformable convolution. Combined with the path aggregation feature pyramid network, the present invention can achieve good detection effects on defects of different scales and irregular shapes.

[0043] Since the defect detection model in the present invention is trained based on the defect data set generated by the dynamic vision sensor, the asynchronous output characteristics of the dynamic vision sensor are taken into account, and the network structure is adjusted according to the characteristics of the data, making the present invention suitable for scenarios such as dynamic vision sensors.

[0044] In addition, since the present invention utilizes the spatiotemporal attention mechanism to fuse the correlation features between different event frames and fully utilizes the timing information, the present invention further significantly improves the accuracy of defect detection.

[0045] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flow chart of an irregular defect detection method based on a dynamic vision sensor and Yolox provided by an embodiment of the present invention;

[0047] Figure 2 is a schematic structural diagram of a defect detection model provided by an embodiment of the present invention;

[0048] Figure 3 This is an example diagram of an aluminum substrate with defects provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0050] Figure 1 This is a flow chart of an irregular defect detection method based on a dynamic vision sensor and Yolox provided by an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of the defect detection model provided by the embodiment of the present invention. Figure 1-2 As shown, an embodiment of the present invention provides an irregular defect detection method based on a dynamic vision sensor and Yolox, comprising:

[0051] S1. Obtain an image sequence and a pre-trained defect detection model. The image sequence includes multiple consecutive event frames. The pre-trained defect detection model includes: Focus structure, DCNV3 structure, CSPDarknet53 network, path aggregation feature pyramid PAFPN, spatiotemporal feature aggregation module and decoupling head;

[0052] S2. Input the image sequence into the pre-trained defect detection model and make it perform the following steps:

[0053] S201. For each event frame, a double-downsampled feature map of the event frame is extracted using the Focus structure. The double-downsampled feature map is deformably convolved with the DCNV3 structure and then input into the CSPDarknet53 network for feature extraction to obtain a first feature map. The first feature map is then input into the path aggregation feature pyramid PAFPN to generate second feature maps of different scales.

[0054] S202: Obtain the second feature maps of the same scale corresponding to all event frames and input them into the spatiotemporal feature aggregation module to calculate the first similarity matrix, the second similarity matrix, and the third similarity matrix at each scale;

[0055] S203: Calculate and obtain multiple time series fusion feature maps based on the first similarity matrix, the second similarity matrix, and the third similarity matrix at each scale and the second feature maps of the same scale corresponding to all event frames;

[0056] S204: Input multiple time series fusion feature maps into the decoupling head to obtain defect detection results of each event frame.

[0057] In this embodiment, multiple continuous event frames are used as input to the defect detection model, and these event frames can be obtained based on a dynamic vision sensor. The dynamic vision sensor is a new type of vision sensor in which each pixel works independently and uses a differential logarithmic photosensitivity method (which only produces a pulse response to changes in light intensity) for imaging. The dynamic vision sensor uses an asynchronous signal triggering method, which gives it the advantages of a large dynamic range, fast imaging speed, low data redundancy, significant imaging characteristics for weak moving targets, and low power consumption. In addition, the large dynamic range enables it to effectively capture moving targets under various lighting conditions such as weak light, strong light, and backlight. The characteristics of the dynamic vision sensor such as sensitivity to moving targets, large dynamic range, and low information redundancy make it very suitable for application scenarios of weak defect detection. Compared with traditional cameras, it can capture weak defects more clearly and stably.

[0058] It should be noted that after the event frame is input into the defect detection model, the image size can be adaptively adjusted to 640×640, and the single-channel image can be converted into a three-channel image.

[0059] Optionally, in step S1 , the image sequence includes three consecutive event frames: a first event frame, a second event frame, and a third event frame.

[0060] In step S201, after each event frame of size 640×640×3 is input into the pre-trained defect detection model, the Focus structure first performs a slicing and concat operation to obtain a doubly downsampled feature map of size 160×160×128 with no information loss. Next, the doubly downsampled feature map is input into the DCNV3 structure for deformable convolution to expand the receptive field and effectively extract irregular and sparse defect features. In this embodiment, the number of group convolutions is set to 8 and the convolution kernel size is 3*3. Deformable convolution allows the convolution kernel position to be non-uniformly sampled to adapt to the irregular shape and position of the target or feature, so that the convolution kernel can perform different weight calculations at each position.

[0061] For example, ordinary convolution can be expressed as:

[0062]

[0063] Compared with ordinary convolution, deformable convolution adds a learnable offset to the location of the convolution, which is expressed as:

[0064]

[0065] Among them, p0 represents any point on the double down-sampling feature map, p n Indicates the offset of each point in the convolution kernel relative to p0, Δp n Indicates the offset, w(p n) represents the weight of the convolution kernel corresponding to the position, x(p0+p n +Δp n ) represents the times down-sampled feature map p0+p n +Δp n The element value at position y(p0) represents the element value at position p0 on the feature map output by the DCNV3 structure, which is obtained by convolving the convolution kernel with the double-downsampled feature map.

[0066] The feature map of size 160×160×128 output by the DCNV3 structure is further input into the CSPDarknet53 network for feature extraction. Through operations such as convolution and residual superposition, a balance between detection accuracy and speed is achieved.

[0067] Furthermore, the step of inputting the first feature map into the path aggregation feature pyramid PAFPN to generate second feature maps of different scales includes:

[0068] The first feature map is input into the path aggregation feature pyramid PAFPN to generate a second feature map of the first scale, a second feature map of the second scale, and a second feature map of the third scale, where the first scale is 80×80×256, the second scale is 40×40×512, and the third scale is 20×20×1024.

[0069] It should be understood that the path aggregation feature pyramid PAFPN processes the feature map output by the DCNV3 structure to construct a multi-level feature pyramid, which includes second feature maps of different scales from coarse to detailed. Optionally, the second feature map includes three scales: 80×80×256, 40×40×512, and 20×20×1024. This design aims to enable the defect detection model to better adapt to the different sizes and complexities of various targets, thereby significantly improving the detection performance.

[0070] In step S202, the steps of obtaining the second feature maps of the same scale corresponding to all event frames and inputting them into the spatiotemporal feature aggregation module, and calculating the first similarity matrix, the second similarity matrix, and the third similarity matrix at each scale include:

[0071] S2021, respectively obtain the second feature maps of the first scale corresponding to the first event frame, the second event frame, and the third event frame, and input them into the spatiotemporal feature aggregation module to calculate a first similarity matrix at the first scale;

[0072] S2022: Obtain second feature maps of the second scale corresponding to the first event frame, the second event frame, and the third event frame, respectively, and input them into a spatiotemporal feature aggregation module to calculate a second similarity matrix at the second scale;

[0073] S2023 , respectively obtain the second feature maps of the third scale corresponding to the first event frame, the second event frame, and the third event frame, and input them into the spatiotemporal feature aggregation module to calculate a third similarity matrix at the third scale.

[0074] The first similarity matrix at the first scale includes: a first similarity matrix A, a first similarity matrix B, and a first similarity matrix C;

[0075] The steps of respectively obtaining the second feature maps of the first scale corresponding to the first event frame, the second event frame, and the third event frame, and inputting the second feature maps into the spatiotemporal feature aggregation module to calculate a first similarity matrix at the first scale include:

[0076] Performing a dot product operation on the second feature map of the first scale corresponding to the first event frame and the second feature map of the first scale corresponding to the second event frame after convolution respectively, to obtain a first similarity matrix A;

[0077] Perform a dot product operation on the second feature map of the first scale corresponding to the second event frame after convolution with itself to obtain a first similarity matrix B;

[0078] The second feature map of the first scale corresponding to the second event frame and the second feature map of the first scale corresponding to the third event frame are convolved and then dot-producted to obtain a first similarity matrix C.

[0079] In this embodiment, the second feature maps of the same scale corresponding to the three event frames are input into the spatiotemporal feature aggregation module. After the second feature maps of the same scale corresponding to the first event frame and the third event frame pass through the convolution layer, they are respectively subjected to a dot product operation with the second feature map of the same scale corresponding to the second event to obtain pixel-level similarity matrices, namely: the first similarity matrix A and the first similarity matrix C. At the same time, the second feature map of the first scale corresponding to the second event frame is subjected to a dot product operation with itself after convolution to obtain the first similarity matrix B. It should be noted that the dimensions of the first similarity matrix A, the first similarity matrix B, and the first similarity matrix C are all B×3×C×H×W, where B represents the batch size, C represents the number of channels, and H and W represent the height and width of the similarity matrix, respectively.

[0080] In addition, the second similarity matrix at the second scale includes: a second similarity matrix A, a second similarity matrix B, and a second similarity matrix C. The third similarity matrix at the third scale includes: a third similarity matrix A, a third similarity matrix B, and a third similarity matrix C. Since the calculation process is the same as that of the first similarity matrix A, the first similarity matrix B, and the first similarity matrix C, it will not be repeated here.

[0081] In step S203, the step of calculating multiple temporal fusion feature maps based on the first similarity matrix, the second similarity matrix, and the third similarity matrix at each scale and the second feature maps of the same scale corresponding to all event frames includes:

[0082] S2031. Calculate a first temporal fusion feature map based on the first similarity matrix A, the first similarity matrix B, the first similarity matrix C at the first scale, and the second feature maps of the first scale corresponding to the first event frame, the second event frame, and the third event frame;

[0083] S2032: Calculate a second temporal fusion feature map based on the second similarity matrix A, the second similarity matrix B, the second similarity matrix C at the second scale and the second feature maps of the second scale corresponding to the first event frame, the second event frame, and the third event frame;

[0084] S2033. Calculate a third temporal fusion feature map based on the third similarity matrix A, the third similarity matrix B, the third similarity matrix C at the third scale and the third feature maps of the third scale corresponding to the first event frame, the second event frame, and the third event frame.

[0085] The step of calculating the first temporal fusion feature map according to the first similarity matrix A, the first similarity matrix B, the first similarity matrix C at the first scale and the second feature maps of the first scale corresponding to the first event frame, the second event frame, and the third event frame includes:

[0086] Performing a dot product operation on the first similarity matrix A and the second feature map of the first scale corresponding to the first event frame, performing a dot product operation on the first similarity matrix B and the second feature map of the first scale corresponding to the second event frame, and performing a dot product operation on the first similarity matrix C and the second feature map of the first scale corresponding to the third event frame, respectively, to obtain a first weighted feature, a second weighted feature, and a third weighted feature;

[0087] The first weighted feature, the second weighted feature, and the third weighted feature are fused using a convolution operation, and then fused with the second feature map of the first scale corresponding to the second event frame through a residual structure to obtain a first temporal fusion feature map.

[0088] It should be noted that the generation methods of the second time series fusion feature map and the third time series fusion feature map are similar to those of the first time series fusion feature map, so they will not be repeated here.

[0089] In step S204, the steps of inputting multiple time series fusion feature maps into the decoupling head to obtain the defect detection results of each event frame include:

[0090] S2041. Input the first time series fusion feature map, the second time series fusion feature map, and the third time series fusion feature map into the decoupling head to obtain defect information of each event frame; the defect information includes at least: the location of the predicted box containing the defect and the defect category, where the defect category is one of "spot", "scratch", and "stain";

[0091] S2042: Determine defect detection results based on predicted box positions of defects of the same category in the first event frame, the second event frame, and the third event frame.

[0092] The step of determining the defect detection result according to the predicted frame positions of the same type of defects in the first event frame, the second event frame, and the third event frame includes:

[0093] Obtain the predicted box positions of defects of the same category in the first event frame and the second event frame and calculate the first intersection-over-union ratio between the two;

[0094] Obtain the predicted box positions of defects of the same category in the second event frame and the third event frame and calculate the second intersection-over-union ratio between the two;

[0095] The prediction box corresponding to the first intersection-and-union ratio less than the preset threshold in the second event frame is deleted, and the prediction box corresponding to the second intersection-and-union ratio less than the preset threshold in the second event frame is deleted, and the defect detection result is determined based on the remaining prediction boxes in the second event frame.

[0096] Optionally, the defect detection model in this embodiment is trained based on a preset loss function. The preset loss function includes a classification loss function, a positioning loss function, and a VariFocal loss function, which are expressed as:

[0097] Loss=loss(box)+VFL(p,q)+loss(class).

[0098] Specifically, in the training process of Yolox, the loss function used is usually composed of the classification loss function loss(class), the positioning loss function loss(box) and the confidence loss function loss(obj). In this embodiment, the confidence loss function loss(obj) is replaced by the VariFocal loss function to adapt to the situation where the proportion of defects in the sample image is extremely small.

[0099] It should be understood that when training the above defect detection model using a dynamic vision sensor-based defect detection dataset, the positive and negative samples in the defect detection dataset are extremely unbalanced. Positive samples are images with defects, while negative samples are images without defects. Defects account for a very small proportion of the entire image. The VariFocal loss function downweights easily classified samples, reducing their contribution to the total loss, while increasing the weight of difficult-to-classify foreground samples to increase their focus.

[0100] The calculation formula of the VariFocal loss function is as follows:

[0101]

[0102] Here, p represents the predicted defect category score, and q is the target category label score, a soft label related to the IOU. When q > 0, VFL has no hyperparameters for positive samples, meaning no attenuation. However, when q = 0, VFL has hyperparameters for negative samples. γ reduces the contribution of negative samples, while α prevents over-suppression and generally reduces the contribution of negative samples. This allows the defect detection model to focus more on difficult-to-classify foreground samples (defects) during training, thereby improving the performance of object detectors in cases of class imbalance.

[0103] Next, the irregular defect detection method based on dynamic vision sensor and Yolox provided by the present invention is demonstrated through simulation experiments.

[0104] 1. Simulation experiment conditions:

[0105] The hardware platform is: the processor is Intel(R) Xeon(R) CPU E5-2680 v3, the main frequency is 2.90GHz, the memory is 16GB, and the GPU is RTX 2080Ti.

[0106] The software platform is: Windows 10 operating system, Python 3.8, PyTorch 1.7.0 and PyCharm 2017.

[0107] Specifically, the irregular defect detection method based on dynamic vision sensors and Yolox is used to detect defects on the surface of industrial products. The defect detection dataset based on dynamic vision sensors is collected from different aluminum substrates and contains three types of defects: spots, scratches, and stains. Figure 3 This is an example of an aluminum substrate with a defect, as provided by an embodiment of the present invention. The rectangular box in the image represents a scratch. The defect detection dataset contains 16,643 images and 9,094 labels, including 89 image sequences as a training set, 12 image sequences as a validation set, and 15 image sequences as a test set.

[0108] Table 1 shows the distribution of three types of defect and no-object image sequences in the training set, validation set, and test set. Table 2 shows the number of different types of defect labels in the training set, validation set, and test set.

[0109] Table 1

[0110]

[0111]

[0112] Table 2

[0113] category training set Validation set Test set Dot Mark 2381 233 251 scratches 1781 200 178 stains 3060 637 373 total 7222 1070 802

[0114] From Table 1-2, we can see that the dataset meets the task requirements of defect detection and the dataset distribution is reasonable.

[0115] 2. Simulation content and results analysis:

[0116] This simulation experiment selected existing object detection algorithms for comparison, including SSD-event, Faster-RCNN, YoloV, MEGA, and RDN. To evaluate the performance of the model, the map@0.4 metric (the mean average precision of the model at an IoU threshold of 0.4) was used. The experimental results are shown in Table 3:

[0117] Table 3

[0118] Model Map0.4 point scratches stains SSD-event 19.94 5 35 19 YOLOV5 53.90 51.20 75.40 35.00 YOLO V7 59.21 53.07 61.01 63.89 YOLOV 53.7 11.16 62.81 67 Faster-Rcnn 20.20 0 43 18 MEGA 41.22 34.85 40.72 48.08 RDN 53.30 56.61 30.12 73.17 The present invention 72.34 57 74 86

[0119] As can be seen from Table 3, the present invention achieves the highest detection performance, which intuitively demonstrates the superiority of the present invention in the defect detection task based on dynamic vision sensors.

[0120] The above simulation experiments show that the present invention embeds deformable convolution and path aggregation pyramid networks to construct a multi-scale feature association module to extract irregular defect features, designs a spatiotemporal feature aggregation module to fuse spatiotemporal feature information, replaces the VariFocal loss function, and implements decision-level fusion in the inference stage. This allows the method to perform efficient defect detection tasks on data based on dynamic vision sensors, solving the problems that traditional image algorithms cannot be effectively applied to data with asynchronous output characteristics based on dynamic vision sensors, cannot effectively extract irregular defect features, and do not make good use of data timing information. Experiments have shown that this method is superior to existing technologies in terms of both interpretability and irregular defect detection task performance. It is an efficient and interpretable irregular defect detection method based on dynamic vision sensors.

[0121] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0122] Descriptions with reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0123] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A method for detecting irregular defects based on a dynamic vision sensor and Yolox, characterized in that: include: Obtain an image sequence and a pre-trained defect detection model, wherein the image sequence includes multiple consecutive event frames, and the pre-trained defect detection model includes: a Focus structure, a DCNV3 structure, a CSPDarknet53 network, a path aggregation feature pyramid PAFPN, a spatiotemporal feature aggregation module, and a decoupling head; the image sequence includes three consecutive event frames: a first event frame, a second event frame, and a third event frame, and the three event frames are all obtained based on a dynamic vision sensor; The image sequence is fed into a pre-trained defect detection model, which performs the following steps: For each event frame, a double-downsampled feature map of the event frame is extracted using the Focus structure. The double-downsampled feature map is deformably convolved with the DCNV3 structure and then input into the CSPDarknet53 network for feature extraction to obtain a first feature map. The first feature map is input into the path aggregation feature pyramid PAFPN to generate a second feature map of the first scale, a second feature map of the second scale, and a second feature map of the third scale. Obtain the second feature maps of the same scale corresponding to all event frames and input them into the spatiotemporal feature aggregation module to calculate the first similarity matrix, second similarity matrix, and third similarity matrix at each scale, including: Obtaining the second feature maps of the first scale corresponding to the first event frame, the second event frame, and the third event frame, respectively, and inputting them into the spatiotemporal feature aggregation module to calculate a first similarity matrix at the first scale; the first similarity matrix at the first scale includes: a first similarity matrix A, a first similarity matrix B, and a first similarity matrix C; The steps of respectively obtaining the second feature maps of the first scale corresponding to the first event frame, the second event frame, and the third event frame, and inputting the second feature maps into the spatiotemporal feature aggregation module to calculate a first similarity matrix at the first scale include: Performing a dot product operation on the second feature map of the first scale corresponding to the first event frame and the second feature map of the first scale corresponding to the second event frame after convolution respectively, to obtain a first similarity matrix A; Perform a dot product operation on the second feature map of the first scale corresponding to the second event frame after convolution with itself to obtain a first similarity matrix B; Perform a dot product operation on the second feature map of the first scale corresponding to the second event frame and the second feature map of the first scale corresponding to the third event frame after convolution to obtain a first similarity matrix C; Obtain the second feature maps of the second scale corresponding to the first event frame, the second event frame, and the third event frame respectively, and input them into the spatiotemporal feature aggregation module to calculate the second similarity matrix at the second scale; Obtain the second feature maps of the third scale corresponding to the first event frame, the second event frame, and the third event frame respectively, and input them into the spatiotemporal feature aggregation module to calculate the third similarity matrix at the third scale; Calculate a plurality of temporal fusion feature maps based on the first similarity matrix, the second similarity matrix, and the third similarity matrix at each scale and the second feature maps of the same scale corresponding to all event frames; the second similarity matrix at the second scale includes: a second similarity matrix A, a second similarity matrix B, and a second similarity matrix C; and the third similarity matrix at the third scale includes: a third similarity matrix A, a third similarity matrix B, and a third similarity matrix C; The step of calculating multiple temporal fusion feature maps according to the first similarity matrix, the second similarity matrix, and the third similarity matrix at each scale and the second feature maps of the same scale corresponding to all event frames includes: A first temporal fusion feature map is calculated based on the first similarity matrix A, the first similarity matrix B, the first similarity matrix C at the first scale and the second feature maps of the first scale corresponding to the first event frame, the second event frame, and the third event frame, including: Performing a dot product operation on the first similarity matrix A and the second feature map of the first scale corresponding to the first event frame, performing a dot product operation on the first similarity matrix B and the second feature map of the first scale corresponding to the second event frame, and performing a dot product operation on the first similarity matrix C and the second feature map of the first scale corresponding to the third event frame, respectively, to obtain a first weighted feature, a second weighted feature, and a third weighted feature; The first weighted feature, the second weighted feature, and the third weighted feature are fused using a convolution operation, and then fused with a second feature map of the first scale corresponding to the second event frame through a residual structure to obtain a first temporal fusion feature map; A second temporal fusion feature map is calculated based on the second similarity matrix A, the second similarity matrix B, the second similarity matrix C at the second scale and the second feature maps of the second scale corresponding to the first event frame, the second event frame, and the third event frame; A third temporal fusion feature map is calculated based on the third similarity matrix A, the third similarity matrix B, the third similarity matrix C at the third scale and the third feature maps of the third scale corresponding to the first event frame, the second event frame, and the third event frame; The multiple time series fusion feature maps are input into the decoupling head to obtain the defect detection results of each event frame.

2. The irregular defect detection method based on dynamic vision sensor and Yolox according to claim 1 is characterized in that: The step of inputting the first feature map into the path aggregation feature pyramid PAFPN to generate a second feature map of a first scale, a second feature map of a second scale, and a second feature map of a third scale includes: The first feature map is input into the path aggregation feature pyramid PAFPN to generate a second feature map of a first scale, a second feature map of a second scale, and a second feature map of a third scale, where the first scale is 80×80×256, the second scale is 40×40×512, and the third scale is 20×20×1024.

3. The irregular defect detection method based on dynamic vision sensor and Yolox according to claim 1, characterized in that: The step of inputting the multiple time series fusion feature maps into a decoupling head to obtain defect detection results for each event frame includes: Inputting the first temporal fusion feature map, the second temporal fusion feature map, and the third temporal fusion feature map into a decoupling head to obtain defect information of each event frame; the defect information includes at least: the predicted box position containing the defect and the defect category, where the defect category is one of "spot", "scratch", and "stain"; Determine the defect detection result according to the predicted box positions of defects of the same category in the first event frame, the second event frame, and the third event frame.

4. The irregular defect detection method based on dynamic vision sensor and Yolox according to claim 3 is characterized in that: The step of determining defect detection results according to the predicted box positions of defects of the same category in the first event frame, the second event frame, and the third event frame includes: Obtain the predicted box positions of defects of the same category in the first event frame and the second event frame and calculate the first intersection-over-union ratio between the two; Obtain the predicted box positions of defects of the same category in the second event frame and the third event frame and calculate the second intersection-over-union ratio between the two; The prediction box corresponding to the first intersection-and-union ratio less than the preset threshold in the second event frame is deleted, and the prediction box corresponding to the second intersection-and-union ratio less than the preset threshold in the second event frame is deleted, and the defect detection result is determined based on the remaining prediction boxes in the second event frame.

5. The irregular defect detection method based on dynamic vision sensor and Yolox according to claim 1, characterized in that: The defect detection model is trained based on a preset loss function, which includes a classification loss function, a positioning loss function, and a VariFocal loss function.

Images