A defect detection method and system based on dynamic vision sensor

Abstract

The present invention relates to a defect detection method and system based on a dynamic vision sensor. The system comprises: a data acquisition module, a defect detection module, a data filtering module, and a data visualization module. The data acquisition module is used to acquire detection images, obtain grayscale information change data, and synthesize them into event frame images. The defect detection module detects the event frame images using a pre-trained defect detection model to obtain defect detection results. The data filtering module is used to sort the defect detection results and determine the final detection result. The data visualization module performs visualization processing and outputs the type, confidence level, and coordinates of the defect. The present invention uses a dynamic vision processor to synthesize a certain number of events into an event frame, which is then input into a convolutional neural network. Feature extraction and feature fusion are then performed to further reduce the computational complexity of the convolutional neural network, resulting in improved system performance.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a defect detection method and system based on a dynamic vision sensor. Background Art

[0002] During the manufacturing process, surface defects are often unavoidable, such as deformation, burrs, scratches, and oil stains caused by various reasons during processing. Surface defects not only affect the product's aesthetics but also its performance. Therefore, surface defect detection is crucial. Surface defect detection not only effectively controls product quality but also allows analysis of the causes of defects based on the test results, enabling improvements to production processes and reducing the number of defective products.

[0003] Traditional surface inspection methods rely on manual inspection by the human eye. This method is inefficient, lacks real-time performance, has low accuracy, is significantly affected by subjective factors, and suffers from significant under-recognition and missed detection of minor defects, making it difficult to meet actual production needs. Inspection methods based on machine vision can largely overcome these drawbacks. However, for certain scenarios, machine vision inspection based on traditional cameras also presents challenges. These include large data volumes, motion blur, and significant information redundancy. The resulting large data volumes place higher demands on back-end algorithms and hardware. Dynamic vision sensors offer advantages such as low power consumption, low latency, and a high dynamic range, making them easier for traditional cameras to detect and identify objects in complex backgrounds. Dynamic vision sensors also have the ability to filter critical information, reducing redundant data at the front end, significantly improving computational efficiency, and reducing the overall burden on back-end modules.

[0004] With the gradual maturity of deep learning technology, defect detection technology based on deep learning has become a research hotspot. This technology is combined with a defect detection system formed by dynamic vision sensors. By extracting features from images and generating feature maps for defect detection, it has stronger competitiveness in the field of surface defect detection of industrial products. In the paper "ADifferentiable Recurrent Surface for Asynchronous Event-Based Data" (2020arXiv.org), Marco Cannici et al. proposed a method of using a long short-term memory network to extract features from the output event stream of a dynamic vision sensor, and using a convolutional neural network to perform target recognition on the feature extraction results, further improving the speed and accuracy of image processing. However, this method still has shortcomings: in the process of extracting features from the event stream output by the dynamic vision sensor, the features are extracted for all events in sequence. In this process, redundant information needs to be introduced, which greatly increases the computational complexity of the convolutional neural network. Summary of the Invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a defect detection method and system based on a dynamic vision sensor. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] The present invention provides a defect detection method based on a dynamic vision sensor, comprising:

[0007] Collect multiple detection images of the detected sample within one motion cycle;

[0008] Obtaining corresponding grayscale information change data according to the multiple detection images;

[0009] Synthesizing the grayscale information change data to obtain a corresponding event frame image;

[0010] Inputting the event frame image into the pre-trained defect detection model to obtain the corresponding defect detection result;

[0011] According to the defect detection results corresponding to all event frame images, the final detection result of the detected sample is determined and visualized.

[0012] In one embodiment of the present invention, the pre-training of the defect detection model includes:

[0013] (a) Obtain defect training dataset T1:

[0014] The collected event frame image S is divided into a training set and a test set for defect annotation to obtain the defect dataset T1;

[0015] (b) Construct a convolutional neural network F that includes a feature extraction module, a feature fusion module, and a detection and recognition module;

[0016] (c) From the defect dataset T1, randomly select the event frame images labeled in the training set as the input of the convolutional neural network. The loss value of the convolutional neural network is:

[0017] Loss=Loss box +Loss obj +Loss cls ;

[0018] Among them, Loss is the loss value of the convolutional neural network, Loss box The error caused by the prediction frame, Loss obj The error caused by confidence, Loss cls Error caused by the category;

[0019] (d) Use the gradient descent method to update the parameters of the convolutional neural network F through the loss value Loss of the convolutional neural network. The parameter update formula is:

[0020]

[0021]

[0022] Among them, ω u represents the parameters of the convolutional layer before updating, ω u+1 represents the updated parameters of the convolutional layer, θ u represents the parameters of the fully connected layer before updating, θ u+1 Represents the updated parameters of the fully connected layer;

[0023] (4e) The convolutional neural network F is iteratively trained to obtain a pre-trained defect detection model.

[0024] In one embodiment of the present invention, the feature fusion module is used to combine the feature maps F of different scales output by the feature extraction module. a1 、F a2 、F a3 Perform feature fusion, first transfer and fuse the high-level feature information through upsampling, and then fuse it through downsampling to obtain the predicted feature map F b1 、F b2 、F b3 .

[0025] In one embodiment of the present invention, determining the final detection result of the inspected sample based on the defect detection results corresponding to all event frame images and performing visualization processing includes:

[0026] The defect detection results of the inspected sample within a motion cycle are stored, the predicted category sequence is counted, and the category with the largest number is taken as the final detection category. In this category, the frame image with the highest confidence is taken as the final detection result and visualized.

[0027] The present invention also provides a defect detection system based on a dynamic visual sensor, comprising: a data acquisition module, a defect detection module, a data filtering module and a data visualization module;

[0028] The data acquisition module is used to collect multiple detection images of the detected sample within a movement cycle, obtain corresponding grayscale information change data, and synthesize the grayscale information change data to obtain the corresponding event frame image;

[0029] The defect detection module detects the event frame image using a pre-trained defect detection model to obtain the corresponding defect detection result;

[0030] The data filtering module sorts the defect detection results within a cycle and determines the final detection results of the detected samples;

[0031] The data visualization module is used to visualize the final detection result and output the type, confidence level and coordinates of the identified defects.

[0032] In one embodiment of the present invention, the data acquisition module includes: a robotic arm, a dynamic vision sensor, and a dynamic vision processor;

[0033] The dynamic vision sensor is used to collect images, compare the grayscale information of the images and output grayscale information change data;

[0034] The robotic arm is used to clamp the sample to be tested and perform periodic reciprocating motion in front of the dynamic vision sensor;

[0035] The dynamic vision processor is used to analyze the grayscale information change data and synthesize it into an event frame image.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] 1. The dynamic vision sensor-based defect detection method and system of the present invention uses a dynamic vision sensor to avoid the overexposure phenomenon of traditional cameras due to information redundancy. By compressing the data stream into event frames, the amount of data input to the network is greatly reduced, the detection speed is improved, and it is robust against complex lighting scenes.

[0038] 2. This invention uses convolutional neural networks for defect detection and identification. Convolutional neural networks offer higher computational efficiency on GPUs, improving computational performance compared to existing technologies using spiking neural networks. Compared to existing convolutional neural network-based object recognition methods, this method first uses a dynamic vision processor to synthesize a certain number of events into a single event frame, which is then input into the convolutional neural network. Feature extraction and fusion further reduce the computational complexity of the convolutional neural network. Finally, a data filtering module is added to reduce the amount of data required for visualization and data storage, resulting in improved system performance.

[0039] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following preferred embodiments are specifically cited and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1This is a flow chart of an implementation of a defect detection method based on a dynamic vision sensor provided by an embodiment of the present invention;

[0041] Figure 2 It is a schematic diagram of the overall structure of a defect detection system based on a dynamic vision sensor provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the following is a detailed description of a defect detection method and system based on a dynamic vision sensor proposed in accordance with the present invention, in combination with the accompanying drawings and specific implementation methods.

[0043] The aforementioned and other technical contents, features, and effects of the present invention are clearly presented in the following detailed description of the specific embodiments in conjunction with the accompanying drawings. Through the description of the specific embodiments, a deeper and more specific understanding of the technical means and effects adopted by the present invention to achieve the intended purpose can be obtained. However, the accompanying drawings are provided for reference and illustration purposes only and are not intended to limit the technical solutions of the present invention.

[0044] Example 1

[0045] See Figure 1 , Figure 1 This is a flowchart of the implementation of the defect detection method based on dynamic vision sensors provided by an embodiment of the present invention. As shown in the figure, the defect detection method based on dynamic vision sensors of the present invention includes:

[0046] Step 1: Collect multiple detection images of the detected sample within one motion cycle;

[0047] Among them, one motion cycle refers to a back-and-forth motion of the detected sample in front of the dynamic vision sensor. In this embodiment, the dynamic vision sensor is used to collect images. Since the dynamic vision sensor is a differential imaging of energy, the detection image set can only be collected when there is relative motion between the imaged monitoring sample and the dynamic vision sensor.

[0048] Step 2: Obtain corresponding grayscale information change data based on multiple detection images;

[0049] Step 3: Synthesize the grayscale information change data to obtain the corresponding event frame image;

[0050] Among them, the event frame image is synthesized by the grayscale information change data of multiple detection images at multiple moments, rather than separately collected images. The purpose is to clearly identify the imaging object by synthesizing a certain amount of grayscale information change data.

[0051] Step 4: Input the event frame image into the pre-trained defect detection model to obtain the corresponding defect detection results;

[0052] In this embodiment, the pre-training of the defect detection model includes:

[0053] (a) Obtain defect training dataset T1:

[0054] The collected event frame image S is divided into a training set and a test set for defect annotation to obtain the defect dataset T1;

[0055] (b) Construct a convolutional neural network F that includes a feature extraction module, a feature fusion module, and a detection and recognition module;

[0056] (c) From the defect dataset T1, randomly select the event frame images labeled in the training set as the input of the convolutional neural network. The loss value of the convolutional neural network is:

[0057] Loss=Loss box +Loss obj +Loss cls ;

[0058] Among them, Loss is the loss value of the convolutional neural network, Loss box Loss is the error caused by the prediction frame. obj The error caused by confidence, Loss cls Error caused by the category;

[0059] (d) Use the gradient descent method to update the parameters of the convolutional neural network F through the loss value Loss of the convolutional neural network. The parameter update formula is:

[0060]

[0061]

[0062] Among them, ω u represents the parameters of the convolutional layer before updating, ω u+1 represents the updated parameters of the convolutional layer, θ u represents the parameters of the fully connected layer before updating, θ u+1 Represents the updated parameters of the fully connected layer;

[0063] (4e) The convolutional neural network F is iteratively trained to obtain a pre-trained defect detection model.

[0064] In this embodiment, the feature fusion module is used to combine the feature maps F of different scales output by the feature extraction module. a1 、F a2 、F a3Perform feature fusion, first transfer and fuse the high-level feature information through upsampling, and then fuse it through downsampling to obtain the predicted feature map F b1 、F b2 、F b3 .

[0065] Step 5: Based on the defect detection results corresponding to all event frame images, determine the final detection results of the inspected sample and perform visualization processing.

[0066] In this embodiment, step 5 includes:

[0067] The defect detection results of the inspected sample within a motion cycle are stored, the predicted category sequence is counted, and the category with the largest number is taken as the final detection category. In this category, the frame image with the highest confidence is taken as the final detection result and visualized.

[0068] See Figure 2 , Figure 2 It is a schematic diagram of the overall structure of a defect detection system based on a dynamic vision sensor provided by an embodiment of the present invention.

[0069] As shown in the figure, the defect detection system based on dynamic visual sensors of this embodiment includes: a data acquisition module, a defect detection module, a data filtering module and a data visualization module.

[0070] In this embodiment, the data acquisition module is used to collect multiple detection images of the detected sample within a motion cycle, obtain corresponding grayscale information change data, and synthesize the grayscale information change data to obtain the corresponding event frame image.

[0071] Optionally, the data acquisition module includes a robotic arm, a dynamic vision sensor, and a dynamic vision processor. The dynamic vision sensor is used to acquire images, compare the grayscale information of the images, and output grayscale information change data; the robotic arm is used to grasp the sample to be inspected and perform periodic reciprocating motion in front of the dynamic vision sensor; and the dynamic vision processor is used to analyze the grayscale information change data and synthesize it into an event frame image.

[0072] In an optional embodiment, the dynamic vision sensor obtains the grayscale information change signal P of each image after focusing. r :

[0073] By P r Get the grayscale information change event stream C of multiple events r

[0074] C r ={c r,l =(x r,l ,yr,l ,g r,l ,t r,l )|0 <l≤N r}, N r >0;

[0075] Among them, P r is the set of grayscale information changes of a single event at this moment and the previous moment, C r is the grayscale information change event stream of multiple events, c r,l represents the lth event, x r,l c r,l The horizontal coordinate value of the trigger position, y r,l c r,l The vertical coordinate value of the trigger position, g r,l c r,l Gray value, t r,l c r,l The trigger time, r is the moment, l is the random event within r moment, N r is the total number of events generated at time r.

[0076] In an optional embodiment, the dynamic vision processor is used to r Parse the data to obtain the address-event stream data, and combine a certain number of events into an event frame S:

[0077] Construct an all-zero matrix M = zeros(H, W);

[0078] Where H is the number of high pixels of the event frame; W is the number of wide pixels of the event frame;

[0079] Take event stream C r The fixed number of events or the number of events with fixed time difference in are filled into the all-zero matrix M to obtain the event frame S.

[0080] Furthermore, a certain number of events are continuous fixed events in the event stream, and the specific number is determined according to the specific imaging object. The purpose is to synthesize a clear event frame S through a certain number of events to identify the imaging object.

[0081] In this embodiment, the defect detection module detects the event frame image using a pre-trained defect detection model to obtain corresponding defect detection results. Specifically, the pre-training process of the defect detection model is as described in the above method embodiment and will not be repeated here.

[0082] In this embodiment, the data filtering module is used to sort the defect detection results within a cycle and determine the final detection results of the detected samples.

[0083] Specifically, statistics are collected on the predicted category sequence of defect detection results of the stored inspected samples within a motion cycle, and a category with the largest number is taken as the final detection category. In this category, the frame image with the highest confidence is taken as the final detection result.

[0084] In this embodiment, the data visualization module is used to visualize the final detection results and output the type, confidence level, and coordinates of the identified defects.

[0085] The defect detection method and system based on dynamic vision sensors in the embodiments of the present invention adopt dynamic vision sensors to avoid the overexposure phenomenon of traditional cameras due to information redundancy. By compressing the data stream into event frames, the amount of data input to the network is greatly reduced, the detection speed is improved, and the system is robust to complex lighting scenes.

[0086] The dynamic vision sensor-based defect detection method and system of the present invention employs a convolutional neural network for defect detection and identification. Convolutional neural networks offer higher computational efficiency on GPUs, improving computational performance compared to existing technologies using spiking neural networks. Compared to existing convolutional neural network-based target recognition methods, this method first uses a dynamic vision processor to synthesize a certain number of events into a single event frame, which is then fed into the convolutional neural network. Feature extraction and fusion further reduce the computational complexity of the convolutional neural network. Finally, a data filtering module is added to reduce the amount of data required for visualization and data storage, resulting in improved system performance.

[0087] It should be noted that, in this document, relational terms such as first and second are used solely to distinguish one entity or operation from another, and do not necessarily require or imply any actual relationship or order between these entities or operations. Furthermore, the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that an article or device comprising a list of elements includes not only those elements but also other elements not explicitly listed. Without further limitation, an element defined by the phrase "comprising a..." does not preclude the presence of additional identical elements in the article or device comprising the element. Terms such as "connected" or "connected" are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. References to orientations or positional relationships, such as "upper," "lower," "left," and "right," are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate description and simplify the description of the present invention. They do not indicate or imply that the device or element referred to must have, be constructed, or operate in a specific orientation, and are therefore not to be construed as limiting the present invention.

[0088] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A defect detection method based on a dynamic vision sensor, characterized in that: include: Collect multiple detection images of the detected sample within one motion cycle; Obtaining corresponding grayscale information change data according to the multiple detection images; Synthesizing the grayscale information change data to obtain a corresponding event frame image; Inputting the event frame image into the pre-trained defect detection model to obtain the corresponding defect detection result; The pre-training of the defect detection model includes: (a) Obtaining defect training dataset : The captured event frame image , divided into training set and test set for defect annotation, and obtain defect data set ; (b) Construct a convolutional neural network including feature extraction module, feature fusion module and detection and recognition module The feature fusion module is used to extract the feature maps of different scales output by the feature extraction module. 、 、 Perform feature fusion, first transfer and fuse the high-level feature information through upsampling, and then fuse it through downsampling to obtain the predicted feature map 、 、 ; (c) From the defect dataset In the example, the event frame images labeled in the training set are randomly selected as the input of the convolutional neural network. The loss value of the convolutional neural network is: ； in, is the loss value of the convolutional neural network, is the error caused by the prediction frame part, is the error caused by the confidence level, Error caused by the category; (d) Loss value of convolutional neural network using gradient descent method Convolutional Neural Network The parameters are updated, and the parameter update formula is: ； ； in, represents the parameters of the convolutional layer before updating, represents the parameters of the updated convolutional layer, represents the parameters of the fully connected layer before updating, Represents the updated parameters of the fully connected layer; (e) Convolutional Neural Network Through iterative training, a pre-trained defect detection model is obtained; Based on the defect detection results corresponding to all event frame images, the final detection results of the inspected sample are determined and visualized, including: storing the defect detection results of the inspected sample within a motion cycle, statistically analyzing the predicted category sequence, taking the category with the largest number as the final detection category, and taking the frame image with the highest confidence in this category as the final detection result, and visually displaying it.

2. A defect detection system based on a dynamic vision sensor, applicable to the defect detection method based on a dynamic vision sensor according to claim 1, characterized in that: include: Data acquisition module, defect detection module, data filtering module and data visualization module; The data acquisition module is used to collect multiple detection images of the detected sample within a movement cycle, obtain corresponding grayscale information change data, and synthesize the grayscale information change data to obtain the corresponding event frame image; The defect detection module detects the event frame image using a pre-trained defect detection model to obtain the corresponding defect detection result; The data filtering module sorts the defect detection results within a cycle and determines the final detection results of the detected samples; The data visualization module is used to visualize the final detection result and output the type, confidence level and coordinates of the identified defects.

3. The defect detection system based on dynamic vision sensor according to claim 2 is characterized in that: The data acquisition module includes: a mechanical arm, a dynamic vision sensor and a dynamic vision processor; The dynamic vision sensor is used to collect images, compare the grayscale information of the images and output grayscale information change data; The robotic arm is used to clamp the sample to be tested and perform periodic reciprocating motion in front of the dynamic vision sensor; The dynamic vision processor is used to analyze the grayscale information change data and synthesize it into an event frame image.

Images