Event camera based workpiece defect detection method and apparatus

Abstract

The application provides a workpiece defect detection method and device based on an event camera, and relates to the technical field of computer vision.The method comprises the following steps: based on the event camera, collecting an event data stream corresponding to the surface of a target workpiece in a current period, and obtaining a surface image sequence of the target workpiece according to the event data stream; inputting the surface image sequence into a defect detection model to obtain initial defect position information of the target workpiece, a defect type and a defect feature map corresponding to the initial defect position information; inputting the defect feature map into a defect region correction model to obtain final defect position information of the target workpiece; and obtaining a defect detection result of the target workpiece according to the defect type and the final defect position information.The application realizes accurate capture of defect information of a workpiece and accurate positioning of a defect position of the workpiece.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a workpiece defect detection method and device based on an event camera. BACKGROUND

[0002] With the development and progress of industry, intelligent manufacturing related technologies have also developed rapidly. In industrial manufacturing, defects in workpieces not only affect the appearance of products, but also have a negative impact on their performance, so workpiece defect detection is an important part of controlling product quality.

[0003] Existing workpiece defect detection usually adopts a traditional machine vision based scheme, which uses a traditional camera as an imaging device to collect single-frame images of workpieces to achieve workpiece defect detection. However, the CCD (Charge Coupled Device) or CMOS (Complementary Metal-Oxide-Semiconductor) sensor used by the traditional camera has strict requirements for imaging conditions, while the imaging environment in actual industrial production is complex, and the light interference also has uncertainty. Therefore, it is difficult to capture defects in workpieces and accurately locate them. SUMMARY

[0004] The present application provides a workpiece defect detection method and device based on an event camera to solve the problem that the prior art uses a traditional camera as an imaging device to collect single-frame images of workpieces to achieve workpiece defect detection, which is difficult to capture defects in workpieces and accurately locate them, and to achieve efficient and accurate defect detection of workpieces.

[0005] The present application provides a workpiece defect detection method based on an event camera, comprising:

[0006] Based on the event camera, event data streams corresponding to the surface of the target workpiece in the current period are collected, and according to the event data streams, a surface image sequence of the target workpiece is obtained;

[0007] The surface image sequence is input into a defect detection model to obtain initial defect position information, defect types of the target workpiece, and a defect feature map corresponding to the initial defect position information;

[0008] The defect feature map is input into a defect region correction model to obtain final defect position information of the target workpiece;

[0009] According to the defect types and the final defect position information, a defect detection result of the target workpiece is obtained;

[0010] The defect detection model is obtained by training based on a surface image sequence of a sample workpiece, a defect position label and a defect type label of the sample workpiece; and the defect region correction model is obtained by training based on a defect feature map corresponding to initial defect position information of the sample workpiece obtained by the defect detection model and the defect position label of the sample workpiece.

[0011] According to the workpiece defect detection method based on an event camera provided in the application, the defect detection model comprises a backbone network, a feature pyramid and an output module.

[0012] The surface image sequence of the target workpiece is input into the defect detection model to obtain initial defect position information, a defect type and a defect feature map corresponding to the initial defect position information of the target workpiece.

[0013] The surface image sequence of the target workpiece is input into the backbone network to obtain a multi-resolution feature matrix of the target workpiece.

[0014] The multi-resolution feature matrix is input into the feature pyramid to obtain a feature map of the target workpiece.

[0015] The feature map is input into the output module to obtain initial defect position information and a defect type of the target workpiece.

[0016] According to the initial defect position information of the target workpiece, a defect feature map corresponding to the initial defect position information of the target workpiece is obtained in the feature map.

[0017] According to the workpiece defect detection method based on an event camera provided in the application, the backbone network comprises a first feature extraction module, a second feature extraction module, a third feature extraction module and a fourth feature extraction module.

[0018] The surface image sequence of the target workpiece is input into the backbone network to obtain a multi-resolution feature matrix of the target workpiece, which comprises:

[0019] The surface image sequence of the target workpiece is input into the first feature extraction module to obtain a first feature matrix, the first feature matrix is input into the second feature extraction module to obtain a second feature matrix, the second feature matrix is input into the third feature extraction module to obtain a third feature matrix, and the third feature matrix is input into the fourth feature extraction module to obtain a fourth feature matrix.

[0020] The multi-resolution feature matrix is obtained according to the second feature matrix, the third feature matrix and the fourth feature matrix.

[0021] The application provides a workpiece defect detection method based on an event camera, and the feature pyramid comprises a first up-sampling module and a second up-sampling module.

[0022] The method comprises the following steps:

[0023] The fourth feature matrix is input into the first up-sampling module to obtain a fifth feature matrix.

[0024] The fifth feature matrix and the third feature matrix are fused to obtain a sixth feature matrix.

[0025] The sixth feature matrix is input into the second up-sampling module to obtain a seventh feature matrix.

[0026] The seventh feature matrix and the second feature matrix are fused to obtain the feature map of the target workpiece.

[0027] The application provides a workpiece defect detection method based on an event camera, and the method comprises the following steps:

[0028] According to the event timestamp of each event data in the event data stream, the image frame to which the event data belongs is determined.

[0029] According to the event grayscale value and event coordinates of each event data in each image frame, the surface image corresponding to the image frame is obtained.

[0030] According to the surface images corresponding to a plurality of different image frames, the surface image sequence of the target workpiece is obtained.

[0031] The application provides a workpiece defect detection method based on an event camera, and the method comprises the following steps:

[0032] The position information of the target workpiece in the surface image corresponding to each image frame is obtained.

[0033] According to the position information of the target workpiece, the local surface image of the target workpiece is cropped from the surface image corresponding to each image frame.

[0034] According to the local surface images corresponding to a plurality of different image frames, the surface image sequence of the target workpiece is generated.

[0035] According to the application, a workpiece defect detection method based on an event camera is provided, wherein a loss function of the defect detection model comprises a classification loss function corresponding to a defect type classification task and a regression loss function corresponding to a defect position prediction task.

[0036] The classification loss function is constructed based on a cross-entropy loss function, and the regression loss function is constructed based on a piecewise smooth loss function.

[0037] The application further provides a workpiece defect detection device based on an event camera, comprising:

[0038] The acquisition module is configured to acquire an event data stream corresponding to a surface of a target workpiece in a current period based on an event camera, and acquire a surface image sequence of the target workpiece according to the event data stream.

[0039] The first detection module is configured to input the surface image sequence into a defect detection model to obtain initial defect position information of the target workpiece, a defect type and a defect feature map corresponding to the initial defect position information.

[0040] The correction module is configured to input the defect feature map into a defect region correction model to obtain final defect position information of the target workpiece.

[0041] The second detection module is configured to acquire a defect detection result of the target workpiece according to the defect type and the final defect position information.

[0042] The defect detection model is obtained by training based on a surface image sequence of a sample workpiece, a defect position label and a defect type label of the sample workpiece; and the defect region correction model is obtained by training based on a defect feature map corresponding to initial defect position information of the sample workpiece obtained by the defect detection model and the defect position label of the sample workpiece.

[0043] The application further provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the workpiece defect detection method based on an event camera according to any one of the above when executing the program.

[0044] The application further provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the workpiece defect detection method based on an event camera according to any one of the above.

[0045] The application further provides a computer program product comprising a computer program, wherein the computer program is executable on a processor to implement the workpiece defect detection method based on an event camera according to any one of the above.

[0046] The event camera-based workpiece defect detection method and device provided by the application can effectively extract the fine defect event data stream of the workpiece surface disturbed by ambient light by using the characteristics of the event camera, such as small data redundancy and strong anti-light interference capability, and process the event data stream, so as to obtain a continuous multi-frame surface image sequence as input information of a defect detection model, so that the defect detection model filters out random noise according to context information in the continuous frame surface image sequence, and outputs accurate initial defect position information, a defect type and a defect feature map corresponding to the initial defect position information, and inputs the defect feature map corresponding to the initial defect position information into a defect region correction model, so as to further improve the detection accuracy of the defect position information, and then accurately capture the defect information of the workpiece and accurately locate the defect position of the workpiece. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0048] Figure 1 is one of the flowcharts of the event camera-based workpiece defect detection method provided by the application;

[0049] Figure 2 is another flowchart of the event camera-based workpiece defect detection method provided by the application;

[0050] Figure 3 is a structural schematic diagram of the defect region correction model in the event camera-based workpiece defect detection method provided by the application;

[0051] Figure 4 is a structural schematic diagram of the defect detection model in the event camera-based workpiece defect detection method provided by the application;

[0052] Figure 5 is a flowchart of the event data stream processing in the event camera-based workpiece defect detection method provided by the application;

[0053] Figure 6 is one of the simulation result schematic diagrams of the event camera-based workpiece defect detection method provided by the application;

[0054] Figure 7 is another simulation result schematic diagram of the event camera-based workpiece defect detection method provided by the application;

[0055] Figure 8is a structural schematic diagram of a workpiece defect detection device based on an event camera provided by the present application.

[0056] Figure 9 is a structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0057] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions in the present application will be described below in conjunction with the accompanying drawings in the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0058] It should be noted that the workpiece defect detection method in the present embodiment can be applied to various workpieces, including but not limited to aluminum substrates and flanges. In the following, the workpiece defect detection method in the present embodiment is described taking the aluminum substrate as an example.

[0059] The magnetic head is one of the core components of the hard disk, and the aluminum substrate is an important raw material for the magnetic head. The defect detection link of the aluminum substrate is related to the final quality of the hard disk. At present, the defect detection link of the aluminum substrate still relies on manual observation under high light for a long time, which not only has low efficiency, but also has uneven quality of defect detection and high labor cost.

[0060] In view of this, some scholars have proposed a traditional machine vision scheme for workpiece defect detection. However, this scheme has strict requirements for imaging conditions, and the imaging environment in actual industrial production is complex, and the light interference also has uncertainty, i.e., the traditional machine vision scheme can effectively capture workpiece defects when the defects are obvious, but it is difficult to find and accurately locate defects when the defect area is small or the light interference is strong.

[0061] Since the event camera is based on energy difference for imaging, it has characteristics such as large dynamic range and less data redundancy, so it has unique advantages in the application of aluminum substrate defect detection. In addition, with the continuous progress of deep learning technology in recent years, its excellent feature extraction capability and powerful learning ability have been fully utilized, and it has also been widely concerned and applied in industrial defect detection. Therefore, in view of the above problems, the present embodiment provides a workpiece defect detection method based on an event camera, which can to some extent alleviate the imaging unclearness caused by extreme light and motion blur, and use the event camera for data acquisition to make up for the difficulty of traditional cameras in capturing defects under extreme light and motion blur, and combine deep learning algorithm for workpiece defect detection, which can effectively realize efficient and accurate defect detection of workpieces.

[0062] The following will be described in conjunction withFigures 1-7 The event camera-based workpiece defect detection method provided in the application is described.

[0063] As shown in Figure 1 Fig. 1 is one of the flow diagrams of the event camera-based workpiece defect detection method provided in the embodiment, and the method comprises the following steps:

[0064] In step 101, based on an event camera, event data streams corresponding to the surface of a target workpiece in a current period are collected, and a surface image sequence of the target workpiece is obtained according to the event data streams.

[0065] The event camera is based on energy difference imaging and has characteristics such as large dynamic range and less data redundancy, so it has unique advantages in workpiece defect detection. The event camera only records event data, that is, it outputs a piece of asynchronous sparse event data stream in a certain period.

[0066] Each period can be divided according to actual needs, and each period includes a plurality of sampling time points determined at a preset sampling frequency.

[0067] Optionally, when it is necessary to detect defects of the target workpiece, the event camera is aimed at the target workpiece, and a plurality of event data corresponding to the surface of the target workpiece are recorded at a preset sampling frequency in the current period to form an event data stream, that is, one event data is sampled at each sampling time point, so that a plurality of event data can be obtained in the current period, and then the plurality of event data are summarized to obtain the event data stream. Each event data at least includes an event gray value, an event timestamp and an event coordinate; the event coordinate includes an event horizontal coordinate and an event vertical coordinate.

[0068] After the event data stream is obtained, a plurality of event data in the event data stream at each interval of a preset timestamp interval can be merged and converted to form a frame of surface image, and then a plurality of surface images can be obtained, and a surface image sequence can be generated according to the plurality of surface images. It should be noted that the preset time interval can be set according to actual needs, for example, the preset timestamp interval step = 40 ms, that is, the frame is extracted according to the preset timestamp interval t = 40 ms, and the plurality of event data extracted by the frame are drawn in the form of image to obtain each frame of surface image.

[0069] It should be noted that the plurality of surface images included in the surface image sequence can be set according to actual needs, such as 10 consecutive surface images.

[0070] Furthermore, the method of merging multiple event data to form a surface image frame can be to sequentially perform surface image overlay based on the event grayscale value, event timestamp, and event coordinates in each event data, or to summarize the event grayscale value, event timestamp, and event coordinates of multiple event data, and perform unified surface image characterization based on the summary result. This embodiment does not specifically limit this method.

[0071] Furthermore, after acquiring multiple surface images, the multiple surface images can be directly used to construct a surface image sequence of the target workpiece, or the multiple surface images can be preprocessed such as image cropping before constructing a surface image sequence of the target workpiece. This embodiment does not specifically limit this.

[0072] Step 102: Input the surface image sequence into the defect detection model to obtain the initial defect location information, defect type, and defect feature map corresponding to the initial defect location information of the target workpiece; wherein, the defect detection model is trained based on the surface image sequence of the sample workpiece, the defect location label, and the defect type label of the sample workpiece;

[0073] The defect detection model is used to perform defect location prediction and defect type classification tasks to accurately detect the initial defect location information and defect type of the workpiece. The defect location information includes, but is not limited to, the width and height of the defect area. The defect type includes, but is not limited to, defect or normal, and may also include subcategories under the defect category, such as scratches, dents, protrusions or particles, etc. This embodiment does not specifically limit this.

[0074] The defect detection model can be built based on one or more deep learning networks, including but not limited to residual networks, pyramids, and fully connected networks. This embodiment does not specifically limit this.

[0075] Optionally, before performing step 102, a defect detection model needs to be pre-trained using the sample dataset to accurately and quickly detect the initial defect location information and defect type of various workpieces. Specific training steps include:

[0076] like Figure 2 As shown, the training phase of the defect detection model requires first creating a sample dataset; specifically, obtaining the surface image sequence of the sample workpiece, and labeling the defect location and defect type of the sample workpiece to obtain the defect location label and defect type label of the sample workpiece; and constructing the sample dataset based on the surface image sequence, defect location label and defect type label of the sample workpiece.

[0077] Then, the surface image sequence of the sample workpiece in the sample data set is input into the initial defect detection model to obtain a defect type prediction result and an initial defect position information prediction result of the sample workpiece output by the initial defect detection model, and the initial defect detection model is iteratively trained according to the defect type prediction result and the initial defect position information prediction result and the defect position label and the defect type label, so as to obtain the defect detection model, so that the defect detection model obtained by training can accurately and quickly detect the initial defect position information of various workpieces and also can accurately and quickly detect the defect type of various workpieces.

[0078] After obtaining the defect detection model, the multiple frames of surface images in the surface image sequence of the target workpiece can be first spliced in the channel dimension to obtain input information, and then the input information is input into the defect detection model, and the defect detection model performs a defect type classification task and a defect position prediction task for the target workpiece according to the input information to obtain the initial defect position information and the defect type of the target workpiece.

[0079] For example, the surface image sequence includes 10 continuous surface images, and the 10 continuous surface images in the surface image sequence can be spliced in the channel dimension, and the spliced data is taken as the input information of the defect detection model.

[0080] In addition, the initial defect position information and the feature map can also be input into an ROI_Pooling (Regions Of Interest Pooling, Region of Interest Pooling) network to extract a defect feature map corresponding to the initial defect position information.

[0081] In step 103, the defect feature map is input into a defect region correction model to obtain the final defect position information of the target workpiece; the defect region correction model is obtained based on the defect feature map corresponding to the initial defect position information of the sample workpiece obtained by the defect detection model and the defect position label of the sample workpiece.

[0082] The defect region correction model is used to fine-tune the initial defect position information, i.e., the width and height of the initial defect region, output by the defect detection model to obtain more accurate final defect position information.

[0083] The defect region correction model can also be generated based on one or more deep learning networks. Preferably, the defect region correction model is generated based on a VGG16 (Visual Geometry Group Network, 16-layer visual geometry group network) network.

[0084] For example, the surface image sequence includes 10 continuous surface images, and the 10 continuous surface images in the surface image sequence can be spliced in the channel dimension, and the spliced data is taken as the input information of the defect detection model. Figure 3As shown, it is a structural schematic diagram of a defect region correction model, which is generated based on two layers of convolution kernel (hereinafter referred to as conv) with a size of 3x3 and a channel number of 64, two layers of convolution kernel with a size of 3x3 and a channel number of 8, three layers of convolution kernel with a size of 3x3 and a channel number of 256, six layers of convolution kernel with a size of 3x3 and a channel number of 512, and a full connection layer (hereinafter referred to as FC) and an output layer. The output layer is used to output the width and height of the fine-tuned defect region, i.e., the final defect position information.

[0085] As shown in Figure 2 As shown in the figure, while the defect detection model is being trained, a defect region correction model that can accurately and quickly detect the final defect position information of various workpieces is also needed to be pre-trained using a sample data set. The specific training steps include:

[0086] First, a sample data set is created. Specifically, the surface image sequence of the sample workpiece is obtained, the surface image sequence of the sample workpiece is input into the trained defect detection model, the defect position of the sample workpiece is detected and the feature map is extracted to obtain the initial defect position information and the feature map of the sample workpiece, and the defect feature map corresponding to the initial defect position information in the feature map is obtained according to the initial defect position information of the sample workpiece, so as to use the defect feature map as the sample data. The sample data set is constructed according to the defect feature map of the sample workpiece and the defect position label of the sample workpiece.

[0087] Then, the defect feature map in the sample data set is input into the initial defect region correction model to obtain the final defect position information prediction result output by the initial defect region correction model. Based on the final defect position information prediction result and the defect position label, the initial defect region correction model is iteratively trained to obtain the defect region correction model, so that the trained defect region correction model can accurately and quickly detect more accurate defect position information of various workpieces.

[0088] After obtaining the defect region correction model, the defect feature map corresponding to the initial defect position information of the target workpiece obtained based on the defect detection model can be input into the defect region correction model. The defect region correction model adjusts the initial defect position information of the target workpiece obtained based on the defect detection model according to the defect feature map corresponding to the initial defect position information of the target workpiece, to obtain the accurate defect position information of the target workpiece, i.e., the final defect position information of the target workpiece.

[0089] Step 104, according to the defect type and the final defect position information, obtaining the defect detection result of the target workpiece;

[0090] Optionally, after obtaining the defect type and the final defect position information of the target workpiece, the defect type and the final defect position information of the target workpiece can be summarized and marked to obtain the defect detection result of the target workpiece.

[0091] The workpiece defect detection method based on the event camera provided in the embodiment can effectively extract the subtle defect event data stream of the workpiece surface disturbed by the ambient light by using the characteristics of the event camera, such as small data redundancy and strong anti-light interference capability, process the event data stream, obtain a continuous multi-frame surface image sequence as input information of a defect detection model, so that the defect detection model filters out random noise according to context information in the continuous frame surface image sequence, outputs accurate initial defect position information, a defect type and a defect feature map corresponding to the initial defect position information, and inputs the defect feature map corresponding to the initial defect position information into a defect region correction model, so as to further improve the detection accuracy of the defect position information, and then accurately capture the defect information of the workpiece and accurately locate the defect position of the workpiece.

[0092] In some embodiments, the defect detection model comprises a backbone network, a feature pyramid and an output module.

[0093] The inputting of the surface image sequence into the defect detection model to obtain the initial defect position information, the defect type of the target workpiece and the defect feature map corresponding to the initial defect position information comprises:

[0094] The inputting of the surface image sequence of the target workpiece into the backbone network to obtain the multi-resolution feature matrix of the target workpiece;

[0095] The inputting of the multi-resolution feature matrix into the feature pyramid to obtain the feature map of the target workpiece;

[0096] The inputting of the feature map into the output module to obtain the initial defect position information and the defect type of the target workpiece;

[0097] According to the initial defect position information of the target workpiece, the defect feature map corresponding to the initial defect position information of the target workpiece is obtained in the feature map.

[0098] The defect detection model comprises a backbone network, a feature pyramid and an output module connected in sequence; the backbone network is used to extract a multi-resolution feature matrix, and the specific structure can be generated based on one or more deep networks; the feature pyramid is used to up-sample and fuse the multi-resolution feature matrix, and the specific structure can also be generated based on one or more deep networks; and the output module is used to classify a defect type and predict initial defect position information according to the feature map output by the feature pyramid.

[0099] Optionally, the detecting step of the defect detection model specifically comprises:

[0100] Firstly, the multi-resolution feature matrix of the target workpiece is extracted from the surface image sequence of the target workpiece by each feature extraction module in the backbone network to obtain the multi-resolution feature matrix of the target workpiece; then, the multi-resolution feature matrix is up-sampled and feature fused by the feature pyramid to obtain the feature map of the target workpiece; and then, the initial defect position information and the defect type of the target workpiece are obtained by executing the defect type classification task and the defect position prediction task according to the feature map by the output module.

[0101] After obtaining the initial defect position information of the target workpiece, the defect feature map corresponding to the initial defect position information can be extracted from the feature map output by the feature pyramid.

[0102] In this embodiment, the powerful feature extraction capability and learning capability of the backbone network, the feature pyramid and the output module in the defect detection model can suppress random noise from the continuous frames and perform multi-resolution and multi-level feature extraction on the surface image sequence of the target workpiece to accurately locate the initial defect position information and identify the defect type.

[0103] In some embodiments, the backbone network comprises a first feature extraction module, a second feature extraction module, a third feature extraction module and a fourth feature extraction module.

[0104] The surface image sequence of the target workpiece is input into the backbone network to obtain the multi-resolution feature matrix of the target workpiece, comprising:

[0105] The surface image sequence of the target workpiece is input into the first feature extraction module to obtain a first feature matrix, the first feature matrix is input into the second feature extraction module to obtain a second feature matrix, the second feature matrix is input into the third feature extraction module to obtain a third feature matrix, and the third feature matrix is input into the fourth feature extraction module to obtain a fourth feature matrix.

[0106] The multi-resolution feature matrix is obtained according to the second feature matrix, the third feature matrix and the fourth feature matrix.

[0107] The backbone network can be generated based on ResNet18 (18-layer residual network); as shown in Figure 4 According to the residual structure of the backbone network, ResNet18 can be divided into four feature extraction modules, namely the first feature extraction module, the second feature extraction module, the third feature extraction module and the fourth feature extraction module.

[0108] The first feature extraction module includes two residual blocks; each of the second feature extraction module, the third feature extraction module and the fourth feature extraction module includes one residual block and a down-sampling convolution block.

[0109] Optionally, the feature extraction step of the backbone network specifically includes:

[0110] Firstly, the first feature extraction module is used to perform feature matrix extraction on the surface image sequence of the target workpiece to obtain a first feature matrix of the target workpiece; then, the second feature extraction module is used to further perform feature matrix extraction on the basis of the first feature matrix to obtain a second feature matrix of the target workpiece; then, the third feature extraction module is used to further perform feature matrix extraction on the basis of the second feature matrix to obtain a third feature matrix of the target workpiece; then, the fourth feature extraction module is used to further perform feature matrix extraction on the basis of the third feature matrix to obtain a fourth feature matrix of the target workpiece. The second feature matrix, the third feature matrix and the fourth feature matrix have different resolutions.

[0111] The multi-resolution feature matrix can be obtained by summarizing the extracted second feature matrix, the third feature matrix and the fourth feature matrix.

[0112] In the embodiment, the multi-resolution feature extraction on the surface image sequence of the target workpiece can be realized by relying on the plurality of feature extraction modules in the backbone network, so that more abundant defect detail features can be obtained, and the defect position information of the target workpiece can be accurately located and the defect type of the target workpiece can be recognized.

[0113] In some embodiments, the feature pyramid includes a first up-sampling module and a second up-sampling module;

[0114] The inputting of the multi-resolution feature matrix into the feature pyramid to obtain the feature map of the target workpiece includes:

[0115] The fourth feature matrix is inputted into the first up-sampling module to obtain a fifth feature matrix;

[0116] The fifth feature matrix and the third feature matrix are fused to obtain a sixth feature matrix;

[0117] The sixth feature matrix is inputted into the second up-sampling module to obtain a seventh feature matrix;

[0118] The seventh feature matrix and the second feature matrix are fused to obtain the feature map of the target workpiece.

[0119] The feature pyramid (Feature Pyramid Networks, FPN) is used to fuse feature matrices of different resolutions, so as to improve the detection robustness of the defect detection model for surface images of different resolutions, and thus improve the monitoring effect.

[0120] As shown in Figure 4 The feature pyramid includes a first upsampling module and a second upsampling module. Each of the first upsampling module and the second upsampling module includes a convolution layer with a 1x1 kernel and two upsampling layers.

[0121] Optionally, the feature fusion step of the feature pyramid specifically includes:

[0122] First, the fourth feature matrix output by the backbone network is subjected to 1x1 convolution and upsampling by the first upsampling module to obtain a fifth feature matrix; then, the third feature matrix output by the backbone network and the fifth feature matrix are subjected to feature fusion to obtain a sixth feature matrix; then, the sixth feature matrix is subjected to 1x1 convolution and upsampling by the second upsampling module to obtain a seventh feature matrix, and then the second feature matrix output by the backbone network and the seventh feature matrix are subjected to feature fusion to obtain a feature map of the target workpiece.

[0123] In this embodiment, the fusion of multi-resolution feature matrices can be realized by the multiple upsampling modules in the feature pyramid, so that the feature map obtained by the fusion contains more rich defect detail features and defect overall features, and thus the defect position information of the target workpiece can be accurately located and the defect type of the target workpiece can be identified.

[0124] In some embodiments, the surface image sequence of the target workpiece is obtained according to the event data stream, including:

[0125] According to the event timestamp of each event data in the event data stream, the image frame to which the event data belongs is determined;

[0126] According to the event gray value and event coordinates of each event data in each image frame, the surface image corresponding to the image frame is obtained;

[0127] According to the surface images corresponding to multiple different image frames, the surface image sequence of the target workpiece is obtained.

[0128] As shown in Figure 5 The step of obtaining the surface image sequence in step 101 further includes:

[0129] Firstly, a preset timestamp interval, such as 40 ms, is set, and each event data in the event data stream is merged into a corresponding image frame according to the preset timestamp interval and the event timestamp of each event data in the event data stream. For each event data in the same image frame, the event information of the event signal is depicted in the corresponding image frame according to the event grayscale value, event horizontal coordinate and event vertical coordinate of the event data, so that the event data in the same image frame can be converted into a surface image corresponding to the image frame. Then, the surface images corresponding to the plurality of different image frames obtained in the current period are summarized to obtain a surface image sequence of the target workpiece, and the surface image sequence is input into the defect detection model and the defect region correction model, so that the defect detection result of the target workpiece can be quickly and accurately obtained.

[0130] In industrial applications, the event camera can also show the defect position when the light interference is strong, but at the same time, some random noise will be introduced in the non-defect position, that is, when the event signal is captured, there will be random noise interference, so a single frame of event image is easy to be disturbed by noise and is extremely easy to produce false detection, so it is difficult to accurately locate the position of the real defect. In the embodiment, the event data stream output by the event camera is processed according to the event timestamp, a plurality of surface images can be obtained, and a plurality of continuous surface images are selected as input information of the defect detection model, so that the model can filter out random noise according to the context information of the surface image sequence, and then accurately locate the real defect information of the workpiece.

[0131] In some embodiments, the surface image sequence of the target workpiece is obtained according to the surface images corresponding to the plurality of different image frames, including:

[0132] The position information of the target workpiece is obtained in the surface image corresponding to each image frame.

[0133] According to the position information of the target workpiece, the local surface image of the target workpiece is cropped in the surface image corresponding to each image frame.

[0134] The surface image sequence of the target workpiece is generated according to the local surface images corresponding to the plurality of different image frames.

[0135] Optionally, the step of obtaining the surface image sequence of the target workpiece further includes:

[0136] Since there are redundant non-workpiece areas in the original surface image obtained based on the event signal stream, in order to further improve the efficiency and accuracy of workpiece defect detection, an image detection algorithm can be used to detect the surface image corresponding to each image frame, determine the position information of the target workpiece in the surface image corresponding to each image frame, and then determine the region where the target workpiece is located in the surface image corresponding to each image frame based on the position information of the target workpiece. This region is then cropped out as a local surface image of the target workpiece. Then, the local surface images corresponding to multiple different image frames are summarized to obtain the surface image sequence of the target workpiece.

[0137] Furthermore, the image detection algorithm can be an image detection algorithm adapted to the target workpiece. For example, if the target workpiece is an aluminum substrate, since the aluminum substrate has a disk structure, Hough circle transform can be used to determine the local surface image of the aluminum substrate in the surface image corresponding to each image frame based on the surface image obtained for each image frame, and then the surface image sequence of the aluminum substrate can be obtained by summarizing.

[0138] In some embodiments, the loss function of the defect detection model includes a classification loss function corresponding to the defect type classification task and a regression loss function corresponding to the defect location prediction task;

[0139] The classification loss function is constructed based on the cross-entropy loss function, and the regression loss function is constructed based on the piecewise smoothing loss function.

[0140] Among them, the defect detection model is used to implement the defect type classification task and the defect location prediction task; accordingly, the loss function of the defect detection model can be jointly constructed based on the classification loss function corresponding to the defect type classification task and the regression loss function corresponding to the defect location prediction task.

[0141] Optionally, to eliminate the data imbalance between defective and normal workpiece samples in the training data, the classification loss function for the defect type classification task can be constructed using the cross-entropy loss function, with the specific calculation formula as follows:

[0142] Focal_Loss=-(1-P t ) γ ·log(P t );

[0143]

[0144] Where Focal_Loss is the loss function for the defect type classification task, p is the defect type prediction result of the sample workpiece output by the defect detection model, and y is the defect type label of the sample workpiece; P tFor the balanced defect type prediction result, γ is an adjustment factor.

[0145] In addition, in order to avoid the influence of outlier data on the defect detection model during the training process, improve the stability of the gradient value of the defect detection model during the training process, and further make the training smooth, the regression loss function corresponding to the defect position prediction task can be generated by using a segmented smooth loss function. The specific calculation formula is:

[0146]

[0147] Wherein, Smooth_L1_Loss is the regression loss function corresponding to the defect position prediction task, and x is the deviation value between the defect position information prediction result of the sample workpiece output by the defect detection model and the defect position label.

[0148] After obtaining the classification loss function corresponding to the defect type classification task and the regression loss function corresponding to the defect position prediction task, the classification loss function corresponding to the defect type classification task and the regression loss function corresponding to the defect position prediction task can be weighted and fused, or directly added and fused, to obtain the loss function of the defect detection model. Then, based on the loss function of the defect detection model, the defect detection model capable of accurately detecting the initial defect position information and the defect type is trained.

[0149] It should be noted that the defect region correction model is only used to correct the defect region, that is, only the width and height of the defect detection frame need to be further regressed. Therefore, the loss function used by the defect region correction model can also be generated based on the above segmented smooth loss function.

[0150] In order to further verify the effectiveness of the workpiece defect detection method based on the event camera provided in the embodiment, the embodiment adopts specific experimental data for simulation verification.

[0151] Among them, the experimental workpiece is an aluminum wafer, and the labeled sample aluminum wafer defect data in the training set is used to train the defect detection model and the defect region correction model, and the unlabeled target aluminum wafer defect data in the test set is used to test the effectiveness of the method.

[0152] As Figure 6The figure shows the PR (Precision-Recall) curves of the defect detection model trained on the test set when using a single frame surface image as input and when using a sequence of 10 consecutive frames of surface images as input. The horizontal axis of the PR curve represents recall, and the vertical axis represents precision. The PR curve is frequently used to characterize the performance of a model in object detection tasks. The PR curve represents a trend graph with Recall and Precision as the horizontal and vertical axes in a two-dimensional coordinate system. The area under the PR curve is the most commonly used performance metric in object detection (hereinafter referred to as mAP); a larger mAP indicates a better model performance. Figure 6 It can be seen that the accuracy of using only a single frame as input is generally low, while the area under the PR curve is significantly improved when using 10 consecutive frames as input. This is because using consecutive frames allows the model to filter out random noise on a single frame, thereby greatly improving the model's accuracy. Thus, the effectiveness and advancement of this embodiment are effectively verified.

[0153] like Figure 7 The figure shows the PR curves before and after adding the location correction model. It can be seen that the recall rate of the PR curve is significantly improved after adding the location correction model. This is because the location correction module makes the defect location more accurate, thus further verifying the advancement of this embodiment.

[0154] In summary, the above experiments effectively demonstrate the effectiveness of the defect detection method provided in this embodiment.

[0155] The workpiece defect detection device based on an event camera provided by the present invention will be described below. The workpiece defect detection device based on an event camera described below can be referred to in correspondence with the workpiece defect detection method based on an event camera described above.

[0156] like Figure 8 As shown, this embodiment provides a workpiece defect detection device based on an event camera. The device includes:

[0157] The acquisition module 801 is used to acquire event data streams corresponding to the surface of the target workpiece within the current period based on the event camera, and to obtain the surface image sequence of the target workpiece according to the event data streams;

[0158] The first detection module 802 is used to input the surface image sequence into the defect detection model to obtain the initial defect location information, defect type, and defect feature map corresponding to the initial defect location information of the target workpiece.

[0159] The correction module 803 is configured to input the defect feature map into a defect area correction model to obtain final defect position information of the target workpiece;

[0160] The second detection module 804 is configured to obtain a defect detection result of the target workpiece according to the defect type and the final defect position information.

[0161] The defect detection model is obtained based on a surface image sequence of a sample workpiece, defect position labels and defect type labels of the sample workpiece; and the defect area correction model is obtained based on a defect feature map corresponding to initial defect position information of the sample workpiece obtained by the defect detection model and the defect position labels of the sample workpiece.

[0162] The workpiece defect detection device based on the event camera provided in this embodiment can effectively extract the subtle defect event data stream of the workpiece surface disturbed by the ambient light by using the characteristics of the event camera, such as small data redundancy and strong anti-light interference capability, and process the event data stream. A continuous multi-frame surface image sequence can be obtained as input information of the defect detection model, so that the defect detection model can filter out random noise according to the context information in the continuous frame surface image sequence, output accurate initial defect position information, defect type and defect feature map corresponding to the initial defect position information, and input the defect feature map corresponding to the initial defect position information into the defect area correction model to further improve the detection accuracy of the defect position information, thereby accurately capturing the defect information of the workpiece and accurately positioning the defect position of the workpiece.

[0163] Figure 9 An example of an entity structure diagram of an electronic device is shown in FIG. 1. Figure 9As shown, the electronic device can include a processor 901, a communications interface 902, a memory 903, and a communications bus 904, wherein the processor 901, the communications interface 902, and the memory 903 complete mutual communication through the communications bus 904. The processor 901 can invoke a logical instruction in the memory 903 to execute an event camera-based workpiece defect detection method, which includes: based on an event camera, collecting an event data stream corresponding to a surface of a target workpiece in a current period, and obtaining a surface image sequence of the target workpiece according to the event data stream; inputting the surface image sequence into a defect detection model to obtain initial defect position information, a defect type, and a defect feature map corresponding to the initial defect position information of the target workpiece; inputting the defect feature map into a defect region correction model to obtain final defect position information of the target workpiece; and obtaining a defect detection result of the target workpiece according to the defect type and the final defect position information; wherein the defect detection model is obtained by training based on a surface image sequence of a sample workpiece, defect position labels, and defect type labels of the sample workpiece; and the defect region correction model is obtained by training based on a defect feature map corresponding to the initial defect position information of the sample workpiece obtained by the defect detection model, and the defect position labels of the sample workpiece.

[0164] In addition, the logical instruction in the memory 903 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application or parts of the present application that essentially contribute or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0165] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program being stored in a non-transitory computer-readable storage medium, and the computer program being executable by a processor to enable a computer to perform the event camera-based workpiece defect detection method provided by any of the above methods, the method comprising: acquiring, based on an event camera, an event data stream corresponding to a surface of a target workpiece in a current period, and obtaining a surface image sequence of the target workpiece according to the event data stream; inputting the surface image sequence into a defect detection model to obtain initial defect position information, a defect type of the target workpiece, and a defect feature map corresponding to the initial defect position information; inputting the defect feature map into a defect region correction model to obtain final defect position information of the target workpiece; and obtaining a defect detection result of the target workpiece according to the defect type and the final defect position information; wherein the defect detection model is obtained by training based on a surface image sequence of a sample workpiece, defect position labels and defect type labels of the sample workpiece; and the defect region correction model is obtained by training based on a defect feature map corresponding to the initial defect position information of the sample workpiece obtained by the defect detection model, and the defect position labels of the sample workpiece.

[0166] In yet another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement an event camera-based workpiece defect detection method provided by any of the above methods, the method comprising: acquiring, based on an event camera, an event data stream corresponding to a surface of a target workpiece in a current period, and obtaining a surface image sequence of the target workpiece according to the event data stream; inputting the surface image sequence into a defect detection model to obtain initial defect position information, a defect type of the target workpiece, and a defect feature map corresponding to the initial defect position information; inputting the defect feature map into a defect region correction model to obtain final defect position information of the target workpiece; and obtaining a defect detection result of the target workpiece according to the defect type and the final defect position information; wherein the defect detection model is obtained by training based on a surface image sequence of a sample workpiece, defect position labels and defect type labels of the sample workpiece; and the defect region correction model is obtained by training based on a defect feature map corresponding to the initial defect position information of the sample workpiece obtained by the defect detection model, and the defect position labels of the sample workpiece.

[0167] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0168] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0169] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for detecting defects in a workpiece based on an event camera, the method comprising: The method comprises: acquiring an event data stream corresponding to a surface of a target workpiece in a current period based on an event camera, and obtaining a surface image sequence of the target workpiece according to the event data stream; inputting the surface image sequence into a defect detection model to obtain initial defect position information, a defect type of the target workpiece, and a defect feature map corresponding to the initial defect position information; inputting the defect feature map into a defect region correction model to obtain final defect position information of the target workpiece; wherein the defect region correction model is generated based on a VGG16 network, and the defect region correction model determines the width and height of a fine-tuned defect region based on the defect feature map to obtain the final defect position information; obtaining a defect detection result of the target workpiece according to the defect type and the final defect position information; wherein the defect detection model is obtained by training based on a surface image sequence of a sample workpiece, defect position labels and defect type labels of the sample workpiece; and the defect region correction model is obtained by training based on a defect feature map corresponding to the initial defect position information of the sample workpiece obtained by the defect detection model and the defect position labels of the sample workpiece.

2. The event camera based workpiece defect detection method of claim 1, wherein, The defect detection model comprises a backbone network, a feature pyramid and an output module. The method comprises: inputting the surface image sequence of the target workpiece into the backbone network to obtain a multi-resolution feature matrix of the target workpiece; inputting the multi-resolution feature matrix into the feature pyramid to obtain a feature map of the target workpiece; inputting the feature map into the output module to obtain the initial defect position information and the defect type of the target workpiece; obtaining a defect feature map corresponding to the initial defect position information of the target workpiece in the feature map according to the initial defect position information of the target workpiece.

3. The event camera based workpiece defect detection method of claim 2, wherein, The backbone network comprises a first feature extraction module, a second feature extraction module, a third feature extraction module and a fourth feature extraction module. The method comprises: inputting the surface image sequence of the target workpiece into the first feature extraction module to obtain a first feature matrix, inputting the first feature matrix into the second feature extraction module to obtain a second feature matrix, inputting the second feature matrix into the third feature extraction module to obtain a third feature matrix, inputting the third feature matrix into the fourth feature extraction module to obtain a fourth feature matrix; obtaining the multi-resolution feature matrix according to the second feature matrix, the third feature matrix and the fourth feature matrix.

4. The event camera based workpiece defect detection method of claim 3, wherein, The feature pyramid comprises a first upsampling module and a second upsampling module. The method comprises: inputting the multi-resolution feature matrix into the feature pyramid to obtain a feature map of the target workpiece, comprising: input the fourth feature matrix into the first up-sampling module to obtain a fifth feature matrix; fuse the fifth feature matrix and the third feature matrix to obtain a sixth feature matrix; input the sixth feature matrix into the second up-sampling module to obtain a seventh feature matrix; fuse the seventh feature matrix and the second feature matrix to obtain a feature map of the target workpiece.

5. The event camera based workpiece defect detection method of any of claims 1-4, wherein, The surface image sequence of the target workpiece is obtained according to the event data stream, including: determining the image frame to which each event data in the event data stream belongs according to the event timestamp of each event data; obtaining the surface image corresponding to each image frame according to the event gray value and event coordinates of each event data in each image frame; obtaining the surface image sequence of the target workpiece according to the surface images corresponding to multiple different image frames.

6. The event camera based workpiece defect detection method of claim 5, wherein, The surface image sequence of the target workpiece is obtained according to the surface images corresponding to multiple different image frames, including: obtaining the position information of the target workpiece in the surface image corresponding to each image frame; cropping the local surface image of the target workpiece in the surface image corresponding to each image frame according to the position information of the target workpiece; generating the surface image sequence of the target workpiece according to the local surface images corresponding to multiple different image frames.

7. The event camera based workpiece defect detection method of any of claims 1-4, wherein, The loss function of the defect detection model includes a classification loss function corresponding to a defect type classification task and a regression loss function corresponding to a defect position prediction task; The classification loss function is constructed based on a cross-entropy loss function, and the regression loss function is constructed based on a piecewise smooth loss function.

8. An event camera based workpiece defect detection apparatus, comprising: including: The acquisition module is configured to acquire an event data stream corresponding to a surface of a target workpiece in a current period based on an event camera, and obtain a surface image sequence of the target workpiece according to the event data stream. The first detection module is configured to input the surface image sequence into a defect detection model to obtain initial defect position information of the target workpiece, a defect type, and a defect feature map corresponding to the initial defect position information. The correction module is configured to input the defect feature map into a defect region correction model to obtain final defect position information of the target workpiece, wherein the defect region correction model is generated based on a VGG16 network, and the defect region correction model determines the width and height of a fine-tuned defect region based on the defect feature map to obtain the final defect position information. The second detection module is configured to obtain a defect detection result of the target workpiece according to the defect type and the final defect position information. The defect detection model is trained based on a surface image sequence of a sample workpiece, a defect position label and a defect type label of the sample workpiece; and the defect region correction model is trained based on a defect feature map corresponding to initial defect position information of the sample workpiece obtained by the defect detection model and the defect position label of the sample workpiece.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the event camera-based workpiece defect detection method of any one of claims 1-7 when executing the program.

10. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the event camera based workpiece defect detection method according to any one of claims 1 to 7.

Images