Surface defect detection method and equipment based on three-dimensional image

Abstract

The present invention provides a surface defect detection method and device based on three-dimensional images. The method comprises: acquiring structured three-dimensional image data of a target object; extracting primary features from the structured three-dimensional image data to obtain a primary feature map of the target object; performing multi-scale feature extraction on the primary feature map to obtain a multi-scale feature map of the same size as the primary feature map; processing the primary feature map using a pre-trained invalid pixel mask prediction network to obtain mask data of the same size as the primary feature map; performing feature cropping on the multi-scale feature map based on the mask data to obtain a feature map corresponding to valid pixels; extracting high-level features from the feature map corresponding to valid pixels to obtain a high-level feature map of the target object; and performing defect classification processing based on the high-level feature map to obtain a defect classification result for the target object. By filtering out invalid pixels at the feature level, both efficiency and accuracy are improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of image processing technology, and specifically to a surface defect detection method and device based on three-dimensional images. Background Art

[0002] Surface defects, such as spots, dents, scratches, nicks, and cracks, can affect not only the appearance but also the quality and performance of a product. Detecting surface defects is therefore crucial and widely used in industrial production, for example in weld inspection and wafer testing.

[0003] With the continuous development of image processing technology, the method of automated detection of surface defects based on image data has gradually replaced the early visual inspection method performed manually. With the complexity of machine vision scenes, 3D vision sensors have also gradually been put into practical applications for surface defect detection. The three-dimensional image data output by 3D vision sensors can provide richer information for surface defect detection, but due to the influence of the acquisition environment, such as occlusion, reflection, multiple reflections, etc., it is often impossible to obtain the height information of the pixel points. In other words, there are usually invalid pixels in the three-dimensional image data output by the 3D vision sensor. At present, invalid pixels in three-dimensional image data are mainly processed by image repair processing or image filling processing, which is not only inefficient but also reduces the accuracy of surface defect detection. Summary of the Invention

[0004] The embodiments of the present invention provide a surface defect detection method and device based on three-dimensional images, aiming to improve the accuracy of surface defect detection.

[0005] In a first aspect, an embodiment of the present invention provides a surface defect detection method based on three-dimensional images, comprising:

[0006] Acquire structured three-dimensional image data of the target object, the three-dimensional image data including height information, which is used to reflect the surface morphology information of the target object;

[0007] Extract primary features from structured three-dimensional image data to obtain a primary feature map of the target object;

[0008] Perform multi-scale feature extraction on the primary feature map to obtain a multi-scale feature map of the same size as the primary feature map;

[0009] The primary feature map is processed using a pre-trained invalid pixel mask prediction network to obtain mask data of the same size as the primary feature map. The mask data is used to identify the location information of invalid pixels and / or valid pixels.

[0010] Perform feature cropping on the multi-scale feature map according to the mask data, filter out the feature map corresponding to invalid pixels, and obtain the feature map corresponding to valid pixels;

[0011] Extract high-level features from the feature map corresponding to the effective pixels to obtain a high-level feature map of the target object;

[0012] Defect classification processing is performed based on the high-level feature map to obtain the defect classification results of the target object.

[0013] In one embodiment, the invalid pixel mask prediction network includes a first convolution layer, a first activation layer, a second convolution layer, a second activation layer, a third convolution layer, and a third activation layer. The convolution kernel size of the third convolution layer is 1×1, and the activation function used in the third activation layer is a Softmax activation function. The invalid pixel mask prediction network is trained using a backpropagation algorithm based on a first loss function. The first loss function is determined according to the following expression:

[0014]

[0015] Among them, L1 represents the value of the first loss function, α s Indicates the preset weight, p g Represents the confidence information of the prediction, p t Indicates the confidence information of the annotation.

[0016] In one embodiment, extracting high-level features from a feature map corresponding to valid pixels includes:

[0017] A highly adaptive module is used to process the feature map corresponding to the effective pixels, and the feature map corresponding to the effective pixels is converted into a feature map based on the same reference surface;

[0018] Extract high-level features from feature maps based on the same base surface.

[0019] In one embodiment, processing the feature map corresponding to the valid pixels using the highly adaptive module includes:

[0020] Perform convolution operation on the feature map corresponding to the valid pixels to obtain the convolution feature map;

[0021] After the convolution feature map is processed by global average pooling, it is passed through the first fully connected layer for dimensionality reduction to obtain the reduced dimensionality feature map;

[0022] Determine the mean feature data on each channel based on the feature map after dimensionality reduction;

[0023] Generate a mean feature map of the same size as the convolution feature map on each channel according to the mean feature data on each channel;

[0024] Perform pixel-level subtraction on the convolution feature map and the mean feature map on each channel to obtain a feature map based on the same reference surface.

[0025] In one embodiment, the highly adaptive module is trained using a back-propagation algorithm according to a second loss function, where the second loss function is determined according to the following expression:

[0026] L2=(xz-μ) 2 ;

[0027] Wherein, L2 represents the value of the second loss function, x represents the feature vector input to the highly adaptive module, z represents the feature vector output by the highly adaptive module, and μ represents the mean feature vector determined by the highly adaptive module.

[0028] In one embodiment, when performing defect classification processing, the loss function is determined according to the following expression:

[0029] L=L2+α1L3;

[0030] L3=a1·y g ·log(y p )+a0·(1-y g )log(1-y p );

[0031] Among them, L represents the value of the loss function when performing defect classification processing, α1 represents the preset weight value, L3 represents the value of the third loss function, a1 and a0 represent the weights of the classification categories respectively, and y g Represents the probability value of the annotation, y p Represents the predicted probability value.

[0032] In one embodiment, before extracting primary features from the structured three-dimensional image data, the method further includes:

[0033] The structured three-dimensional image data is pre-processed for data enhancement, where the pre-processing includes one or more of translation, rotation, scaling, and contrast enhancement.

[0034] In a second aspect, an embodiment of the present invention provides a surface defect detection device based on three-dimensional images, comprising:

[0035] An image acquisition module is used to obtain structured three-dimensional image data of the target object. The three-dimensional image data includes height information, which is used to reflect the surface morphology information of the target object;

[0036] A primary feature extraction module is used to extract primary features from structured three-dimensional image data to obtain a primary feature map of the target object;

[0037] A multi-scale feature extraction module is used to extract multi-scale features from the primary feature map to obtain a multi-scale feature map of the same size as the primary feature map;

[0038] A mask prediction module is used to process the primary feature map using a pre-trained invalid pixel mask prediction network to obtain mask data of the same size as the primary feature map. The mask data is used to identify the location information of invalid pixels and / or valid pixels.

[0039] The invalid pixel processing module is used to perform feature cropping on the multi-scale feature map according to the mask data, filter out the feature map corresponding to the invalid pixels, and obtain the feature map corresponding to the valid pixels;

[0040] The high-level feature extraction module is used to extract high-level features from the feature map corresponding to the effective pixels to obtain the high-level feature map of the target object;

[0041] The defect classification module is used to perform defect classification processing based on the high-level feature map to obtain the defect classification results of the target object.

[0042] In a third aspect, an embodiment of the present invention provides a surface defect detection device, comprising: an image acquisition device and a processor;

[0043] The image acquisition device is used to obtain structured three-dimensional image data of the target object;

[0044] The processor is configured to execute the surface defect detection method based on three-dimensional images as described in any one of the first aspects.

[0045] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement a surface defect detection method based on three-dimensional images as described in any one of the first aspects.

[0046] The surface defect detection method and device based on three-dimensional images provided by the embodiment of the present invention reduces the amount of data to be processed by obtaining structured three-dimensional image data of the target object; performs multi-scale feature extraction on the primary feature map extracted from the structured three-dimensional image data to adapt to surface defects of different sizes; obtains mask data of the same size as the primary feature map by using a pre-trained invalid pixel mask prediction network, and performs feature cropping on the multi-scale feature map according to the mask data, filters out the feature maps corresponding to invalid pixels, and eliminates the influence of invalid pixels on surface defect detection; finally, performs defect classification processing based on the high-level feature map extracted from the feature map corresponding to the valid pixels, obtains the defect classification result of the target object, and realizes surface defect detection of the target object. In summary, the method provided by the embodiment of the present invention can not only improve the efficiency of surface defect detection, consume less time, meet the needs of real-time detection in industrial applications, but also improve the accuracy of surface defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A flow chart of a surface defect detection method based on three-dimensional images provided by one embodiment of the present invention;

[0048] Figure 2 A flow chart of a surface defect detection method based on three-dimensional images provided in another embodiment of the present invention;

[0049] Figure 3 A schematic diagram of the structure of a highly adaptive module provided by one embodiment of the present invention;

[0050] Figure 4 A flow chart of a surface defect detection method based on three-dimensional images provided by another embodiment of the present invention;

[0051] Figure 5 A schematic structural diagram of a surface defect detection device provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be further described in detail below by means of specific embodiments in conjunction with the accompanying drawings. Similar elements in different embodiments are numbered with associated similar elements. In the following embodiments, many detailed descriptions are provided to enable the present application to be better understood. However, those skilled in the art will readily appreciate that some of the features may be omitted in different circumstances, or may be replaced by other elements, materials, or methods. In some cases, some operations related to the present application are not shown or described in the specification. This is to avoid the core portion of the present application being overwhelmed by excessive descriptions, and for those skilled in the art, it is not necessary to describe these related operations in detail. They will fully understand the related operations based on the description in the specification and the general technical knowledge in the art.

[0053] In addition, the features, operations, or characteristics described in the specification may be combined in any appropriate manner to form various embodiments. Furthermore, the steps or actions in the method description may be reordered or adjusted in a manner readily apparent to those skilled in the art. Therefore, the various sequences in the specification and drawings are provided solely for the purpose of clearly describing a particular embodiment and are not intended to be mandatory, unless otherwise specified.

[0054] The serial numbers assigned to components herein, such as "first," "second," etc., are used solely to distinguish the objects being described and do not convey any sequential or technical meaning. References to "connection" and "coupling" herein, unless otherwise specified, include both direct and indirect connections (couplings).

[0055] Please refer to Figure 1 , a surface defect detection method based on three-dimensional images provided by an embodiment of the present invention may include:

[0056] S101 : Acquire structured three-dimensional image data of a target object, where the three-dimensional image data includes height information, and is used to reflect surface morphology information of the target object.

[0057] Compared to texture information, topography information is more useful for classifying surface defects. Therefore, in this embodiment, 3D image data that reflects the topography of the target object's surface is acquired. While 3D point cloud data can also reflect topography, its unstructured nature significantly increases the data volume, significantly reducing the efficiency of surface defect detection and hindering industrial applications. Therefore, in this embodiment, structured 3D image data of the target object is acquired.

[0058] In this embodiment, for example, a 3D vision sensor can be used to acquire structured 3D image data of a target object. This structured 3D image data can be represented, for example, by a 2D matrix, where each element in the matrix represents the height information of the corresponding pixel. This height information can be determined based on the distance between the pixel and the 3D vision sensor. Compared to unstructured 3D point cloud data, structured 3D image data significantly reduces the amount of data required, improving the efficiency of surface defect detection and facilitating industrial applications.

[0059] S102: Extracting primary features from the structured three-dimensional image data to obtain a primary feature map of the target object.

[0060] Primary features, also known as low-level features in machine learning, refer to feature data obtained after relatively shallow convolution, pooling, and activation function processing, retaining a relatively large amount of positional information. Existing networks for extracting low-level features can be used to extract primary features from structured 3D image data to obtain a primary feature map of the target object. This embodiment does not limit the specific network structure.

[0061] S103: Perform multi-scale feature extraction on the primary feature map to obtain a multi-scale feature map of the same size as the primary feature map.

[0062] Surface defects to be detected include both large and small objects. To accommodate surface defects of varying sizes, this embodiment, after obtaining a primary feature map of the target object, further performs multi-scale feature extraction on the primary feature map. This embodiment employs existing multi-scale feature extraction methods for performing multi-scale feature extraction on the primary feature map, and this embodiment does not limit the specific implementation of the multi-scale feature extraction method. By performing multi-scale feature extraction on the primary feature map, it is possible to accommodate surface defects of varying sizes, helping to improve the accuracy of surface defect detection.

[0063] S104: Use a pre-trained invalid pixel mask prediction network to process the primary feature map to obtain mask data of the same size as the primary feature map, where the mask data is used to identify position information of invalid pixels and / or valid pixels.

[0064] S105 , performing feature cropping on the multi-scale feature map according to the mask data, filtering out feature maps corresponding to invalid pixels, and obtaining feature maps corresponding to valid pixels.

[0065] Invalid pixels are generated when a 3D vision sensor is unable to obtain pixel height information during the imaging process due to reflections, multiple reflections, occlusions, and other factors. They are often represented by a fixed, large numerical value. Currently, the common practice is to binarize the image data to obtain mask data for invalid pixels. It should be noted that in high-precision surface defect detection, the image data is large, reaching hundreds of millions of pixels. The collected image data is often represented as floating-point numbers. Binarization of floating-point numbers is time-consuming and difficult to meet the real-time detection requirements of industrial applications.

[0066] In order to solve this problem, the present application abandons the traditional binarization processing method, and instead uses a pre-trained invalid pixel mask prediction network to process the primary feature map, and obtains the mask data corresponding to the invalid pixels at the feature level. Compared with the binarization processing, the method provided by this embodiment is time-saving and efficient, and can meet the needs of real-time monitoring of industrial applications. The mask data obtained in this embodiment is the same size as the primary feature map, and the same size here means that the length and width are consistent with the primary feature map, and the number of channels of the mask data is 1. The mask data in this embodiment can only identify the position information of invalid pixels, or only identify the position information of valid pixels, or simultaneously identify the position information of invalid pixels and valid pixels. In an optional implementation, a first identifier can be used to identify the position information of invalid pixels, and a second identifier can be used to identify the position information of valid pixels. For example, 0 is used to identify the position information of invalid pixels, and 1 is used to identify the position information of valid pixels.

[0067] After obtaining mask data for identifying the locations of invalid and / or valid pixels, feature cropping can be performed on the multi-scale feature map based on the mask data. Specifically, the mask data is used to filter out feature maps corresponding to invalid pixels in each channel of the multi-scale feature map. Feature cropping in this embodiment involves filtering out feature maps corresponding to invalid pixels from the multi-scale feature map, thereby obtaining feature maps corresponding to valid pixels. This eliminates the impact of invalid pixels on surface defect detection and improves the accuracy of surface defect detection.

[0068] S106. Extract high-level features from the feature map corresponding to the valid pixels to obtain a high-level feature map of the target object.

[0069] High-level features, or high-level features in machine learning, refer to feature data obtained after multiple layers of convolution, pooling, and activation function processing, and have relatively good feature abstraction capabilities. Existing networks for extracting high-level features can be used to extract primary features from feature maps corresponding to valid pixels to obtain a primary feature map of the target object. This embodiment does not limit the specific network structure.

[0070] S107: Perform defect classification processing based on the high-level feature map to obtain a defect classification result of the target object.

[0071] In this embodiment, after extracting a high-level feature map with good feature abstraction capabilities from the feature map corresponding to the effective pixels, surface defect detection of the target object can be achieved based on the high-level feature map. In an optional implementation, the high-level feature map can be input into a preset defect classification network, and the high-level feature map can be processed using the defect classification network to obtain a defect classification result of the target object. The defect classification result can be determined according to the specific detection task. Taking battery coating defect detection as an example, the defect classification result can be one or more of holes, tapes, scratches, cracks, wrinkles, breakage, tailing, decarburization, bubbling, bubbles, joints, and bright spots.

[0072] The surface defect detection method based on three-dimensional images provided in this embodiment reduces the amount of data to be processed by obtaining structured three-dimensional image data of the target object; performs multi-scale feature extraction on the primary feature map extracted from the structured three-dimensional image data to adapt to surface defects of different sizes; obtains mask data of the same size as the primary feature map by using a pre-trained invalid pixel mask prediction network, and performs feature cropping on the multi-scale feature map based on the mask data, filters out the feature maps corresponding to invalid pixels, and eliminates the influence of invalid pixels on surface defect detection; finally, performs defect classification processing based on the high-level feature map extracted from the feature map corresponding to the valid pixels, obtains the defect classification result of the target object, and realizes surface defect detection of the target object. In summary, the method provided in this embodiment can not only improve the efficiency of surface defect detection, consume less time, meet the needs of real-time detection in industrial applications, but also improve the accuracy of surface defect detection.

[0073] In an optional implementation, the invalid pixel mask prediction network may include a first convolution layer, a first activation layer, a second convolution layer, a second activation layer, a third convolution layer, and a third activation layer, the convolution kernel size of the third convolution layer is 1×1, and the activation function used in the third activation layer is a Softmax activation function. The invalid pixel mask prediction network is trained using a backpropagation algorithm based on a first loss function, and the first loss function is determined according to the following expression:

[0074]

[0075] Among them, L1 represents the value of the first loss function, α s Indicates the preset weight, p g Represents the confidence information of the prediction, p t Indicates the confidence information of the annotation.

[0076] The invalid pixel mask prediction network can be pre-trained using a training data set. The training data set contains several primary feature maps of three-dimensional image data and the annotations of their corresponding mask data. During pre-training, the primary feature map of the three-dimensional image data is used as the input of the invalid pixel mask prediction network, and the annotations of its corresponding mask data are used as the expected output. Iterative training is performed with the goal of minimizing the first loss function until the value of the first loss function is less than a preset threshold, and a pre-trained invalid pixel mask prediction network is obtained. After the primary feature map is input into the pre-trained invalid pixel mask prediction network, it undergoes two layers of convolution and activation processing, then undergoes 1×1 convolution, and then is activated using the Softmax activation function, so that mask data of the same size as the input primary feature map can be obtained.

[0077] Compared with obtaining the mask data of invalid pixels at the image data level through binarization, the method of obtaining mask data through the invalid pixel mask prediction network in this embodiment takes less time and is more stable in obtaining the mask data of invalid pixels at the feature level.

[0078] When 3D vision sensors collect structured three-dimensional image data, the reference surface may be inconsistent due to changes in posture, distance, and acquisition environment. This will cause the distribution of the collected structured three-dimensional image data to be inconsistent, and the height range will vary greatly, affecting the accuracy of surface defect detection. Please refer to Figure 2 On the basis of the above embodiment, in order to further improve the accuracy of surface defect detection, in the surface defect detection method based on three-dimensional images provided in this embodiment, extracting high-level features from the feature map corresponding to effective pixels may include:

[0079] S1061. Use a highly adaptive module to process the feature map corresponding to the valid pixels, and convert the feature map corresponding to the valid pixels into a feature map based on the same reference plane.

[0080] The highly adaptive module in this embodiment is primarily designed to eliminate the data distribution inconsistency problem caused by inconsistent reference planes. Specifically, the feature map corresponding to valid pixels can be mapped from one feature space to another, but the feature mean is changed. Specifically, the feature mean is transformed to the same preset mean, thereby achieving a unified reference plane at the feature level.

[0081] The highly adaptive module can be pre-trained using a training dataset, and the adaptive parameters can be obtained by learning the images in the training dataset relative to a uniform reference plane. Figure 3 The highly adaptive module provided by one embodiment mainly includes F tr 、F sq and F ex Three operations and a pixel-wise subtraction "-" operation.

[0082] Among them, F tr Represents the convolution operation of the feature map, specifically:

[0083] F tr :X→U,X∈R H′×W′×C′ ,U→R H×W×C ;

[0084] X and U represent the feature maps of two adjacent network layers, respectively. H', W' and C' represent the length, width and number of channels of X, respectively. H, W and C represent the length, width and number of channels of U, respectively.

[0085] F sq Represents the global average pooling operation, specifically:

[0086]

[0087] H and W represent the length and width of the feature map respectively, u i,j Represents the element in the i-th row and j-th column of the feature map.

[0088] F ex A fully connected layer is used for dimensionality reduction to reduce computational effort and improve efficiency. After obtaining the mean feature data for each channel, a pixel-level subtraction operation ("-") is performed on the channel. This pixel-level subtraction removes the influence of feature height.

[0089] S1062. Extract high-level features from the feature map based on the same reference surface.

[0090] In this embodiment, after converting the feature maps corresponding to valid pixels into feature maps based on the same reference plane, high-level features are extracted from the feature maps based on the same reference plane. It is understood that the high-level features only include features of valid pixels, and the reference plane is unified at the feature level, maintaining consistency in data distribution.

[0091] The surface defect detection method based on three-dimensional images provided in this embodiment, on the basis of the above embodiments, further processes the feature map corresponding to the effective pixels by adopting a highly adaptive module, and converts the feature map corresponding to the effective pixels into a feature map based on the same reference plane, thereby eliminating the problems of inconsistent data distribution, large changes and differences in height ranges caused by inconsistent reference planes at the feature level, and helping to further improve the accuracy of surface defect detection.

[0092] Please refer to Figure 4Based on any of the above embodiments, in a surface defect detection method based on three-dimensional images provided by an embodiment of the present invention, using a highly adaptive module to process a feature map corresponding to effective pixels may include:

[0093] S10611. Perform a convolution operation on the feature map corresponding to the valid pixels to obtain a convolution feature map.

[0094] S10612. After performing a global average pooling operation on the convolutional feature map, the convolutional feature map is subjected to dimensionality reduction through the first fully connected layer to obtain a feature map after dimensionality reduction.

[0095] It is understandable that when a 1000×1000 image is reduced to 100×100, its pixel mean is approximately the same. In other words, dimensionality reduction only significantly reduces the amount of computation and improves computational efficiency, but does not significantly change the pixel mean. The mean of each channel in the feature map after dimensionality reduction is approximately the same as that of the original convolution feature map.

[0096] S10613. Determine the mean feature data on each channel based on the feature map after dimensionality reduction.

[0097] According to the feature map after dimensionality reduction, the mean feature data on each channel is determined at the feature level.

[0098] S10614. Generate a mean feature map of the same size as the convolution feature map on each channel according to the mean feature data on each channel.

[0099] S10615. Perform pixel-level subtraction on the convolution feature map and the mean feature map on each channel to obtain a feature map based on the same reference plane.

[0100] Assuming there are C channels, C mean feature data can be determined from the reduced feature map. To increase processing speed and facilitate industrial applications, a matrix equal in size to the convolution feature map can be generated for each channel. Each element in the matrix is ​​the mean feature data for that channel, resulting in the mean feature map for that channel. Pixel-level subtraction between the convolution feature map and the mean feature map can then be simplified to a matrix subtraction operation, increasing processing speed.

[0101] Through pixel-level subtraction between the convolution feature map and the mean feature map, the influence of the height on the feature is subtracted and converted into a feature map based on the same reference plane.

[0102] Compared with processing image data to obtain pixel mean at the image data level, the method of using a highly adaptive module to process the feature map corresponding to the effective pixels in this embodiment takes less time, and determining the mean feature data on each channel at the feature level for the feature map corresponding to the effective pixels is also more stable.

[0103] In an optional embodiment, the highly adaptive module is trained using a back propagation algorithm according to a second loss function, and the second loss function is determined according to the following expression:

[0104] L2=(xz-μ) 2 ;

[0105] Wherein, L2 represents the value of the second loss function, x represents the feature vector input to the highly adaptive module, z represents the feature vector output by the highly adaptive module, and μ represents the mean feature vector determined by the highly adaptive module.

[0106] In an optional implementation, when performing defect classification processing, in order to take into account both the loss of the highly adaptive module and the classification loss, the loss function for performing defect classification processing can be determined according to the following expression:

[0107] L=L2+α1L3;

[0108] L3=a1·y g ·log(y p )+a0·(1-y g )log(1-y p );

[0109] Among them, L represents the value of the loss function when performing defect classification processing, α1 represents the preset weight value, L3 represents the value of the third loss function, a1 and a0 represent the weights of the classification categories respectively, and y g Represents the probability value of the annotation, y p Represents the predicted probability value.

[0110] Based on any of the above embodiments, in order to improve generalization ability and scene adaptability, before extracting primary features from the structured three-dimensional image data, the method may also include: performing data enhancement preprocessing on the structured three-dimensional image data, the preprocessing including one or more of translation, rotation, scaling and contrast enhancement.

[0111] An embodiment of the present invention further provides a surface defect detection device based on three-dimensional images, comprising:

[0112] An image acquisition module is used to obtain structured three-dimensional image data of the target object. The three-dimensional image data includes height information, which is used to reflect the surface morphology information of the target object;

[0113] A primary feature extraction module is used to extract primary features from structured three-dimensional image data to obtain a primary feature map of the target object;

[0114] A multi-scale feature extraction module is used to extract multi-scale features from the primary feature map to obtain a multi-scale feature map of the same size as the primary feature map;

[0115] A mask prediction module is used to process the primary feature map using a pre-trained invalid pixel mask prediction network to obtain mask data of the same size as the primary feature map. The mask data is used to identify the location information of invalid pixels and / or valid pixels.

[0116] The invalid pixel processing module is used to perform feature cropping on the multi-scale feature map according to the mask data, filter out the feature map corresponding to the invalid pixels, and obtain the feature map corresponding to the valid pixels;

[0117] The high-level feature extraction module is used to extract high-level features from the feature map corresponding to the effective pixels to obtain the high-level feature map of the target object;

[0118] The defect classification module is used to perform defect classification processing based on the high-level feature map to obtain the defect classification results of the target object.

[0119] The surface defect detection device based on three-dimensional images provided in this embodiment can be used to perform Figure 1 The technical solutions of the corresponding method embodiments have similar implementation principles and technical effects and will not be repeated here.

[0120] The present invention also provides a surface defect detection device, see Figure 5 As shown, the embodiment of the present invention is only Figure 5 This is just an example for explanation, and it does not mean that the present invention is limited to this. Figure 5 This is a schematic structural diagram of an embodiment of the surface defect detection device provided by the present invention. Figure 5 As shown, the surface defect detection device 50 provided in this embodiment may include: an image acquisition device 501 and a processor 502. The image acquisition device 501 and the processor 502 may be connected in a bus manner or in other ways.

[0121] The image acquisition device 501 may be used to acquire structured three-dimensional image data of a target object, and the processor 502 may be used to execute the three-dimensional image-based surface defect detection method provided in any of the above embodiments.

[0122] The image acquisition device 501 may be, for example, a 3D vision sensor or other device capable of acquiring structured three-dimensional image data. The processor 502 may be an integrated circuit chip having signal processing capabilities. The processor 502 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. The methods, steps, and logic block diagrams disclosed in the embodiments of the present invention may be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor.

[0123] I understand. Figure 5 The structure is only for illustration and may also include Figure 5 More or fewer components than shown, or with Figure 5 Different configurations shown. Figure 5 Each component shown in the figure can be implemented using hardware and / or software. For example, the surface defect detection device 50 may also include a memory (not shown in the figure) for storing the collected structured three-dimensional image data of the target object and for storing a computer program that can be executed by the processor 502 to implement the technical solution of the surface defect detection method based on three-dimensional images provided by any of the above method embodiments. The memory can be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0124] This document is described with reference to various exemplary embodiments. However, those skilled in the art will recognize that changes and modifications may be made to the exemplary embodiments without departing from the scope of this document. For example, the various operational steps and components used to perform the operational steps may be implemented in different ways (e.g., one or more steps may be deleted, modified, or incorporated into other steps) depending on the specific application or considering any number of cost functions associated with the operation of the system.

[0125] Additionally, as will be appreciated by those skilled in the art, the principles of this disclosure may be embodied in a computer program product on a computer-readable storage medium pre-loaded with computer-readable program code. Any tangible, non-transitory computer-readable storage medium may be used, including magnetic storage devices (hard disks, floppy disks, etc.), optical storage devices (CD-ROMs, DVDs, Blu-ray discs, etc.), flash memory, and / or the like. These computer program instructions may be loaded onto a general-purpose computer, a special-purpose computer, or other programmable data processing device to form a machine, such that the instructions executed on the computer or other programmable data processing device can generate a device that implements a specified function. These computer program instructions may also be stored in a computer-readable memory, which can instruct the computer or other programmable data processing device to operate in a specific manner, such that the instructions stored in the computer-readable memory can form an article of manufacture that includes an implementation device that implements the specified function. The computer program instructions may also be loaded onto a computer or other programmable data processing device, causing the computer or other programmable device to execute a series of operational steps to produce a computer-implemented process, such that the instructions executed on the computer or other programmable device provide the steps for implementing the specified function.

[0126] The above examples are used to illustrate the present invention, which are only used to help understand the present invention and are not intended to limit the present invention. Those skilled in the art can make several simple deductions, modifications or substitutions based on the concept of the present invention.

Claims

1. A surface defect detection method based on three-dimensional images, characterized in that: include: Acquire structured three-dimensional image data of a target object, wherein the three-dimensional image data includes height information for reflecting topographic information of a surface of the target object; Extracting primary features from the structured three-dimensional image data to obtain a primary feature map of the target object; Performing multi-scale feature extraction on the primary feature map to obtain a multi-scale feature map of the same size as the primary feature map; A pre-trained invalid pixel mask prediction network is used to process the primary feature map to obtain mask data of the same size as the primary feature map, where the mask data is used to identify the position information of invalid pixels and / or valid pixels. The invalid pixel mask prediction network includes a first convolution layer, a first activation layer, a second convolution layer, a second activation layer, a third convolution layer, and a third activation layer. The convolution kernel size of the third convolution layer is 1×1, and the activation function used by the third activation layer is a Softmax activation function. The invalid pixel mask prediction network is pre-trained using a training data set, where the training data set contains annotations of primary feature maps of several three-dimensional image data and their corresponding mask data. A first identifier is used to identify the position information of invalid pixels, and a second identifier is used to identify the position information of valid pixels. The invalid pixel mask prediction network is trained using a backpropagation algorithm based on a first loss function, and the first loss function is determined according to the following expression: Among them, L1 represents the value of the first loss function, α s Indicates the preset weight, p g Represents the confidence information of the prediction, p t Indicates the confidence information of the annotation; Performing feature cropping on the multi-scale feature map according to the mask data, filtering out feature maps corresponding to invalid pixels, and obtaining feature maps corresponding to valid pixels; Extracting high-level features from the feature map corresponding to the effective pixels to obtain a high-level feature map of the target object; Defect classification processing is performed according to the high-level feature map to obtain a defect classification result of the target object.

2. The method according to claim 1, wherein Extracting high-level features from the feature map corresponding to the effective pixels includes: Using a highly adaptive module to process the feature map corresponding to the effective pixel, and convert the feature map corresponding to the effective pixel into a feature map based on the same reference plane; High-level features are extracted from the feature map based on the same reference surface.

3. The method according to claim 2, wherein The processing of the feature map corresponding to the effective pixel by using a highly adaptive module includes: Performing a convolution operation on the feature map corresponding to the effective pixel to obtain a convolution feature map; After performing a global average pooling operation on the convolutional feature map, the convolutional feature map is subjected to dimensionality reduction through a first fully connected layer to obtain a feature map after dimensionality reduction; Determine the mean feature data on each channel according to the feature map after dimensionality reduction; Generating a mean feature map of the same size as the convolution feature map on each channel according to the mean feature data on each channel; Performing pixel-level subtraction on the convolution feature map and the mean feature map on each channel to obtain a feature map based on the same reference plane.

4. The method according to claim 3, wherein The highly adaptive module is trained using a back propagation algorithm according to a second loss function, and the second loss function is determined according to the following expression: L2=(x-z-μ) 2 ; Wherein, L2 represents the value of the second loss function, x represents the feature vector input to the highly adaptive module, z represents the feature vector output by the highly adaptive module, and μ represents the mean feature vector determined by the highly adaptive module.

5. The method according to claim 4, wherein When performing defect classification processing, the loss function is determined according to the following expression: L=L2+α1L3; L3=a1·y g ·log(y p )+a0·(1-y g )log(1-y p ); Among them, L represents the value of the loss function when performing defect classification processing, α1 represents the preset weight value, L3 represents the value of the third loss function, a1 and a0 represent the weights of the classification categories respectively, and y g Represents the probability value of the annotation, y p Represents the predicted probability value.

6. The method according to any one of claims 1 to 5, wherein: Before extracting primary features from the structured three-dimensional image data, the method further includes: The structured three-dimensional image data is pre-processed for data enhancement, where the pre-processing includes one or more of translation, rotation, scaling, and contrast enhancement.

7. A surface defect detection device based on three-dimensional images, characterized in that: include: An image acquisition module is used to obtain structured three-dimensional image data of a target object, wherein the three-dimensional image data includes height information, which is used to reflect the topography information of the surface of the target object; A primary feature extraction module, configured to extract primary features from the structured three-dimensional image data to obtain a primary feature map of the target object; A multi-scale feature extraction module, configured to perform multi-scale feature extraction on the primary feature map to obtain a multi-scale feature map of the same size as the primary feature map; A mask prediction module is configured to process the primary feature map using a pre-trained invalid pixel mask prediction network to obtain mask data of the same size as the primary feature map, wherein the mask data is used to identify the position information of invalid pixels and / or valid pixels. The invalid pixel mask prediction network includes a first convolution layer, a first activation layer, a second convolution layer, a second activation layer, a third convolution layer, and a third activation layer. The convolution kernel size of the third convolution layer is 1×1, and the activation function used in the third activation layer is a Softmax activation function. The invalid pixel mask prediction network is pre-trained using a training data set, wherein the training data set includes annotations of primary feature maps of several three-dimensional image data and their corresponding mask data. The position information of invalid pixels is identified using a first identifier, and the position information of valid pixels is identified using a second identifier. The invalid pixel mask prediction network is trained using a backpropagation algorithm based on a first loss function, and the first loss function is determined according to the following expression: Among them, L1 represents the value of the first loss function, α s Indicates the preset weight, p g Represents the confidence information of the prediction, p t Indicates the confidence information of the annotation; An invalid pixel processing module is used to perform feature cropping on the multi-scale feature map according to the mask data, filter out the feature map corresponding to the invalid pixels, and obtain the feature map corresponding to the valid pixels; A high-level feature extraction module is used to extract high-level features from the feature map corresponding to the effective pixels to obtain a high-level feature map of the target object; The defect classification module is used to perform defect classification processing according to the high-level feature map to obtain a defect classification result of the target object.

8. A surface defect detection device, characterized in that: include: Image acquisition device and processor; The image acquisition device is used to obtain structured three-dimensional image data of the target object; The processor is used to execute the surface defect detection method based on three-dimensional images as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The medium stores a program, which can be executed by a processor to implement the surface defect detection method based on three-dimensional images as described in any one of claims 1 to 6.

Images