A surface defect detection method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a surface defect detection method and device, electronic equipment and storage medium, and relate to the technical field of computer vision. The method comprises: obtaining a depth map of a to-be-detected object as a scene depth map, and obtaining a sample depth map; for each valid pixel point in the scene depth map, determining a pixel point corresponding to the valid pixel point in the sample depth map as a matching pixel point; wherein the position corresponding to the valid pixel point in the to-be-detected object is consistent with the position corresponding to the matching pixel point in the lossless object; determining a first residual value of the valid pixel point based on the difference between the pixel value of the valid pixel point and the pixel value of the corresponding matching pixel point; and performing surface defect detection based on the first residual values of the valid pixel points to obtain a detection result of the to-be-detected object. In this way, the surface defects of the to-be-detected object can be effectively detected.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a surface defect detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the advancement of modern technology, products are becoming increasingly precise and miniaturized. Therefore, the detection of defects and flaws in products has become more stringent. For example, in industrial production, surface defect detection is required on the surfaces of manufactured workpieces to determine whether defects exist, such as cracks or dents.

[0003] Therefore, there is an urgent need for an effective method for detecting surface defects. Summary of the Invention

[0004] The purpose of this application is to provide a surface defect detection method, apparatus, electronic device, and storage medium to effectively detect surface defects in the object to be inspected. The specific technical solution is as follows:

[0005] A first aspect of this application provides a surface defect detection method, the method comprising:

[0006] A depth map of the object to be detected is obtained as a scene depth map, and a sample depth map is also obtained; wherein the sample depth map is obtained based on the lossless object corresponding to the object to be detected.

[0007] For each valid pixel in the scene depth map, the corresponding pixel in the sample depth map is determined as the matching pixel; wherein, the position of the valid pixel in the object to be detected is consistent with the position of the corresponding matching pixel in the lossless object.

[0008] The first residual value of the valid pixel is determined based on the difference between the pixel value of the valid pixel and the pixel value of the corresponding matching pixel.

[0009] Surface defect detection is performed based on the first residual value of each effective pixel to obtain the detection result of the object to be detected.

[0010] In some embodiments, the surface defect detection based on the first residual value of each effective pixel to obtain the detection result of the object to be detected includes:

[0011] For each valid pixel in the scene depth map, obtain the correction coefficient corresponding to the valid pixel; wherein, the correction coefficient corresponding to a valid pixel is used to: reduce the first residual value of the valid pixel when the valid pixel does not belong to the surface area of ​​the object to be detected that currently needs to be detected for surface defects;

[0012] Based on the correction coefficient corresponding to the effective pixel, the first residual value of the effective pixel is corrected to obtain the second residual value of the effective pixel.

[0013] Surface defect detection is performed based on the second residual value of each effective pixel to obtain the detection result of the object to be detected.

[0014] In some embodiments, the correction coefficient corresponding to a valid pixel in the scene depth map includes a first correction coefficient and / or a second correction coefficient;

[0015] The first correction coefficient corresponding to a valid pixel in the scene depth map is calculated through the following steps:

[0016] Determine the position of the matching pixel corresponding to the valid pixel in the non-destructive object; determine the angle between the normal vector at the position and the normal vector of a specified surface region in the non-destructive object; wherein the specified surface region corresponds to the surface region in the object to be inspected that currently needs to be inspected for surface defects; based on the determined angle, determine the first correction coefficient corresponding to the valid pixel;

[0017] The second correction coefficient corresponding to a valid pixel in the scene depth map is calculated through the following steps:

[0018] Determine the first preset neighborhood to which the matching pixel corresponding to the valid pixel belongs in the sample depth map; calculate the ratio of valid pixels in the first preset neighborhood to obtain the second correction coefficient corresponding to the valid pixel.

[0019] In some embodiments, determining the first residual value of the effective pixel based on the difference between the pixel value of the effective pixel and the pixel value of the corresponding matching pixel includes:

[0020] Obtain the difference between the pixel value of each valid pixel in the second preset neighborhood of the scene depth map to which the valid pixel belongs and the pixel value of its corresponding matching pixel.

[0021] The minimum value among all differences is determined as the first residual value of the valid pixel.

[0022] In some embodiments, before performing surface defect detection based on the second residual value of each valid pixel to obtain the detection result of the object to be detected, the method further includes:

[0023] For each invalid pixel in the scene depth map, if the pixel corresponding to the invalid pixel in the sample depth map is valid, then the second residual value of the invalid pixel is determined to be a preset outlier value.

[0024] If the pixel corresponding to the invalid pixel in the sample depth map is invalid, then the second residual value of the invalid pixel is determined to be 0;

[0025] The surface defect detection based on the second residual value of each effective pixel point, to obtain the detection result of the object to be detected, includes:

[0026] Surface defect detection is performed based on the second residual value of invalid and valid pixels in the scene depth map to obtain the detection result of the object to be detected.

[0027] In some embodiments, determining the corresponding pixel in the sample depth map as the matching pixel for each valid pixel in the scene depth map includes:

[0028] The pixels in the scene depth map are mapped to a specified three-dimensional coordinate system to obtain the scene point cloud;

[0029] The pixels in the sample depth map are mapped to a specified three-dimensional coordinate system to obtain the sample point cloud.

[0030] Calculate the pose transformation information between the scene point cloud and the sample point cloud;

[0031] For each valid pixel in the scene depth map, based on the correspondence between points in the scene point cloud and pixels in the scene depth map, the pose transformation information, and the correspondence between points in the sample scene point cloud and pixels in the sample depth map, the corresponding pixel in the sample depth map is determined as the matching pixel.

[0032] In some embodiments, the surface defect detection based on the second residual value of each effective pixel to obtain the detection result of the object to be detected includes:

[0033] Based on the second residual value of each effective pixel, a corresponding grayscale image to be detected is generated; the grayscale image to be detected is input into a pre-trained defect detection model to obtain the detection result of the object to be detected; wherein, the defect detection model is trained based on sample grayscale images and corresponding sample labels, the sample grayscale images contain sample objects with surface defects; the sample labels are used to represent the defects on the surface of the sample objects.

[0034] In some embodiments, the surface defect detection based on the second residual value of each effective pixel to obtain the detection result of the object to be detected includes:

[0035] Based on the second residual value of each effective pixel, a corresponding grayscale image to be detected is generated; the grayscale image to be detected is binarized to obtain a binarized image; the binarized image is subjected to connected component extraction; and the detection result of the object to be detected is determined based on the extraction result.

[0036] A second aspect of this application provides a surface defect detection device, the device comprising:

[0037] The depth map acquisition module is used to acquire the depth map of the object to be detected as the scene depth map, and to acquire the sample depth map; wherein, the sample depth map is obtained based on the lossless object corresponding to the object to be detected;

[0038] The matching module is used to determine, for each valid pixel in the scene depth map, the corresponding pixel in the sample depth map as the matching pixel; wherein, the position of the valid pixel in the object to be detected is consistent with the position of the corresponding matching pixel in the lossless object;

[0039] The first residual value determination module is used to determine the first residual value of the effective pixel based on the difference between the pixel value of the effective pixel and the pixel value of the corresponding matching pixel.

[0040] The detection result acquisition module is used to perform surface defect detection based on the first residual value of each effective pixel point, and obtain the detection result of the object to be detected.

[0041] In some embodiments, the detection result acquisition module includes:

[0042] The correction coefficient acquisition submodule is used to acquire the correction coefficient corresponding to each effective pixel in the scene depth map before performing surface defect detection based on the second residual value of each effective pixel to obtain the detection result of the object to be detected; wherein, the correction coefficient corresponding to an effective pixel is used to: reduce the first residual value of the effective pixel when the effective pixel does not belong to the surface area of ​​the object to be detected that currently needs to be detected for surface defect detection.

[0043] The second residual value determination submodule is used to correct the first residual value of the effective pixel based on the correction coefficient corresponding to the effective pixel to obtain the second residual value of the effective pixel.

[0044] The detection result acquisition submodule is used to perform surface defect detection based on the second residual value of each effective pixel point, and obtain the detection result of the object to be detected.

[0045] In some embodiments, the correction coefficient corresponding to a valid pixel in the scene depth map includes a first correction coefficient and / or a second correction coefficient;

[0046] The first correction coefficient corresponding to a valid pixel in the scene depth map is calculated through the following steps:

[0047] Determine the position of the matching pixel corresponding to the valid pixel in the non-destructive object; determine the angle between the normal vector at the position and the normal vector of a specified surface region in the non-destructive object; wherein the specified surface region corresponds to the surface region in the object to be inspected that currently needs to be inspected for surface defects; based on the determined angle, determine the first correction coefficient corresponding to the valid pixel;

[0048] The second correction coefficient corresponding to a valid pixel in the scene depth map is calculated through the following steps:

[0049] Determine the first preset neighborhood to which the matching pixel corresponding to the valid pixel belongs in the sample depth map; calculate the ratio of valid pixels in the first preset neighborhood to obtain the second correction coefficient corresponding to the valid pixel.

[0050] In some embodiments, the first residual value determination module includes:

[0051] The difference acquisition submodule is used to acquire the difference between the pixel value of each effective pixel in the second preset neighborhood of the scene depth map and the pixel value of its corresponding matching pixel.

[0052] The first residual value determination submodule is used to determine the minimum value among the differences as the first residual value of the valid pixel.

[0053] In some embodiments, the apparatus further includes:

[0054] The first invalid point setting module is used to determine the second residual value of the invalid pixel as a preset abnormal value before the surface defect detection is performed based on the second residual value of each valid pixel to obtain the detection result of the object to be detected. For each invalid pixel in the scene depth map, if the pixel corresponding to the invalid pixel in the sample depth map is valid, the second residual value of the invalid pixel is determined to be a preset abnormal value.

[0055] The second invalid point setting module is used to determine that the second residual value of the invalid pixel is 0 if the pixel corresponding to the invalid pixel in the sample depth map is invalid.

[0056] The detection result acquisition submodule is specifically used to: perform surface defect detection based on the second residual value of invalid and valid pixels in the scene depth map, and obtain the detection result of the object to be detected.

[0057] In some embodiments, the matching module is specifically used for:

[0058] The pixels in the scene depth map are mapped to a specified three-dimensional coordinate system to obtain the scene point cloud;

[0059] The pixels in the sample depth map are mapped to a specified three-dimensional coordinate system to obtain the sample point cloud.

[0060] Calculate the pose transformation information between the scene point cloud and the sample point cloud;

[0061] For each valid pixel in the scene depth map, based on the correspondence between points in the scene point cloud and pixels in the scene depth map, the pose transformation information, and the correspondence between points in the sample scene point cloud and pixels in the sample depth map, the corresponding pixel in the sample depth map is determined as the matching pixel.

[0062] In some embodiments, the detection result acquisition submodule is specifically used for:

[0063] Based on the second residual value of each effective pixel, a corresponding grayscale image to be detected is generated; the grayscale image to be detected is input into a pre-trained defect detection model to obtain the detection result of the object to be detected; wherein, the defect detection model is trained based on sample grayscale images and corresponding sample labels, the sample grayscale images contain sample objects with surface defects; the sample labels are used to represent the defects on the surface of the sample objects.

[0064] In some embodiments, the detection result acquisition submodule is specifically used for:

[0065] Based on the second residual value of each effective pixel, a corresponding grayscale image to be detected is generated; the grayscale image to be detected is binarized to obtain a binarized image; the binarized image is subjected to connected component extraction; and the detection result of the object to be detected is determined based on the extraction result.

[0066] A third aspect of this application provides an electronic device, including:

[0067] Memory, used to store computer programs;

[0068] The processor, when executing a program stored in memory, implements any of the surface defect detection methods described above.

[0069] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the surface defect detection methods described above.

[0070] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the surface defect detection methods described above.

[0071] Beneficial effects of the embodiments in this application:

[0072] This application provides a surface defect detection method, which includes: acquiring a depth map of an object to be detected as a scene depth map, and acquiring a sample depth map; wherein the sample depth map is obtained based on a non-destructive object corresponding to the object to be detected; for each valid pixel in the scene depth map, determining the pixel in the sample depth map corresponding to the valid pixel as a matching pixel; wherein the position of the valid pixel in the object to be detected is consistent with the position of the corresponding matching pixel in the non-destructive object; determining a first residual value of the valid pixel based on the difference between the pixel value of the valid pixel and the pixel value of the corresponding matching pixel; and performing surface defect detection based on the first residual values ​​of each valid pixel to obtain the detection result of the object to be detected.

[0073] Based on the above processing, since the positions of the effective pixels in the scene depth map corresponding to the objects to be detected are the same as the positions of the corresponding matching pixels in the undamaged objects, that is, a single effective pixel in the scene depth map and its corresponding matching pixel correspond to the same position on different objects. Accordingly, the difference between the pixel value of the effective pixel and the pixel value of its corresponding matching pixel in the scene depth map (i.e., the first residual value) can reflect the difference between the objects to be detected and the undamaged objects at that position. Therefore, based on the first residual values ​​of each effective pixel in the scene depth map, surface defects of the objects to be detected can be effectively detected, and corresponding detection results can be obtained.

[0074] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0075] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0076] Figure 1 This is a schematic diagram of a first process for a surface defect detection method provided in an embodiment of this application;

[0077] Figure 2 This is a schematic diagram of a second process for a surface defect detection method provided in an embodiment of this application;

[0078] Figure 3 This is a schematic diagram of a third process for a surface defect detection method provided in an embodiment of this application;

[0079] Figure 4 This is a schematic diagram of the fourth process of the surface defect detection method provided in the embodiments of this application;

[0080] Figure 5 A schematic flowchart for calculating a first correction coefficient is provided for an embodiment of this application;

[0081] Figure 6 A schematic flowchart for calculating a second correction coefficient is provided for an embodiment of this application;

[0082] Figure 7 A schematic diagram of the fifth process of the surface defect detection method provided in the embodiments of this application;

[0083] Figure 8 A flowchart for modeling a sample depth map is provided in this application embodiment;

[0084] Figure 9 A flowchart for surface defect detection of an object to be tested is provided in an embodiment of this application;

[0085] Figure 10 An image illustrating the effect of surface defect detection provided in an embodiment of this application;

[0086] Figure 11 A structural diagram of a surface defect detection device provided in an embodiment of this application;

[0087] Figure 12 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0088] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0089] With the advancement of modern technology, products are becoming increasingly precise and miniaturized. Therefore, the detection of defects and flaws in products has become more stringent. For example, in industrial production, surface defect detection is required on the surfaces of manufactured workpieces to determine whether defects exist, such as cracks or dents.

[0090] This application provides a surface defect detection method that can be applied to electronic devices. For example, the electronic device can be a detection device for locating defect areas on the surface of an object to be inspected. This electronic device can effectively detect surface defects of the object without contacting it.

[0091] See Figure 1 , Figure 1 This is a schematic diagram of a first flowchart of a surface defect detection method provided in an embodiment of this application. The method may include the following steps:

[0092] S101: Obtain the depth map of the object to be detected as the scene depth map, and obtain the sample depth map.

[0093] The sample depth map is obtained based on the non-destructive object corresponding to the object to be detected.

[0094] S102: For each valid pixel in the scene depth map, determine the corresponding pixel in the sample depth map as the matching pixel.

[0095] The position of the valid pixel in the object to be detected is consistent with the position of the corresponding matching pixel in the lossless object.

[0096] S103: Determine the first residual value of the valid pixel based on the difference between the pixel value of the valid pixel and the pixel value of the corresponding matching pixel.

[0097] S104: Surface defect detection is performed based on the first residual value of each effective pixel to obtain the detection result of the object to be detected.

[0098] Based on the above processing, since the positions of the effective pixels in the scene depth map corresponding to the objects to be detected are the same as the positions of the corresponding matching pixels in the undamaged objects, that is, a single effective pixel in the scene depth map and its corresponding matching pixel correspond to the same position on different objects. Accordingly, the difference between the pixel value of the effective pixel and the pixel value of its corresponding matching pixel in the scene depth map (i.e., the first residual value) can reflect the difference between the objects to be detected and the undamaged objects at that position. Therefore, based on the first residual values ​​of each effective pixel in the scene depth map, surface defects of the objects to be detected can be effectively detected, and corresponding detection results can be obtained.

[0099] In step S101, the object to be inspected is the object that needs to undergo surface defect inspection. For example, in industrial production, the object to be inspected can be a workpiece produced on an assembly line; or, in a commodity trading scenario, the object to be inspected can be a commodity that needs to be inspected.

[0100] In one implementation, the non-destructive object corresponding to the object to be inspected can be a real object with no surface defects and of the same type as the object to be inspected. For example, when the object to be inspected is a certain type of workpiece, the non-destructive object corresponding to it can be a pre-produced workpiece of that type with no surface defects. Accordingly, an image acquisition device (such as a 3D camera) can be used to photograph the object to be inspected to obtain a depth map containing the surface features of the object to be inspected (i.e., the scene depth map in this application). Alternatively, an image acquisition device can be used to photograph the non-destructive object corresponding to the object to be inspected to obtain a depth map containing the surface features of the non-destructive object (i.e., the sample depth map in this application). The 3D camera can be an RGB-D (Red-Green-Blue-Depth) camera, a line laser profile scanner, or other image acquisition devices.

[0101] The electronic device may be equipped with the aforementioned image acquisition device for acquiring depth maps; or, the electronic device may communicate with the image acquisition device to acquire the depth map acquired by the image acquisition device.

[0102] In another implementation, the non-destructive object corresponding to the object to be inspected can also be a pre-constructed 3D virtual model based on 3D modeling technology, representing an object with no surface defects. For example, when the object to be inspected is a certain type of workpiece, the non-destructive object corresponding to it can be a pre-constructed 3D virtual model representing that type of workpiece with no surface defects. Correspondingly, the sample depth map can also be obtained by technicians rendering the aforementioned 3D virtual model based on image rendering technology.

[0103] For step S102, for each pixel in the depth map (scene depth map or sample depth map), if the pixel value of the pixel is not empty, then the pixel is a valid pixel. The pixel value of each valid pixel in the depth map represents the distance between the real-world location represented by the valid pixel and the image acquisition device.

[0104] If a pixel's value is empty, then that pixel is an invalid pixel (also called a null pixel). In other words, all pixels in the depth map except for valid pixels are invalid pixels. This means that the distance between the real-world location represented by the invalid pixel and the image acquisition device cannot be determined. For example, there may be no real object at the real-world location represented by the invalid pixel.

[0105] In other words, both the scene depth map and the sample depth map contain valid pixels, and they can also contain invalid pixels.

[0106] For each valid pixel in the scene depth map, a unique corresponding pixel (i.e., a matching pixel) can be determined in the sample depth map. The position of this valid pixel in the object to be detected is the same as the position of the corresponding matching pixel in the undamaged object. For example, the object to be detected is a screw (which can be called screw A), and the corresponding undamaged object can be a pre-manufactured screw with no surface defects (which can be called screw B). Accordingly, if the position of a valid pixel in the scene depth map in the object to be detected represents the tip of screw A, then the position of the matching pixel in the undamaged object represents the same position of the tip of screw B.

[0107] It is understandable that the depth map acquired by the image acquisition device for the same object (the object to be detected or the non-destructive object) may differ under different shooting conditions. These shooting conditions may include: the distance between the image acquisition device and the object, and the shooting angle of the image acquisition device during acquisition, etc.

[0108] In some embodiments, if the shooting factors of the scene depth map and the sample depth map are the same, the image area occupied by the object to be detected in the scene depth map is consistent in position and size with the image area occupied by the undamaged object in the sample depth map. Accordingly, for each valid pixel in the scene depth map, a pixel in the sample depth map with the same pixel coordinates as that valid pixel can be identified as the corresponding pixel (i.e., the matching pixel). The pixel coordinates of the valid pixel represent its position in the two-dimensional coordinate system (also called the image coordinate system) corresponding to the valid pixel in the scene depth map. This reduces the computational load of pixel matching when determining the matching pixel corresponding to each valid pixel in the scene depth map, shortens the matching time, and thus improves matching efficiency. Consequently, it improves defect detection efficiency and enables rapid detection.

[0109] In some embodiments, if the shooting factors when acquiring the scene depth map may differ from those when acquiring the sample depth map, the matching pixels cannot be accurately determined using the above method. Therefore, embodiments of this application provide a method for determining matching pixels based on point clouds. Figure 2 As shown, Figure 2 This is a schematic diagram of a second process for a surface defect detection method provided in an embodiment of this application. Figure 1 Based on this, step S102 includes:

[0110] S1021: Map the pixels in the scene depth map to the specified 3D coordinate system to obtain the scene point cloud.

[0111] S1022: Map the pixels in the sample depth map to the specified 3D coordinate system to obtain the sample point cloud.

[0112] S1023: Calculate pose transformation information between scene point cloud and sample point cloud.

[0113] S1024: For each valid pixel in the scene depth map, based on the correspondence between points in the scene point cloud and pixels in the scene depth map, pose transformation information, and the correspondence between points in the sample scene point cloud and pixels in the sample depth map, determine the corresponding pixel in the sample depth map as the matching pixel.

[0114] In this embodiment of the application, the specified three-dimensional coordinate system can be the world coordinate system, and the scene point cloud is the set of points obtained by mapping each pixel in the scene depth map to the specified three-dimensional coordinate system.

[0115] In one implementation, the pixels in the scene depth map are mapped to a specified 3D coordinate system to obtain the scene point cloud, including the following steps:

[0116] For a pixel in the scene depth map, the product of the pixel value and a preset scale change value is calculated as the first 3D coordinate value of the pixel in a specified 3D coordinate system. The product of the first image coordinate value and the preset scale change value is then calculated as the second 3D coordinate value of the pixel in the specified 3D coordinate system. Finally, the product of the second image coordinate value and the preset scale change value is calculated as the third 3D coordinate value of the pixel in the specified 3D coordinate system. The first and second image coordinate values ​​constitute the pixel's coordinates in the image coordinate system; the first, second, and third 3D coordinate values ​​constitute the 3D coordinates of the point corresponding to the pixel in the specified 3D coordinate system. The preset scale change value is pre-set by the technician based on actual needs; for example, the preset scale change value can be 0.5.

[0117] For example, for a pixel in the scene depth map in the image coordinate system, its pixel coordinates are (x1, y1), and its pixel value is z1. To map this pixel to a specified 3D coordinate system, we can calculate the product of the x-coordinate x1 and a preset scale change value to obtain the x-coordinate x2 of the point in the specified 3D coordinate system; calculate the product of the y-coordinate y1 and the preset scale change value to obtain the y-coordinate y2 of the point in the specified 3D coordinate system; and calculate the product of the pixel value z1 and the preset scale change value to obtain the ordinate z2 of the point in the specified 3D coordinate system. Thus, we can obtain the 3D coordinates of the point transformed into the specified 3D coordinate system, i.e., (x2, y2, z2).

[0118] In this way, the coordinates of each pixel in the scene depth map can be mapped to a specified 3D coordinate system, thus obtaining the scene point cloud corresponding to the scene depth map. Consequently, the correspondence between the points in the scene point cloud and the pixels in the scene depth map is obtained.

[0119] Correspondingly, the sample point cloud is a dataset obtained by mapping each pixel in the sample depth map to a specified 3D coordinate system. The specific process of mapping to obtain the sample point cloud can be referenced from the process of mapping to obtain the scene point cloud described above. Consequently, the correspondence between the points in the sample point cloud and the pixels in the sample depth map is obtained.

[0120] For ease of description, points in the scene point cloud can be called scene points, and points in the sample point cloud can be called sample points.

[0121] For example, for each target scene point in the scene point cloud, a corresponding sample point (called a target sample point) can be determined in the sample point cloud. The target scene point can be all points in the scene point cloud, or it can be a subset of the points in the scene point cloud. For instance, a specified proportion of points can be randomly selected from the scene point cloud as target scene points. This specified proportion could be 80%. A target scene point and its corresponding target sample point can be considered a 3D point pair.

[0122] In one implementation, for a target scene point in a point cloud scene, feature extraction can be performed using an existing neural network to obtain the feature information of the target scene point; the feature information may include brightness features and texture features, etc. Then, the feature information of each sample point in the sample point cloud can be obtained, the similarity between the feature information of each sample point and the feature information of the target scene point can be calculated, and the sample point corresponding to the highest similarity is determined as the target sample point corresponding to the target scene point.

[0123] In another implementation, the scene point cloud and the sample point cloud can be coarsely registered using the sampling consistency method. For a target scene point in the scene point cloud, the target sample point corresponding to the target scene point can be obtained based on a KD-tree (KDimensional-tree) or a lookup table.

[0124] After determining the target sample point corresponding to each target scene point, registration can be performed based on the determined multiple 3D point pairs to obtain the pose transformation information between the scene point cloud and the sample point cloud, that is, to obtain the rotation matrix and translation matrix between the scene point cloud and the sample point cloud. For example, registration can be performed based on the Iterative Closest Point (ICP) algorithm.

[0125] For example, the registration process includes the following steps:

[0126] Step 1: Calculate the rotation matrix R and the translation matrix t such that the value of the preset error function is less than the preset error value, for example, the preset error value can be 0.1. The preset error function is shown in formula (1):

[0127]

[0128] Where E(R,t) represents the preset error function. The set of all target scene points in the scene point cloud can be called the point set P; the set of target sample points corresponding to each target scene point can be called the point set Q. i q represents the coordinates of the i-th target scene point in the point set P; iR·p represents the coordinates of the target sample point in point set Q corresponding to the i-th target scene point; n represents the number of target scene points in the scene point cloud. i + indicates: the coordinates obtained by rotating and translating the coordinates of the i-th target scene point according to the rotation matrix R and the translation matrix t; ||q i -(R·p i +)‖ means q i With R·p i The distance between +.

[0129] Step 2: Based on the rotation matrix R and translation matrix t obtained from formula (1) above, perform rotation and translation transformations on the coordinates of each target scene point in the point set P to obtain the coordinates of the new target scene point, i.e., p i ′ ={R·p i +,p i ∈P}. Where, p i ′ This represents the new coordinates of the i-th target scene point obtained by performing rotation and translation transformations on its coordinates.

[0130] Step 3: Based on formula (2), calculate the distance between each new target scene point and the corresponding target sample point. That is, each target scene point corresponds to a distance, so we can obtain n distances corresponding to n target scene points. Based on the above n distances, we obtain the average distance:

[0131]

[0132] Where d represents the average distance. ‖p i ′ - i || represents p i ′ With q i The distance between them.

[0133] Step 4: If the stopping iteration condition is not met, return to Step 1 and continue execution until the stopping iteration condition is met. The stopping iteration condition can be that d is less than a preset distance value, or the number of iterations is greater than a preset number of iterations. Then, the translation matrix t and rotation matrix R can be obtained from the last iteration. The preset distance value can be 0.2, and the preset number of iterations can be 100.

[0134] Based on the above steps, the final rotation matrix R and translation matrix t are the pose transformation information in this application.

[0135] Furthermore, after obtaining the pose transformation information (i.e., the rotation and translation matrices between the scene point cloud and the sample point cloud), for each valid pixel in the scene depth map, the corresponding point in the scene point cloud (i.e., the target scene point) can be determined based on the correspondence between points in the scene point cloud and pixels in the scene depth map. Based on the pose transformation information, the target sample point corresponding to the target scene point is determined. And based on the correspondence between points in the sample point cloud and pixels in the sample depth map, the point in the sample depth map corresponding to the target sample point is determined as the matching pixel corresponding to the valid pixel in the aforementioned scene depth map. Correspondingly, the electronic device can also record the correspondence between each valid pixel and each matching pixel in the scene depth map.

[0136] Based on the above processing, scene point clouds corresponding to scene depth maps and sample point clouds corresponding to sample depth maps can be established. Calibration based on the scene point clouds and sample point clouds yields pose transformation information that reflects the pose transformation between the non-destructive object and the object to be detected. Furthermore, even when the shooting factors during scene depth map acquisition differ from those during sample depth map acquisition, the accuracy of matching pixels corresponding to each effective pixel in the determined scene depth map can be guaranteed, thus improving the accuracy of defect detection.

[0137] Regarding step S103, in one implementation, for each valid pixel in the scene depth map, the first residual value of the valid pixel is the absolute value of the difference between the pixel value of the valid pixel and the pixel value of the corresponding matching pixel.

[0138] In another implementation, the first residual value of the effective pixel can be determined by combining it with other pixels within a preset neighborhood of the effective pixel. For example... Figure 3 As shown, Figure 3 This is a schematic diagram of a third process for a surface defect detection method provided in an embodiment of this application. Figure 1 Based on this, step S103 includes:

[0139] S1031: Obtain the difference between the pixel value of each valid pixel in the second preset neighborhood of the scene depth map and the pixel value of its corresponding matching pixel.

[0140] S1032: Determine the minimum value among the differences as the first residual value of the valid pixel.

[0141] In this embodiment, for each valid pixel in the scene depth map, the second preset neighborhood to which the valid pixel belongs in the scene depth map is determined based on a neighborhood selection rule preset by an expert. For example, for any pixel in the scene depth map, a range of pixels of a specified size centered on that pixel can be determined as the second preset neighborhood to which the valid pixel belongs in the scene depth map. For example, the specified size can be 3×3, meaning the second preset neighborhood contains 9 pixels.

[0142] For any valid pixel within the second preset neighborhood of the scene depth map to which the valid pixel belongs, a matching pixel is determined. The absolute value of the difference between the pixel value of the valid pixel and the pixel value of the matching pixel is calculated, resulting in multiple absolute values. The minimum of these absolute values ​​is then determined as the first residual value of the valid pixel. It is understood that isolated noise may exist in the acquired scene depth map. If a valid pixel in the scene depth map is isolated noise, the absolute value of the difference between its pixel value and the pixel value of its corresponding matching pixel is usually large. However, the pixels within the second preset neighborhood of the valid pixel are usually not all noise; that is, there are pixels in the second preset neighborhood that are not affected by noise. Therefore, the minimum of the multiple absolute values ​​corresponding to the second preset neighborhood of the valid pixel can be determined as the first residual value of the valid pixel. This avoids the influence of isolated noise and improves the robustness of defect detection.

[0143] For step S104, surface defect detection can be performed based on the first residual value of each valid pixel to obtain the detection result of the object to be detected. The type of detection result can be set according to actual needs. For example, the detection result can indicate whether a defect exists on the surface of the object to be detected; or, the detection result can also indicate the location of the defect on the surface of the object to be detected.

[0144] Accordingly, the specific method for surface defect detection based on the first residual value of each effective pixel will be described in subsequent embodiments.

[0145] In some embodiments, after determining the first residual value of each valid pixel in the scene depth map, the first residual value of each valid pixel can be corrected to obtain a second residual value for each valid pixel. For example... Figure 4 As shown, Figure 4 This is a schematic diagram of the fourth process of the surface defect detection method provided in the embodiments of this application. Figure 1 Based on this, step S104 includes:

[0146] S1041: For each valid pixel in the scene depth map, obtain the correction coefficient corresponding to that valid pixel.

[0147] The correction coefficient corresponding to a valid pixel is used to reduce the first residual value of the valid pixel when the valid pixel does not belong to the surface area of ​​the object to be detected that needs to be inspected for surface defects.

[0148] S1042: Based on the correction coefficient corresponding to the effective pixel, the first residual value of the effective pixel is corrected to obtain the second residual value of the effective pixel.

[0149] S1043: Surface defect detection is performed based on the second residual value of each effective pixel to obtain the detection result of the object to be detected.

[0150] For example, when acquiring a scene depth map, the image acquisition device can take a picture at an angle where the optical axis of the image acquisition device is perpendicular to the surface area where surface defect detection is needed.

[0151] Understandably, due to the influence of the shooting angle, there are other surface areas in the scene depth besides the surface area currently being detected for surface defects. The angle between these other surface areas and the optical axis of the image acquisition device may be between 0° and 90°. Therefore, the first residual value of the effective pixels falling in these other surface areas may contain interference from the vertical component (i.e., the direction parallel to the optical axis of the image acquisition device). Thus, the first residual value of these effective pixels can be reduced by a correction coefficient, i.e., the interference from the vertical component can be eliminated. Consequently, when performing surface defect detection based on the second residual value, this part can be avoided to some extent from being identified as a defect area, improving the accuracy of the detection results.

[0152] In some embodiments, the correction coefficient corresponding to the effective pixel can be obtained in at least two ways. Accordingly, the correction coefficient corresponding to an effective pixel in the scene depth map includes a first correction coefficient and / or a second correction coefficient.

[0153] like Figure 5 As shown, Figure 5 This is a schematic flowchart illustrating the calculation of a first correction coefficient, provided as an embodiment of this application. Figure 5 In the scene depth map, the first correction coefficient corresponding to a valid pixel is calculated through the following steps:

[0154] S501: Determine the position of the matching pixel corresponding to the valid pixel in the lossless object.

[0155] S502: Determines the angle between the normal vector at the location and the normal vector of the specified surface region in the non-destructive object.

[0156] The specified surface area corresponds to the surface area in the object to be inspected that currently needs to be inspected for surface defects.

[0157] S503: Based on the determined included angle, determine the first correction coefficient corresponding to the effective pixel.

[0158] In this embodiment, for a valid pixel in the scene depth map, after determining the matching pixel corresponding to the valid pixel, the neighborhood (also referred to as the third preset neighborhood) to which the matching pixel belongs in the sample depth map can be determined. The method for determining the third preset neighborhood to which the matching pixel belongs in the sample depth map can refer to the description of determining the second preset neighborhood to which a valid pixel in the scene depth map belongs in the above embodiments. The size of the third preset neighborhood and the second preset neighborhood can be the same or different.

[0159] Furthermore, after determining the third preset neighborhood to which the matching pixel belongs in the sample depth map, the normal vector of the third preset neighborhood to which the matching pixel belongs can be determined as the normal vector (also called the first normal vector) at the corresponding position of the matching pixel in the lossless object. The determination of the normal vector at the corresponding position of the matching pixel in the lossless object can be achieved in various ways, for example, it can be based on the conventional Principal Component Analysis (PCA) method.

[0160] In one implementation, after determining the angle between the first normal vector and the normal vector (also called the second normal vector) of the specified surface region, the cosine of this angle can be used as the first correction coefficient for the valid pixel. Correspondingly, the second residual value of the valid pixel can be the product of the first correction coefficient and the first residual value. This eliminates interference from the vertical component in the first residual value.

[0161] For example, for any pixel in a vertical surface region (i.e., a surface perpendicular to the specified surface region) representing a non-destructive object in the sample depth map, the angle between the first normal vector and the second normal vector corresponding to the pixel is 90°, that is, the first correction coefficient corresponding to the pixel is 0 (Cos90°=0), and correspondingly, the second residual value of the effective pixel in the scene depth map corresponding to the pixel is also 0 (first residual value × 0=0).

[0162] Based on the above processing, the first residual value of the effective pixel can be reduced by a correction coefficient, that is, the interference of the vertical component is eliminated. Therefore, when performing surface defect detection based on the second residual value, this part can be avoided from being identified as a defect area to a certain extent. That is, defects on the horizontal plane of the object to be detected are retained, while defects on the vertical plane are filtered out, thus improving the accuracy of the detection results.

[0163] like Figure 6 As shown, Figure 6 This is a schematic diagram of a process for calculating a second correction coefficient, provided as an embodiment of this application. Figure 6 In the scene depth map, the second correction coefficient corresponding to a valid pixel is calculated through the following steps:

[0164] S601: Determine the first preset neighborhood of the matching pixel corresponding to the valid pixel in the sample depth map.

[0165] S602: Calculate the ratio of effective pixels in the first preset neighborhood to obtain the second correction coefficient corresponding to the effective pixel.

[0166] In this embodiment, the method for determining the first preset neighborhood of a matching pixel in the sample depth map corresponding to a valid pixel in the scene depth map can refer to the description of determining the second preset neighborhood of a valid pixel in the scene depth map in the above embodiments. The sizes of the first preset neighborhood and the second preset neighborhood can be the same or different.

[0167] The ratio of valid pixels within the first preset neighborhood is calculated, which is the ratio of the number of valid pixels in the first preset neighborhood to the total number of pixels contained in the first preset neighborhood. A larger ratio, closer to 1, indicates that the image region represented by the first preset neighborhood is more likely to be the image region occupied by the object to be detected; conversely, a smaller ratio, closer to 0, indicates that the image region represented by the first preset neighborhood is more likely to be an image region other than the image region occupied by the object to be detected. Furthermore, after obtaining the second correction coefficient of the valid pixel, the product of the second correction coefficient and the first residual value of the valid pixel can be calculated as the second residual value of the valid pixel.

[0168] It is understandable that isolated noise may exist in the acquired scene depth map. If a valid pixel in the scene depth map is an isolated noise point, and this valid pixel falls outside the image area occupied by the object to be detected, then the first preset neighborhood to which this valid pixel belongs is usually also outside the image area occupied by the object to be detected. That is, other pixels in the first preset neighborhood besides this valid pixel are invalid pixels. Therefore, based on the above processing, the second correction coefficient of a valid pixel can reflect the probability that the valid pixel falls within the image area occupied by the object to be detected, thereby reducing the impact of isolated noise on the detection results.

[0169] In one implementation, the electronic device can pre-calculate a first correction coefficient for each pixel in the sample depth map according to steps S501-S503 above; and pre-calculate a second correction coefficient for each pixel in the sample depth map according to steps S601-S602 above. Subsequently, when determining the correction coefficient of each valid pixel in the scene depth map, the first correction coefficient and / or the second correction coefficient of the matching pixel corresponding to that valid pixel can be directly obtained.

[0170] For a valid pixel in the scene depth map, if the correction coefficient corresponding to the valid pixel includes a first correction coefficient and a second correction coefficient, the first residual value can be corrected based on one of the correction coefficients to obtain the intermediate residual value of the valid pixel. Then, the intermediate residual value can be corrected again based on the other correction coefficient to obtain the second residual value.

[0171] In some embodiments, after obtaining the second residual value of each valid pixel in the scene depth map, the second residual value of each invalid pixel in the scene depth map can also be determined. For example... Figure 7 As shown, Figure 7 This is a schematic diagram of the fifth process of the surface defect detection method provided in the embodiments of this application. Figure 4 Based on this, prior to step S1043, the method further includes:

[0172] S105: For each invalid pixel in the scene depth map, if the corresponding pixel in the sample depth map is valid, then the second residual value of the invalid pixel is determined to be a preset outlier value.

[0173] S106: If the pixel corresponding to the invalid pixel in the sample depth map is invalid, then the second residual value of the invalid pixel is determined to be 0.

[0174] Step S1043 includes:

[0175] Step S10431: Perform surface defect detection based on the second residual value of invalid and valid pixels in the scene depth map to obtain the detection result of the object to be detected.

[0176] In this embodiment, the method for determining the pixel corresponding to each invalid pixel in the sample depth map is the same as the method for determining the pixel corresponding to each valid pixel in the sample depth map. Specifically, you can refer to the relevant description of step S102 in the above embodiment.

[0177] For each invalid pixel in the scene depth map, if the corresponding pixel in the sample depth map is valid (i.e., the pixel value of the pixel corresponding to the invalid pixel in the sample depth map is not empty), it indicates that the image features of the invalid pixel at its location in the scene depth map may differ from the image features of its corresponding pixel at its location in the sample depth map. The reasons for this situation can be varied, such as damage to the object being detected or an error in the acquired scene depth map.

[0178] Therefore, the second residual value of the invalid pixel can be set as a preset outlier. This allows for the identification of the image region containing the invalid pixel in the scene depth map as a defective region during subsequent surface defect detection based on the second residual values ​​of each pixel. The preset outlier value is pre-set by technicians according to actual needs; for example, it can be 10 times the maximum value of the second residual values ​​of all valid pixels in the scene depth map.

[0179] For each invalid pixel in the scene depth map, if the corresponding pixel in the sample depth map is invalid (i.e., its value is empty), it indicates that the invalid pixel's position in the real world, as represented by the scene depth map, falls outside the object to be detected. Similarly, the pixel corresponding to the invalid pixel's position in the real world, as represented by the sample depth map, also falls outside the undamaged object. Therefore, the second residual value of the invalid pixel can be set to 0. Consequently, when performing surface defect detection based on the second residual values ​​of each pixel, the image region to which the invalid pixel belongs in the scene depth map can be determined as a region without defects.

[0180] The second residual value of invalid and valid pixels in the scene depth map is obtained, that is, the second residual value of each pixel in the scene depth map is obtained. Then, surface defect detection can be performed based on the second residual value of each pixel in the scene depth map to obtain the detection result of the object to be detected. For the specific process of surface defect detection based on the second residual value of each pixel in the scene depth map, please refer to the following embodiments.

[0181] In one implementation, a residual map corresponding to the scene depth map can be generated based on the second residual values ​​of invalid and valid pixels in the scene depth map. That is, the second residual values ​​of each invalid and valid pixel in the scene depth map are determined as the pixel values ​​at the same positions in the residual map.

[0182] Since invalid pixels in the scene depth map may also be caused by defects in the object to be detected, based on the above processing, surface defect detection can be further performed on the image area represented by invalid pixels in the scene depth map, thereby improving the detection accuracy.

[0183] For step S104, the first residual value can also be used directly without correction, i.e., the first residual value can be used directly for surface defect detection. For example, surface defect detection can be performed based on the second residual value of invalid pixels in the scene depth map and the first residual value of valid pixels to obtain the detection result of the object to be detected. The specific processing procedure can be referred to the relevant description of step S1043. The method for obtaining the second residual value of invalid pixels in the scene depth map can be referred to the above embodiment.

[0184] In some embodiments, the detection result of the object to be detected can be obtained through a neural network. Step S1043 includes:

[0185] Step 1: Generate the corresponding grayscale image to be detected based on the second residual value of each valid pixel.

[0186] Step 2: Input the grayscale image to be detected into the pre-trained defect detection model to obtain the detection result of the object to be detected.

[0187] The defect detection model is trained based on sample grayscale images and corresponding sample labels. The sample grayscale images contain sample objects with surface defects, and the sample labels are used to represent the defects on the surface of the sample objects.

[0188] In this embodiment, a linear mapping method can be used to generate the grayscale image to be detected. For example, the maximum and minimum values ​​of the second residual values ​​of each effective pixel are determined. The pixel value corresponding to the minimum value in the grayscale image to be detected is set to 0, and the pixel value corresponding to the maximum value in the grayscale image to be detected is set to 255. The maximum pixel value and 255, and the minimum pixel value and 0 are substituted into the linear equation m = k·n + b to determine k and b, where n represents the second residual value of each effective pixel, m represents the pixel value of the pixel in the grayscale image to be detected, and k and b are constants. For the second residual values ​​of each effective pixel that fall between the minimum and maximum values, the above linear mapping formula m = k·n + b can be used for mapping. Mapping each pixel in each effective pixel will yield the grayscale image to be detected.

[0189] For example, sample grayscale images can be input into a neural network model for training. The loss value is obtained based on the sample labels, and the model parameters of the neural network model are adjusted based on the loss value until the obtained loss value is less than a preset loss threshold (e.g., 0.1) or the number of training times reaches a preset number of training times (e.g., 200 times), thus obtaining a trained defect detection model.

[0190] Furthermore, the grayscale image to be detected can be input into a pre-trained defect detection model to obtain the detection result of the object to be detected.

[0191] It is understandable that when it is necessary to detect whether there are defects on the surface of an object, the sample label indicates whether there are defects on the surface of the sample object represented by the sample grayscale image. Correspondingly, the detection result of the object to be detected based on this defect detection model can indicate whether there are defects on the surface of the object. Similarly, when it is necessary to detect the location of defects on the surface of an object, the sample label indicates the location of the defects on the surface of the sample object represented by the sample grayscale image. Correspondingly, the detection result of the object to be detected based on this defect detection model can indicate the location of the defects on the surface of the object.

[0192] Based on the above processing, defect detection models corresponding to different types of detection results can be pre-trained according to the actual needs of defect detection. Then, the residual map is processed based on the trained defect detection model to obtain the detection results, which can effectively detect surface defects of the object to be inspected.

[0193] In some embodiments, the detection result of the object to be detected can be obtained by conventional image processing methods. Step S1043 includes:

[0194] Step (1): Generate the corresponding grayscale image to be detected based on the second residual value of each effective pixel.

[0195] Step (2): Perform binarization on the grayscale image to be detected to obtain a binarized image.

[0196] Step (3): Extract connected components from the binarized graph.

[0197] Step (4): Determine the detection result of the object to be detected based on the extraction results.

[0198] The process of generating the grayscale image to be detected can be referred to the relevant description in step one of the above embodiments, and will not be repeated here.

[0199] After obtaining the grayscale image to be detected, for each pixel in the grayscale image, if the grayscale value of the pixel is greater than a preset grayscale threshold, then the grayscale value of the pixel is set as the first grayscale value; if the grayscale value of the pixel is less than the preset grayscale threshold, then the grayscale value of the pixel is set as the second grayscale value. This yields the binarized image corresponding to the grayscale image to be detected. The first grayscale value is greater than the second grayscale value; for example, the first grayscale value can be 255, and the second grayscale value can be 0. The preset grayscale threshold can be 0.1 or 10.

[0200] Furthermore, connected component extraction can be performed on the binarized graph, and the detection result of the object to be detected can be determined based on the extraction result.

[0201] In this embodiment, a connected component generally refers to an image region composed of foreground pixels with the same pixel value and adjacent positions. Connected component extraction of a binarized image is essentially the process of marking connected components; for example, connected component extraction can be performed based on run-length labeling. Connected component extraction based on run-length labeling includes the following steps:

[0202] For this binarized image, this embodiment of the application uses a first grayscale value of 255 and a second grayscale value of 0 as an example for description. After generating the binarized image corresponding to the grayscale image to be detected, the binarized image can be scanned line by line. The consecutive white pixels (i.e., pixels with a pixel value of 255) in each row are determined as a sequence, which can also be called a clique. A clique label is assigned to the clique, and the start point (column number of the first pixel in the sequence), end point (column number of the last pixel in the sequence), and row number of the clique are recorded.

[0203] For any cluster in any row other than the first row, if the cluster does not overlap with any cluster in the previous row, a new cluster number is assigned to the cluster. This cluster number is unique and greater than any other cluster number.

[0204] If a cluster has only one connected cluster, then the cluster number is determined to be the cluster number of the connected cluster. If a cluster has two or more connected clusters, then the minimum value among the cluster numbers of the connected clusters is determined, and this minimum value is set as the cluster number. The connected clusters and the cluster number of the current cluster are recorded to obtain an equivalence pair. This equivalence pair is used to indicate that the connected clusters and the current cluster belong to the same class. Specifically, for any cluster, its connected clusters refer to the clusters in the previous row that have overlapping areas with the current cluster.

[0205] Equivalence pairs are converted into equivalence sequences. Within an equivalence sequence, all sequences have the same label. Different equivalence sequences have different labels.

[0206] The process begins by determining the current clique to be processed in ascending order of clique indices. Then, it searches for cliques of the same class as the current clique in the equivalent sequence. The image regions represented by these cliques, along with the image regions represented by the current clique, are considered as a connected component. The process of determining the current clique in ascending order of clique indices is then repeated, and so on, until all cliques have been traversed. This process yields all connected components in the binarized image. Furthermore, preset filtering conditions can be used to filter the extracted connected components; for example, connected components with an aspect ratio less than 1 can be filtered out. The filtered connected components are then used as the detection results for the target object.

[0207] Based on the above processing, since the positions of the effective pixels in the scene depth map corresponding to the objects to be detected are the same as the positions of the corresponding matching pixels in the undamaged objects, that is, a single effective pixel in the scene depth map and its corresponding matching pixel correspond to the same position on different objects. Accordingly, the difference between the pixel value of the effective pixel and the pixel value of its corresponding matching pixel in the scene depth map (i.e., the first residual value) can reflect the difference between the objects to be detected and the undamaged objects at that position. Furthermore, the second residual value of the effective pixel can also reflect the difference between the objects to be detected and the undamaged objects at that position. Therefore, by processing the residual map using traditional image processing methods, the detection results can be obtained, effectively detecting surface defects in the objects to be detected.

[0208] like Figure 8 As shown, Figure 8 This is a flowchart for modeling a sample depth map, provided as an embodiment of this application.

[0209] S801: Acquire standard model depth map.

[0210] That is, obtain the sample depth map; where the sample depth map is obtained based on the non-destructive object corresponding to the object to be detected.

[0211] S802: Calculate the model point correction coefficient.

[0212] That is, calculate the first correction coefficient and / or the second correction coefficient for each pixel in the sample depth map.

[0213] S803: Construct a point cloud registration model.

[0214] That is, mapping the pixels in the sample depth map to a specified three-dimensional coordinate system.

[0215] S804: Standard model.

[0216] That is, the sample point cloud is obtained.

[0217] like Figure 9 As shown, Figure 9This is a flowchart of a surface defect detection process for an object to be tested, provided as an embodiment of this application.

[0218] S901: Acquire depth map.

[0219] That is, obtain the depth map of the object to be detected, and use it as the scene depth map.

[0220] S902: Convert to point cloud, point cloud registration.

[0221] That is, the pixels in the scene depth map are mapped to a specified three-dimensional coordinate system to obtain the scene point cloud.

[0222] S903: pose correction.

[0223] That is, to calculate the pose transformation information between the scene point cloud and the sample point cloud.

[0224] For each valid pixel in the scene depth map, based on the correspondence between points in the scene point cloud and pixels in the scene depth map, pose transformation information, and the correspondence between points in the sample scene point cloud and pixels in the sample depth map, the corresponding pixel in the sample depth map is determined as the matching pixel.

[0225] S904: Iterate through the minimum difference value in the neighborhood of each point.

[0226] That is, obtain the difference between the pixel value of each effective pixel in the second preset neighborhood of the scene depth map and the pixel value of its corresponding matching pixel; and determine the minimum value among the differences as the first residual value of the effective pixel.

[0227] S905: Defect area identified.

[0228] That is, surface defect detection is performed based on the first residual value of each effective pixel to obtain the detection result of the object to be detected.

[0229] like Figure 10 As shown, Figure 10 This is an example of surface defect detection provided in this application. Figure 10 The image in the upper middle section represents the scene depth map containing the object to be detected; Figure 10 The two images on the lower middle side represent the residual maps corresponding to the scene depth map on the upper side obtained based on the method provided in the embodiments of this application; Figure 10 The circles in the two images at the bottom center represent areas on the surface of the object being inspected where defects exist. Figure 10As can be seen, the residual map obtained by the surface defect detection method provided in this application can effectively reflect the areas where defects exist on the surface of the object to be detected. Furthermore, whether the residual map is processed by a neural network to obtain the detection result, or by a traditional image processing method, the areas where defects exist on the surface of the object to be detected can be effectively identified, improving the accuracy of the detection.

[0230] Based on the same inventive concept, embodiments of this application also provide a surface defect detection device. See also Figure 11 , Figure 11 A structural diagram of a surface defect detection device provided in this application embodiment is shown. The device includes:

[0231] The depth map acquisition module 1101 is used to acquire the depth map of the object to be detected as a scene depth map, and to acquire a sample depth map; wherein the sample depth map is obtained based on the non-destructive object corresponding to the object to be detected.

[0232] The matching module 1102 is used to determine, for each valid pixel in the scene depth map, the pixel in the sample depth map corresponding to the valid pixel as the matching pixel; wherein, the position of the valid pixel in the object to be detected is consistent with the position of the corresponding matching pixel in the lossless object.

[0233] The first residual value determination module 1103 is used to determine the first residual value of the effective pixel based on the difference between the pixel value of the effective pixel and the pixel value of the corresponding matching pixel.

[0234] The detection result acquisition module 1104 is used to perform surface defect detection based on the first residual value of each effective pixel point to obtain the detection result of the object to be detected.

[0235] In some embodiments, the detection result acquisition module 1104 includes:

[0236] The correction coefficient acquisition submodule is used to acquire the correction coefficient corresponding to each effective pixel in the scene depth map before performing surface defect detection based on the second residual value of each effective pixel to obtain the detection result of the object to be detected; wherein, the correction coefficient corresponding to an effective pixel is used to: reduce the first residual value of the effective pixel when the effective pixel does not belong to the surface area of ​​the object to be detected that currently needs to be detected for surface defect detection.

[0237] The second residual value determination submodule is used to correct the first residual value of the effective pixel based on the correction coefficient corresponding to the effective pixel to obtain the second residual value of the effective pixel.

[0238] The detection result acquisition submodule is used to perform surface defect detection based on the second residual value of each effective pixel point, and obtain the detection result of the object to be detected.

[0239] In some embodiments, the correction coefficient corresponding to a valid pixel in the scene depth map includes a first correction coefficient and / or a second correction coefficient;

[0240] The first correction coefficient corresponding to a valid pixel in the scene depth map is calculated through the following steps:

[0241] Determine the position of the matching pixel corresponding to the valid pixel in the non-destructive object; determine the angle between the normal vector at the position and the normal vector of a specified surface region in the non-destructive object; wherein the specified surface region corresponds to the surface region in the object to be inspected that currently needs to be inspected for surface defects; based on the determined angle, determine the first correction coefficient corresponding to the valid pixel;

[0242] The second correction coefficient corresponding to a valid pixel in the scene depth map is calculated through the following steps:

[0243] Determine the first preset neighborhood to which the matching pixel corresponding to the valid pixel belongs in the sample depth map; calculate the ratio of valid pixels in the first preset neighborhood to obtain the second correction coefficient corresponding to the valid pixel.

[0244] In some embodiments, the first residual value determination module 1103 includes:

[0245] The difference acquisition submodule is used to acquire the difference between the pixel value of each effective pixel in the second preset neighborhood of the scene depth map and the pixel value of its corresponding matching pixel.

[0246] The first residual value determination submodule is used to determine the minimum value among the differences as the first residual value of the valid pixel.

[0247] In some embodiments, the apparatus further includes:

[0248] The first invalid point setting module is used to determine the second residual value of the invalid pixel as a preset abnormal value before the surface defect detection is performed based on the second residual value of each valid pixel to obtain the detection result of the object to be detected. For each invalid pixel in the scene depth map, if the pixel corresponding to the invalid pixel in the sample depth map is valid, the second residual value of the invalid pixel is determined to be a preset abnormal value.

[0249] The second invalid point setting module is used to determine that the second residual value of the invalid pixel is 0 if the pixel corresponding to the invalid pixel in the sample depth map is invalid.

[0250] The detection result acquisition submodule is specifically used to: perform surface defect detection based on the second residual value of invalid and valid pixels in the scene depth map, and obtain the detection result of the object to be detected.

[0251] In some embodiments, the matching module 1102 is specifically used for:

[0252] The pixels in the scene depth map are mapped to a specified three-dimensional coordinate system to obtain the scene point cloud;

[0253] The pixels in the sample depth map are mapped to a specified three-dimensional coordinate system to obtain the sample point cloud.

[0254] Calculate the pose transformation information between the scene point cloud and the sample point cloud;

[0255] For each valid pixel in the scene depth map, based on the correspondence between points in the scene point cloud and pixels in the scene depth map, the pose transformation information, and the correspondence between points in the sample scene point cloud and pixels in the sample depth map, the corresponding pixel in the sample depth map is determined as the matching pixel.

[0256] In some embodiments, the detection result acquisition submodule is specifically used for:

[0257] Based on the second residual value of each effective pixel, a corresponding grayscale image to be detected is generated; the grayscale image to be detected is input into a pre-trained defect detection model to obtain the detection result of the object to be detected; wherein, the defect detection model is trained based on sample grayscale images and corresponding sample labels, the sample grayscale images contain sample objects with surface defects; the sample labels are used to represent the defects on the surface of the sample objects.

[0258] In some embodiments, the detection result acquisition submodule is specifically used for:

[0259] Based on the second residual value of each effective pixel, a corresponding grayscale image to be detected is generated; the grayscale image to be detected is binarized to obtain a binarized image; the binarized image is subjected to connected component extraction; and the detection result of the object to be detected is determined based on the extraction result.

[0260] This application also provides an electronic device, such as... Figure 12 As shown, it includes:

[0261] Memory 1201 is used to store computer programs;

[0262] When the processor 1202 executes the program stored in the memory 1201, it implements the steps of any of the surface defect detection methods in the above embodiments.

[0263] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 1202, the communication interface, and the memory 1201 communicating with each other via the communication bus.

[0264] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0265] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0266] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0267] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0268] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described surface defect detection methods.

[0269] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the surface defect detection methods described above.

[0270] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a storage medium (e.g., a solid-state drive (SSD)).

[0271] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0272] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0273] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A surface defect detection method characterized by, The method comprises: obtaining a depth map of a to-be-detected object as a scene depth map, and obtaining a sample depth map; wherein the sample depth map is obtained based on a lossless object corresponding to the to-be-detected object; for each valid pixel point in the scene depth map, determining a pixel point in the sample depth map corresponding to the valid pixel point as a matching pixel point; wherein the position corresponding to the valid pixel point in the to-be-detected object is consistent with the position corresponding to the matching pixel point in the lossless object; determining a first residual value of the valid pixel point based on the difference between the pixel value of the valid pixel point and the pixel value of the corresponding matching pixel point; for each valid pixel point in the scene depth map, obtaining a correction coefficient corresponding to the valid pixel point; wherein the correction coefficient corresponding to a valid pixel point is used to reduce the first residual value of the valid pixel point in the case that the valid pixel point does not belong to the surface region of the to-be-detected object that currently needs surface defect detection; correcting the first residual value of the valid pixel point based on the correction coefficient corresponding to the valid pixel point to obtain a second residual value of the valid pixel point; performing surface defect detection based on the second residual value of each valid pixel point to obtain a detection result of the to-be-detected object.

2. The method of claim 1, wherein, The correction coefficient corresponding to a valid pixel point in the scene depth map comprises a first correction coefficient and / or a second correction coefficient; The first correction coefficient corresponding to a valid pixel point in the scene depth map is calculated by the following steps: determining the position corresponding to the matching pixel point of the valid pixel point in the lossless object; determining the included angle between the normal vector at the position and the normal vector of a specified surface region in the lossless object; wherein the specified surface region corresponds to the surface region of the to-be-detected object that currently needs surface defect detection; determining the first correction coefficient corresponding to the valid pixel point based on the determined included angle; The second correction coefficient corresponding to a valid pixel point in the scene depth map is calculated by the following steps: determining the first preset neighborhood to which the matching pixel point corresponding to the valid pixel point belongs in the sample depth map; calculating the ratio of the valid pixel points in the first preset neighborhood to obtain the second correction coefficient corresponding to the valid pixel point.

3. The method according to any of claims 1-2, characterized in that, The first residual value of the valid pixel point is determined based on the difference between the pixel value of the valid pixel point and the pixel value of the corresponding matching pixel point, comprising: obtaining the difference between the pixel value of each valid pixel point in the second preset neighborhood to which the valid pixel point belongs in the scene depth map and the pixel value of the corresponding matching pixel point; determining the minimum value in each difference as the first residual value of the valid pixel point.

4. The method of claim 1, wherein, Before the surface defect detection based on the second residual value of each valid pixel point to obtain the detection result of the to-be-detected object, the method further comprises: for each invalid pixel point in the scene depth map, if the pixel point corresponding to the invalid pixel point in the sample depth map is valid, determining the second residual value of the invalid pixel point as a preset abnormal value; If a pixel corresponding to the invalid pixel in the sample depth map is invalid, the second residual value of the invalid pixel is determined as 0; The surface defect detection is performed based on the second residual values of the effective pixel points, and a detection result of the object to be detected is obtained. The surface defect detection is performed based on the second residual values of the invalid pixel points and the effective pixel points in the scene depth map, and a detection result of the object to be detected is obtained.

5. The method of claim 1, wherein, For each effective pixel point in the scene depth map, a pixel corresponding to the effective pixel point in the sample depth map is determined as a matching pixel point, including: Mapping the pixel points in the scene depth map to a specified three-dimensional coordinate system to obtain a scene point cloud; Mapping the pixel points in the sample depth map to the specified three-dimensional coordinate system to obtain a sample point cloud; Calculating pose transformation information between the scene point cloud and the sample point cloud; For each effective pixel point in the scene depth map, based on the correspondence between the points in the scene point cloud and the pixel points in the scene depth map, the pose transformation information, and the correspondence between the points in the sample scene point cloud and the pixel points in the sample depth map, the corresponding pixel point in the sample depth map is determined as a matching pixel point.

6. The method of claim 1, wherein, The surface defect detection is performed based on the second residual values of the effective pixel points, and a detection result of the object to be detected is obtained, including: Based on the second residual values of the effective pixel points, a corresponding detection grayscale image is generated; The detection grayscale image is input into a pre-trained defect detection model to obtain a detection result of the object to be detected; wherein the defect detection model is obtained by training based on a sample grayscale image and a corresponding sample label, the sample grayscale image containing a sample object with surface defects; and the sample label is used to represent the defects on the surface of the sample object.

7. The method of claim 1, wherein, The surface defect detection is performed based on the second residual values of the effective pixel points, and a detection result of the object to be detected is obtained, including: Based on the second residual values of the effective pixel points, a corresponding detection grayscale image is generated; The detection grayscale image is binarized to obtain a binary image; The binary image is subjected to connected domain extraction processing; The detection result of the object to be detected is determined according to the extraction result.

8. A surface defect detection apparatus characterized by comprising: The device includes: A depth map acquisition module is configured to acquire a depth map of an object to be detected as a scene depth map, and acquire a sample depth map; wherein the sample depth map is obtained based on a lossless object corresponding to the object to be detected; A matching module is configured to determine, for each effective pixel point in the scene depth map, a pixel corresponding to the effective pixel point in the sample depth map as a matching pixel point; wherein the position corresponding to the effective pixel point in the object to be detected is consistent with the position corresponding to the matching pixel point in the lossless object. A first residual value determination module is configured to determine a first residual value of the effective pixel point based on the difference between the pixel value of the effective pixel point and the pixel value of the corresponding matching pixel point. A detection result acquisition module includes: The correction coefficient obtaining submodule is configured to obtain a correction coefficient corresponding to each effective pixel point in the scene depth map before surface defect detection is performed based on the second residual value of each effective pixel point to obtain a detection result of the object to be detected; wherein the correction coefficient corresponding to one effective pixel point is used to reduce the first residual value of the effective pixel point in a case where the effective pixel point does not belong to a surface region of the object to be detected that currently needs to be subjected to surface defect detection. The second residual value determining submodule is configured to correct the first residual value of the effective pixel point based on the correction coefficient corresponding to the effective pixel point to obtain the second residual value of the effective pixel point. The detection result obtaining submodule is configured to perform surface defect detection based on the second residual value of each effective pixel point to obtain the detection result of the object to be detected.

9. The apparatus of claim 8, wherein, The correction coefficient corresponding to one effective pixel point in the scene depth map includes a first correction coefficient and / or a second correction coefficient. The first correction coefficient corresponding to one effective pixel point in the scene depth map is calculated by the following steps: determining a position corresponding to the matching pixel point of the effective pixel point in the lossless object; determining an included angle between a normal vector at the position and a normal vector of a specified surface region in the lossless object; wherein the specified surface region corresponds to a surface region of the object to be detected that currently needs to be subjected to surface defect detection; determining the first correction coefficient corresponding to the effective pixel point based on the determined included angle. The second correction coefficient corresponding to one effective pixel point in the scene depth map is calculated by the following steps: determining a first preset neighborhood to which the matching pixel point of the effective pixel point belongs in the sample depth map; calculating a ratio of effective pixel points in the first preset neighborhood to obtain the second correction coefficient corresponding to the effective pixel point.

10. The device of any of claims 8-9, wherein, The first residual value determining module includes: The difference value obtaining submodule is configured to obtain a difference value between a pixel value of each effective pixel point in a second preset neighborhood to which the effective pixel point belongs in the scene depth map and a pixel value of a corresponding matching pixel point of the effective pixel point. The first residual value determining submodule is configured to determine a minimum value in the difference values as the first residual value of the effective pixel point.

11. The apparatus of claim 8, wherein, The device further includes: The first invalid point setting module is configured to, before surface defect detection is performed based on the second residual value of each effective pixel point to obtain a detection result of the object to be detected, determine the second residual value of each invalid pixel point in the scene depth map as a preset abnormal value if a pixel point corresponding to the invalid pixel point in the sample depth map is valid. The second invalid point setting module is configured to determine the second residual value of the invalid pixel point as 0 if the pixel point corresponding to the invalid pixel point in the sample depth map is invalid. The detection result obtaining submodule is specifically configured to perform surface defect detection based on the second residual value of the invalid pixel point and the effective pixel point in the scene depth map to obtain the detection result of the object to be detected.

12. The apparatus of claim 8, wherein, The matching module is specifically configured to: Map the pixel points in the scene depth map to a specified three-dimensional coordinate system to obtain a scene point cloud; Map the pixel points in the sample depth map to a specified three-dimensional coordinate system to obtain a sample point cloud; Calculate the pose transformation information between the scene point cloud and the sample point cloud; For each valid pixel point in the scene depth map, based on the correspondence between the points in the scene point cloud and the pixel points in the scene depth map, the pose transformation information, and the correspondence between the points in the sample scene point cloud and the pixel points in the sample depth map, determine the corresponding pixel point in the sample depth map as a matching pixel point.

13. The apparatus of claim 8, wherein, The detection result acquisition submodule is specifically configured to: Generate a corresponding to-be-detected grayscale image based on the second residual values of the valid pixel points; Input the to-be-detected grayscale image into a pre-trained defect detection model to obtain the detection result of the to-be-detected object; wherein the defect detection model is obtained by training based on a sample grayscale image and a corresponding sample label, the sample grayscale image contains a sample object with a surface defect; and the sample label is used to represent the defect on the surface of the sample object.

14. The apparatus of claim 8, wherein, The detection result acquisition submodule is specifically configured to: Generate a corresponding to-be-detected grayscale image based on the second residual values of the valid pixel points; Perform binaryzation processing on the to-be-detected grayscale image to obtain a binaryzation image; Perform connected domain extraction processing on the binaryzation image; Determine the detection result of the to-be-detected object according to the extraction result.

15. An electronic device, comprising: It comprises: A memory for storing a computer program; A processor for executing the program stored on the memory to implement the method of any one of claims 1-7.

16. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method of any one of claims 1-7.

Images