Optimized path planning method for defect detection based on effective area coverage

Abstract

A movable camera is moved along an inspection path for optically inspecting an inspection surface of an object for defect detection. In planning the inspection path, a set of viewpoints on the inspection surface is generated. Each viewpoint is associated with a patch area which is the largest area of the inspection surface within the field of view of the camera when the camera is positioned above the viewpoint to capture an image of the field of view. An effective area of the patch area is advantageously predicted by a neural network from three-dimensional geometric features of the patch area such that the predicted effective area is effective for defect detection based on the captured image. The effective area refers to its corresponding area on the captured image being free of blur, neither underexposed nor overexposed. The inspection path is determined from the respective effective areas associated with the set of viewpoints.

Description

[0001] List of abbreviations

[0002] 2D

[0003] 3D

[0004] AutoML (Automatic Machine Learning)

[0005] DNN (Deep Neural Network)

[0006] FOV (Field of View)

[0007] NN Neural Network Technical Field

[0008] This invention relates generally to imaging the inspection surface of an object using a movable camera for defect detection. In particular, this invention relates to a computer-implemented method for planning the inspection path of a movable camera to perform optical inspection of the inspection surface for defect detection. Background Technology

[0009] Surface inspection is an important and necessary procedure in manufacturing and maintenance because it is related to quality control and even product safety. For example, surface inspection is useful for identifying cracks in train tracks or airplanes.

[0010] Traditionally, inspection processes involve judgments made by the human eye, leading to poor reliability. To ensure uniformity of inspection, automated or robotic inspection systems have been developed, offering the advantage of improved efficiency and productivity. One issue related to the inspection process is that cameras or imaging sensors can only cover a portion of the object being inspected from a specific viewpoint, necessitating the planning of the paths traversed by the camera or imaging sensor. Path planning methods have been developed in this field, for example... E.GLORIEUXA,P.FRANCIOSA andD.CEGLAREK,"Coverage Path Planning with Targeted Viewpoint Sampling forRobotic Free-Form Surface Inspection,"Robotics and Computer-IntegratedManufacturing,vol.61,February 2020,101843,WRSCOTT,G.ROTH and J.-F.RIVEST,"View Planning for Automated Three-Dimensional Object Reconstruction and Inspection, "ACM Computing Surveys, vol.35, no.1, March 2003, pp.64-96; US 8,059,151 B2, and CN113343355A.

[0011] Existing path planning methods primarily focus on providing full coverage of the inspection surface of the object being inspected, largely ignoring the impact of different optical properties of the surface on the inspection process. The inspection surface may be reflective, translucent, etc. If the camera is poorly positioned, receiving strong reflected light or unwanted glare, it can lead to an ineffective inspection process. In some cases, even if the object's surface is inspected using captured images, poor image quality may prevent proper inspection. Therefore, the inspection process is incomplete, and the results obtained are unreliable.

[0012] The desired outcome is that the inspection process completely covers the entire surface of the object being inspected, and that all captured images of the object are valid for processing by machine vision inspection algorithms. There is a need in the art for a technique to plan the inspection path through which the camera moves, thereby covering the entire inspection surface while avoiding movement to unfavorable positions that would produce invalid captured images. Summary of the Invention

[0013] A first aspect of the invention is to provide a computer-implemented method for planning an inspection path on an inspection surface of an object. The inspection path guides a movable camera to move along it for optical inspection of the inspection surface to detect defects.

[0014] In this method, a set of viewpoints is generated on the inspection surface. Each viewpoint is associated with a cube region, which is the largest area of ​​the inspection surface within the camera's field of view when the camera is positioned above the individual viewpoint to capture an image of the camera's field of view. Advantageously, the effective area of ​​the cube region is predicted by a neural network based on the three-dimensional geometric features of the cube region, thereby making the predicted effective area effective for defect detection based on the captured image associated with the individual viewpoint. The neural network predicts the effective area after training. An inspection path is determined based on multiple respective effective areas associated with the viewpoint set.

[0015] Preferably, the determination of the inspection path based on multiple respective valid regions of the block region associated with the viewpoint set includes the following first and second steps. In the first step, an optimized subset is determined from the viewpoint set such that the determined optimized subset is the smallest non-empty subset of the viewpoint set, and its constraint is the union of the block regions of the respective valid regions associated with different surviving viewpoints in the optimized subset, covering the entire inspection area. The optimized subset forms the surviving viewpoint set. In the second step, the inspection path is determined as an optimized path that connects all corresponding surviving viewpoints and visits each corresponding surviving viewpoint once, having the minimum movement cost among all possible paths, wherein each path connects all corresponding surviving viewpoints and visits each corresponding surviving viewpoint once.

[0016] Optimizing subsets can be determined based on the solution to the set covering problem.

[0017] The optimal path can be determined based on the solution to the open-loop traveling salesman problem.

[0018] Preferably, the three-dimensional geometric features are provided by multiple three-dimensional point data, which include the three-dimensional coordinates of the constituent points of the point cloud used to model the block region.

[0019] Also preferably, the effective region of the block region is predicted by a neural network, determining whether an individual component point of the block region is within or outside the effective region. Therefore, it allows the effective region to be represented by a point cloud that models the block region and indicates whether an individual component point is inside or outside the effective region.

[0020] In this method, a CAD model of the object is preferably obtained. This CAD model includes at least design data for geometrically modeling the surface under inspection. As a result, a set of viewpoints is generated based on this CAD model. Also preferably, after the effective region is predicted, the effective region associated with a single surviving viewpoint is projected onto the CAD model.

[0021] In some embodiments, the neural network is a deep neural network.

[0022] In some embodiments, the neural network is selected from PointNet++ and ConvPoint.

[0023] In some embodiments, the neural network is generated by automated machine learning tools.

[0024] In this method, a neural network can be trained using a training dataset. The training dataset may include multiple training samples. A single training sample may include a cube region used for training and a real, effective ground region.

[0025] In some embodiments, the method further includes preparing a training dataset.

[0026] The preparation of the training dataset includes the following actions. First, a first test image and a second test image of the training block region are received. The first and second test images are captured by a camera located at the same position above the training block region under white illumination and bright / dark bar illumination conditions, respectively, to illuminate the training block region. Second, the image block region is identified on the first test image. This image block region is the training block region appearing on the first test image. Third, based on the first test image, a first portion of the image block region is identified as either too bright or too dark. Fourth, based on the second test image, a second portion of the image block region is identified as out of focus. Fifth, the first and second portions are removed from the image block region to obtain a first two-dimensional effective region on the first test image. Sixth, a noise reduction operation is performed to remove false regions in any corner of the first two-dimensional effective region, thereby generating a second two-dimensional effective region. Seventh, a center region merging operation is performed on the second two-dimensional effective region to generate a third two-dimensional effective region, which eliminates false boundaries by merging adjacent regions near their centers in the second two-dimensional effective region. Eighth, acquire a second set of three-dimensional point data to model the cube region used for training. Ninth, form a third set of three-dimensional point data from the second set of three-dimensional point data and the third set of two-dimensional effective regions to model the ground's true effective region, thereby determining the ground's true effective region for the cube region used for training.

[0027] Preferably, preparing the training dataset further includes determining whether the block region for training can be used for training by checking the following: whether the image block region has insufficient contrast; whether there is an abnormal brightness distribution on the first or second test image; whether the determined ground truth region has an abnormal geometric center; and whether there are one or more holes larger than a predetermined size in the determined ground truth region.

[0028] A second aspect of the present invention is to provide a computer-implemented method for detecting defects in one or more samples of an object.

[0029] The method includes planning an inspection path on an inspection surface of an object, the inspection path guiding a movable camera along the inspection path to perform optical inspection of the inspection surface for defect detection, wherein the inspection path is planned according to any embodiment of the planning method disclosed herein; processing one or more object samples for defect detection, wherein the inspection path is used by the camera to optically inspect the one or more object samples; and, while processing a single object sample for defect detection, controlling the camera to move along the inspection path and capturing a first image of the field of view associated with a single viable viewpoint on the single object sample when the camera is positioned above the single viable viewpoint, thereby processing the captured first image to detect any possible defects on the single object sample.

[0030] Other aspects of the invention are disclosed as illustrated in the following embodiments. Attached Figure Description

[0031] Figure 1 This shows a typical image of a portion of the area of ​​an object being inspected, captured by a camera used for optical inspection.

[0032] Figure 2 A flowchart illustrating exemplary steps taken during the planning process is provided to illustrate a method for planning inspection paths on the inspection surface of an object, as disclosed herein.

[0033] Figure 3 It provides a graphical description of the generation of multiple viewpoints on a single example object, where the generation of multiple viewpoints participates in the planning process.

[0034] Figure 4 An example is provided for generating a square region by positioning the camera above a viewpoint and guiding the camera to focus on that viewpoint to capture an image for defect detection, thereby identifying the square region in the image.

[0035] Figure 5 Showing for Figure 4An illustrative example of a valid region defined by a block region, wherein the valid region of the block region is suitable or effective for defect detection.

[0036] Figure 6 A flowchart is shown illustrating an exemplary basic process taken when preparing a training dataset for a neural network to be used in the training planning process, wherein the basic process is used to generate training samples by generating ground-based real effective regions of square regions for training.

[0037] Figure 7 This shows an example of projecting a bright / dark bar pattern onto a real-life object when a camera captures images used to prepare a training dataset.

[0038] Figure 8 A flowchart illustrating an exemplary defect detection process for demonstrating the method disclosed herein for defect detection of one or more samples of an object is shown, wherein the defect detection process uses a planning process to plan an inspection path for optical inspection of one or more object samples.

[0039] Those skilled in the art will understand that the components shown in the accompanying drawings are for simplicity and clarity and are not necessarily depicted to scale. Detailed Implementation

[0040] Unless otherwise stated, the term "object" as used herein refers to an item used for inspection to determine whether any potential defects are present on that object. In one example, the object is a railway track, and the defect is a crack in the track. In another example, the object is a car door, and the defect is discoloration on some external areas of the car door.

[0041] An object has an outer surface. Here, the "inspection surface" of an object refers to an area of ​​the outer surface intended to be inspected to identify any potential defects. Depending on the specific circumstances, the inspection surface may be the entire outer surface of the object or only a portion thereof. Note that the inspection surface exists in three-dimensional space and may be curved within that space.

[0042] As used in this article, an "object sample" or "sample of an object" is an item manufactured using an object as a design model during the creation of an object sample. In other words, an object can be considered a design prototype, while an object sample is a manufactured version of that object. Typically, a large number of object samples are manufactured in practice.

[0043] As used herein, a "viewpoint" or "camera viewpoint" is a position on the inspection surface where a camera is moved to perform optical inspection of the surface. Moving the camera to the viewpoint means that the camera is positioned above the viewpoint by maintaining a predetermined non-zero distance from it, allowing the camera to capture an image of the viewpoint and its vicinity on the inspection surface. This image has a field of view (FOV), which is the extent of the view captured by the camera in the image.

[0044] The viewpoint is associated with a “patch,” which in this paper is considered the largest area of ​​the surface to be examined within the camera’s FOV when the camera is positioned above the viewpoint to capture an image. Without reference to the relevant viewpoint, the patch can be identified from the image as the largest area of ​​the surface to be examined within the camera’s FOV. Note that the patch is a surface in three-dimensional space and may be curved in three-dimensional space. As used in this paper, the “patch-on-image” associated with the patch is a two-dimensional portion of the image, where the two-dimensional image portion corresponds to the patch.

[0045] A first aspect of the invention discloses a method for planning an inspection path on an inspection surface of an object, wherein the inspection path guides a movable camera to move along the path for optical inspection of the inspection surface to detect defects. The objective of the disclosed method is that the camera covers the entire inspection surface for optical inspection while avoiding camera movement to unfavorable positions that would result in invalid captured images unusable for defect detection.

[0046] Before going into detail about the disclosed methods, it is enlightening to show typical images captured by a camera for defect detection. Figure 1A typical image 100 is described, depicting a portion of an object's inspection area captured by a camera. This image 100 has a field of view (FOV) 105. The object being optically inspected, or the inspection area, is typically larger than the FOV 105. A square region 110 is identified as part of the inspection area within the FOV and is visible to the camera. In the art, it is generally assumed that the entire unclosed surface of the square region 110 captured within the FOV 105 is effective for defect inspection. However, this assumption is not always correct. First, the quality of image 100 across the entire FOV 105 may not be high enough to display sufficiently clear detail for optical inspection and defect detection purposes. Specifically, some areas may have poor image quality, often due to blurring or out-of-focus areas caused by camera problems or imperfections. Perhaps only a specific area (typically the center) is effective for detection, while other areas, although captured, are unsuitable for inspection purposes. This defect may occur, for example, if the camera lens is not designed to provide sufficient sharpness at the periphery of image 100. Due to poor image quality, a first invalid region 121 unsuitable for optical inspection was identified on image 100. Secondly, one or more portions of the block region 110 on image 100 may be underexposed or overexposed. In one example, if the block region 110 is glossy (i.e., has high reflectivity with specular reflection), a portion of the block region 110 may become overexposed on image 100. In another example, the block region 110 is not illuminated sufficiently uniformly, such that only a portion (typically the central portion) of image 100 is suitable for optical inspection, while the outer areas are unsuitable. A second invalid region 122 unsuitable for optical inspection was identified on image 100. The remaining area of ​​the block region 110 outside the first and second invalid regions 121, 122 is referred to as the valid region 130, which is suitable or effectively used for defect detection. The disclosed method involves identifying valid regions on different captured images.

[0047] With the help of Figure 2 The disclosed method is described below, with the figure showing a flowchart illustrating exemplary steps taken in a process 200 of planning an inspection path on an inspection surface of an object according to certain embodiments of the invention. This inspection path guides a movable camera to move along it for optical inspection of the inspection surface, thereby detecting defects.

[0048] If the three-dimensional geometric features of the object's inspection surface (i.e., a three-dimensional geometric model, or three-dimensional design data) are available, the planning process 200 may begin at step 240. In step 240, a set of viewpoints on the inspection surface is generated. A single viewpoint is associated with a cube region. This cube region is the largest area of ​​the inspection surface within the camera's FOV when the camera is positioned above the single viewpoint to capture an image of the FOV. To illustrate the point, Figure 3 Illustrations are provided for generating multiple viewpoints (e.g., viewpoints 321-324) for example object 310. As a rule of thumb, in most practical cases, a set of 1000 viewpoints can be used to cover the entire inspection surface. If the inspection surface is occupied by more curved areas, more viewpoints can be included in that set. Figure 4 An example is provided of a square region 420 that is generated by positioning the camera 480 above the viewpoint 425 (located on the inspection surface 410) and guiding the camera 480 to focus on the viewpoint 425 when capturing an image for defect detection, so that the square region 420 (as part of the inspection surface 410) is identified in the image.

[0049] The locations of the different viewpoints used to prepare the viewpoint set can be randomly generated across the entire inspection surface. Alternatively, these locations can be generated using other viewpoint sampling techniques, such as those mentioned above by E. GLORIEUXA, P. FRANCIOSA, and D. CEGLAREK.

[0050] After generating multiple viewpoints in step 240, in step 250, after the neural network is trained, the effective region of the square region (associated with each individual viewpoint) is predicted by the neural network. Specifically, the effective region is predicted by the neural network based on the three-dimensional geometric features of the square region, thereby making the predicted effective region effective for defect detection based on the captured image associated with each individual viewpoint. As mentioned above, the region on the square region that is effective for defect detection is one whose corresponding region on the captured image is (1) not blurred and (2) neither underexposed nor overexposed. Note that since the effective region is part of the square region, the effective region is a surface in three-dimensional space and may be curved in three-dimensional space. To illustrate the point, Figure 5 Described as Figure 4 An example of a valid region 520 defined by a block region 420. This valid region 520 is a curved region. Although in Figure 5Viewpoint 425 is located within the effective region 520, but this is not always the case. A viewpoint may be located outside its corresponding effective region, even though that effective region is still related to the aforementioned viewpoint. After multiple viewpoints are processed by the neural network, multiple respective effective regions associated with that set of viewpoints are obtained. The inspection path can then be determined based on these multiple respective effective regions, for example, through steps 260 and 270.

[0051] Preferably, the neural network is a deep neural network (DNN). As used herein, a "deep neural network" is a multi-layer neural network with at least one hidden layer. Deep learning enables more accurate predictions of effective regions.

[0052] Currently, most widely available neural networks (including deep neural networks) used for processing curved surfaces, such as PointNet++ and ConvPoint, accept a point cloud modeling the curved surface as input to these neural networks. In step 250, preferably, the three-dimensional geometric features of the cube region are given by a plurality of three-dimensional point data, wherein the plurality of three-dimensional point data includes the three-dimensional coordinates of the constituent points of the point cloud modeling the cube region.

[0053] If the cube region is modeled by a point cloud, then a preferred approach is for step 250 to include determining whether an individual component point of the cube region is within or outside the valid region. Through this selection, the neural network is trained to perform binary segmentation for each component point. Furthermore, it allows the valid region to be represented by a point cloud that models the cube region and indicates for each individual component point whether that individual component point is within or outside the valid region. Note that the neural network can be configured to take a point cloud as input and output a map that indicates for all component points in the point cloud whether an individual component point is within or outside the valid region.

[0054] In the actual implementation of the planning process 200, the neural network can be selected from PointNet++ and ConvPoint. Alternatively, the neural network can be generated by AutoML tools (automated machine learning tools) during the startup phase.

[0055] After obtaining multiple valid regions associated with the viewpoint set in step 250, the viewpoint set is minimized while maintaining the entire inspection area to be covered by the optical inspection performed by the camera. During minimization, some viewpoints in the set are discarded, and some are retained. The retained viewpoints are called survived viewpoints. In step 260, specifically, an optimal subset of the viewpoint set is determined, wherein the optimal subset forms the survived viewpoint set, such that the determined optimal subset is the smallest non-empty subset of the viewpoint set, constrained by the union of the corresponding valid regions of the block regions associated with different survived viewpoints in the optimal subset, covering the entire inspection area. With regard to this constraint, since the valid region is a surface in three-dimensional space, the "union" of multiple surfaces in three-dimensional space is obtained by merging all surfaces to form a single surface object in three-dimensional space, such that (1) the first point on any surface exists on the single surface object, and (2) the second point on the single surface object exists on one or more surfaces. The formation of this surface union is similar to the union operation under set theory. As used in this paper, in a set of elements under certain constraints, the "minimum non-empty subset" refers to the non-empty subset with the fewest number of elements among all possible non-empty subsets, where each subset is derived from this set of elements and satisfies the aforementioned constraint. In step 260, the optimal subset can be determined based on known solutions to set covering problems in the field of combinatorial optimization.

[0056] Based on the determined set of viewpoints, in step 270, the inspection path is determined as the optimal path that connects all corresponding surviving viewpoints and visits each surviving viewpoint once, and minimizes the camera's movement cost among all possible paths, each of which connects all corresponding surviving viewpoints and visits each surviving viewpoint once. The movement cost can be the total distance of the inspection path or the total time required for the camera to travel from one end of the inspection path to the other. Other choices of movement cost that are deemed suitable by those skilled in the art are possible. If the movement cost is the total distance of the inspection path, then the optimal path is the shortest path among all possible paths connecting all corresponding surviving viewpoints. In step 270, the optimal path can be determined based on solutions to the open-loop traveling salesman problem known in the field of combinatorial optimization.

[0057] In some embodiments, the planning process 200 further includes a step 230 of obtaining a CAD model of the object prior to step 240 of generating a viewpoint set. The CAD model includes at least design data for geometrically modeling the surface under inspection. The viewpoint set can then be generated from the CAD model in step 240.

[0058] CAD models can be selected as geometric models that designers can easily use when designing objects. For example, this geometric model can be a polygon mesh, which is a set of vertices, edges, and faces that define the outer surface of an object. Although CAD models are usually chosen for ease of industrial design, the 3D geometric features of the blocky regions required by the neural network in step 250, such as point cloud models of the blocky regions, are often not directly compatible with CAD models. Therefore, when performing step 250, it is also necessary to convert the CAD model into the aforementioned 3D geometric features.

[0059] After the effective area of ​​the block region is predicted in step 250, the 3D model representing the effective area, such as the map described above, is not directly compatible with the CAD model. In some embodiments, the planning process 200 further includes step 280, after the effective area is predicted in step 250 and the set of surviving viewpoints is determined in step 260, projecting the effective area associated with a single surviving viewpoint onto the CAD model. Specifically, multiple respective effective areas associated with the set of surviving viewpoints are projected onto the CAD model. Thus, the CAD model is expanded to include additional information about the set of surviving viewpoints and multiple respective effective areas associated with the surviving viewpoint set. This additional information can then be used for any appropriate purpose. In particular, with information about the effective areas, any possible defects in searching for objects based on the captured image are limited to the effective areas rather than the entire block region.

[0060] In step 250, a prerequisite for the neural network is that it is trained. The neural network can be trained by simply loading pre-computed neural network model weights into it, if those weights are available. Alternatively, the neural network can be set up during the training phase and trained with training samples.

[0061] In some embodiments, the planning process 200 further includes Train a neural network using the training dataset Step 220. The training dataset includes multiple training samples. In fact, a neural network can be trained using multiple training samples from real-world objects.

[0062] In the first arrangement (which is preferred), the real object used for training contains a wide range of surface optical properties, such as surface reflectivity, the degree of diffusion of reflected light, and object transparency. Therefore, the neural network is trained such that the influence of different surface optical properties on the predicted effective region is reduced. This allows the neural network to predict the effective region in step 250 based on the three-dimensional geometry of the cube region without requiring its surface optical features as input. Once the neural network is trained, it can be applied to test on any new object (with different shapes, different degrees of reflectivity, etc.).

[0063] In the second arrangement, the real-world objects used for training have the same surface optical properties as the objects used for testing. The surface properties of the test object are taken into account when predicting the effective region. Similarly, the surface optical properties of the object do not need to be input into the neural network during prediction. Likewise, once the neural network is trained, it can be applied to test any new object. Reliably, to achieve the same prediction accuracy after neural network training, the second arrangement requires fewer training samples than the first arrangement.

[0064] In both arrangements, a procedure is required to derive multiple training samples. In some embodiments, the planning process 200 further includes a step 210 of preparing a training dataset. A single training sample in the training dataset includes a cube region for training and a ground truth region. The cube region for training is the cube region used to train the neural network. The ground truth region is the valid region of the cube region used for training.

[0065] In some embodiments, the cube regions used for training are modeled from point clouds. Therefore, the training dataset is developed for the specific case where the neural network accepts point clouds as input. Step 210 includes a basic process for generating the ground-based, effective regions of the cube regions used for training. Figure 6 An exemplary basic process 600 for preparing a training dataset for the specific situation described above is shown.

[0066] In step 610, first and second test images of the training block region are received or acquired. These two images are captured by a camera. Specifically, the first and second test images are captured by a camera located at the same position above the training block region under white illumination and under bright / dark bar illumination conditions illuminating the training block region, respectively.

[0067] White lighting is the first lighting condition used to illuminate the training cube area with ordinary white light with a pre-selected color temperature. White lighting provides uniform illumination to the training cube area to reveal its details.

[0068] The bright / dark bar lighting condition is a second lighting condition that projects a light pattern consisting of alternating bright and dark bars onto the square area used for training. To illustrate this, Figure 7 An example is shown where a display 725 projects a bright / dark bar pattern 720 onto a real object 710. The bright / dark bar illumination condition introduces contrast into a training block region captured on a second test image. The introduced contrast helps to identify a portion of the training block region that is blurred (out of focus) on the second test image.

[0069] In step 610, the camera's optics / hardware are configured to operate in the same normal operating mode as the mode used for defect detection of objects.

[0070] In step 620, patch-on-image regions are identified on the first test image, where the patch-on-image regions are the training patch regions that appear on the first test image. Those skilled in the art will understand that image segmentation techniques can be used to distinguish and identify the patch-on-image regions from the background on the first test image.

[0071] In step 630, a first portion of the overexposed or underexposed image block region is identified based on the first test image. This first portion can be identified by comparing the image block region on the first test image pixel-by-pixel with a predetermined maximum allowable gray level threshold and a predetermined minimum allowable gray level threshold. As mentioned above, overexposure or underexposure results in poor image quality, and overexposed or underexposed image regions are unsuitable for optical inspection. It is preferable to discard the first portion from the image block region.

[0072] In step 640, a second portion of the out-of-focus image patch region is determined based on the second test image. Since the images of the training patch regions on the first and second test images are identical, the first and second test images can be aligned based on the image mentioned last above. Mapping the training patch regions between the first and second test images is possible. The second test image has a corresponding image patch region. Based on the second test image, the out-of-focus region of the corresponding image patch region can be identified. Out-of-focus is identifiable under bright / dark bar illumination conditions. Then, the out-of-focus region on the second test image is mapped to a second portion of the image patch region on the first test image. Preferably, the second portion is discarded from the image patch region patch.

[0073] In step 650, the first and second portions are removed from the image block regions to generate a first two-dimensional effective region on the first test image. The first two-dimensional effective region is then processed to generate the ground truth effective region for the block regions used in training.

[0074] It is advantageous to remove outliers from the first two-dimensional valid region. In step 660, a denoising operation is performed to remove spurious regions around any corner of the first two-dimensional valid region to generate a second two-dimensional valid region on the first test image.

[0075] After performing various image processing operations, the second two-dimensional effective region may be fragmented. It is advantageous to merge the scattered regions near the second two-dimensional effective region so that the true effective ground region is not over-segmented. In step 670, a center region merging operation is performed on the second two-dimensional effective image. This operation eliminates false boundaries by merging adjacent regions near their centers within the second two-dimensional effective region. As a result of this operation, a third two-dimensional effective region is obtained on the first test image.

[0076] When preparing the training dataset, the third 2D effective region is considered as a projection of the ground truth effective region on the first test image. Note that the third 2D effective region is still a planar object. It is necessary to map the third 2D effective region to the ground truth effective region in 3D space. In step 680, a second set of multiple 3D point data is first obtained to model the cube region used for training. In step 690, a third set of multiple 3D point data simulating the ground truth effective region is formed from the second set of multiple 3D point data and the third 2D effective region. Thus, the ground truth effective region of the cube region used for training is obtained.

[0077] Due to inherent problems in the samples, it is not always possible to determine the corresponding ground truth regions from any samples of the block regions used for training. The corresponding ground truth regions may not even exist. Therefore, if inherent problems with the block regions used for training are identified, the basic process 600 can be stopped, and human intervention may be necessary.

[0078] In some embodiments, step 210 of preparing the training dataset further includes determining whether the block region for training is usable for training by checking the following: whether the block region on the image has insufficient contrast; whether there is an abnormal brightness distribution on the first or second test image; whether there is an abnormal geometric center in the ground truth region determined in basic process 600; and whether there are one or more holes larger than a predetermined size in the determined ground truth region. In the case of low contrast across the entire image, defect inspection cannot be performed normally, even under bright / dark bar illumination conditions. This indicates a serious optical / hardware problem and therefore requires human attention. If the image under consideration (i.e., the first or second test image) exhibits an abnormal brightness distribution, it also signifies a serious optical / hardware problem, such as when a large portion of the top / bottom / left / right side of the image under consideration is completely black or completely white. This also requires human attention. In the case of an abnormal geometric center in the ground truth region, this geometric center is far from the center of the first or second test image. This indicates that the optical / hardware setup is not properly aligned. The discovery of one or more holes larger than a certain size within the ground truth region requires human attention.

[0079] although Figure 2 A specific scenario is described to illustrate that the planning process 200 is performed in the order of steps 210, 220, 230, 240, 250, 260, 270, and 280. However, those skilled in the art will understand that the order of some of these steps can be modified without altering the overall result of the planning process 200. For example, step 230 may be performed before step 220.

[0080] Similarly, Figure 6 A specific scenario is described to illustrate that the basic process 600 is executed in the order of steps 610, 620, 630, 640, 650, 660, 670, 680, and 690. Nevertheless, those skilled in the art will understand that the order of some of these steps can be modified without altering the overall result of the basic process 600. For example, step 680 may be performed before step 610.

[0081] The planning process 200 can be executed or implemented by one or more computing processors, or by one or more computers or computing servers. The disclosed method for planning inspection paths can be a computer-implemented method.

[0082] As a note, the disclosed method has the following advantages.

[0083] First, planning the inspection path based on the effective area method avoids the occurrence of invalid inspection processes.

[0084] Second, check that the surface is completely covered.

[0085] Third, once the neural network is trained, it can be applied to any new object. There is no need to reconfigure the camera's optics / hardware or prepare training data to retrain the neural network for inspecting new objects. A neural network already trained to plan an inspection path for one object can be directly reused to plan another inspection path for a new object.

[0086] Fourth, the disclosed method can quickly plan inspection paths.

[0087] A second aspect of the invention discloses a method for defect detection on one or more samples of an object. In defect detection, a movable camera is guided to move along an inspection path on the inspection surface of a single object sample to perform optical inspection of the inspection surface, thereby detecting potential defects or multiple defects. A planning process 200 is used to plan the inspection path.

[0088] With the help of Figure 8 The disclosed method is illustrated in the figure, which depicts a flowchart of an exemplary defect detection process 800 for detecting defects in one or more object samples.

[0089] The defect detection process 800 employs any embodiment of the planning process 200 disclosed above to plan the inspection path. The defect detection process is then performed on one or more object samples. The planned inspection path is used by a camera during the optical inspection of each of the one or more object samples.

[0090] When processing a single object sample for defect detection (step 820), the camera is controlled to move along the inspection path and captures an image of the field of view (FOV) of a single surviving viewpoint (selected from the set of surviving viewpoints determined in planning process 200) on the single object sample. The captured image is then processed to detect any possible defects on the single object sample.

[0091] Step 820 is repeated until all or more object samples have been processed (step 830).

[0092] The defect detection process 800 can be executed or implemented by one or more computing processors, or by one or more computers or computing servers. The disclosed method for performing defect detection on one or more object samples can be a computer-implemented method.

[0093] This invention may be embodied in other specific forms without departing from its spirit or essential characteristics. Therefore, embodiments of the invention should be considered illustrative rather than restrictive in all respects. The scope of the invention is defined by the appended claims rather than the foregoing description, and thus all variations within the meaning and equivalence of the claims are to be included therein.

Claims

1. A computer-implemented method for planning an inspection path on an inspection surface of an object, the inspection path guiding a movable camera along the inspection path to perform optical inspection of the inspection surface for defect detection, the method comprising: A viewpoint set is generated on the inspection surface; Individual viewpoints are associated with square regions; When the camera is positioned above this single viewpoint to capture an image of the camera's field of view, the square area is the largest area of ​​the inspection surface within the camera's field of view; Based on the three-dimensional geometric features of the square region, a neural network predicts the effective region of the square region, thereby making the predicted effective region effective for defect detection based on the captured image associated with the individual viewpoint, wherein the neural network predicts the effective region after training; and The inspection path is determined based on multiple valid regions associated with the viewpoint set.

2. The method of claim 1, wherein determining the inspection path based on a plurality of respective valid regions of the block regions associated with the viewpoint set includes: Determine an optimal subset of the viewpoint set to form a set of surviving viewpoints, such that the determined optimal subset is the smallest non-empty subset of the viewpoint set, and its constraint is the union of the block regions of each effective region associated with different surviving viewpoints in the optimal subset, covering the entire inspection area. as well as The inspection path is determined to be an optimized path that connects all the corresponding surviving viewpoints and visits each surviving viewpoint once, while minimizing the camera's movement cost among all possible paths that connect all the corresponding surviving viewpoints and visit each surviving viewpoint once.

3. The method according to claim 1, wherein the three-dimensional geometric features are given by a plurality of three-dimensional point data, which include the three-dimensional coordinates of the constituent points of a point cloud modeling the cube region.

4. The method according to claim 3, wherein, The prediction of the effective region of the square region by the neural network includes: Determine whether a single component point of the block region is within or outside the effective region, thereby allowing the effective region to be represented by the point cloud modeling the block region, and indicating whether the single component point is within or outside the effective region.

5. The method according to claim 1, further comprising: A CAD model of the object is obtained, which includes at least design data for geometric modeling of the inspected surface, thereby generating the viewpoint set based on the CAD model.

6. The method of claim 5, further comprising: After the effective region is predicted, the effective region associated with a single surviving viewpoint is projected onto the CAD model.

7. The method of claim 3, wherein the neural network is a deep neural network.

8. The method of claim 3, wherein the neural network is selected from PointNet++ and ConvPoint.

9. The method of claim 3, wherein the neural network is generated by an automated machine learning tool.

10. The method according to claim 2, wherein, The optimized subset is determined based on the solution to the set coverage problem.

11. The method of claim 2, wherein the optimized path is determined based on a solution to the open-loop traveling salesman problem.

12. The method according to claim 3, further comprising: The neural network is trained using the training dataset.

13. The method of claim 12, wherein the training dataset comprises a plurality of training samples, and a single training sample comprises a block region for training and a ground real effective region.

14. The method of claim 13, further comprising preparing the training dataset, wherein preparing the training dataset includes: Receive a first test image and a second test image of the training block region, wherein the first test image and the second test image are captured by a camera located at the same position above the training block region while illuminating the training block region under white lighting conditions and bright / dark bar lighting conditions, respectively; Identify image block regions on the first test image, wherein the image block regions are the block regions for training that appear on the first test image; Based on the first test image, identify whether the first part of the image block region is too bright or too dark; Based on the second test image, the second part of the image block region is identified as out of focus; The first portion and the second portion are removed from the image block region to generate a first two-dimensional valid region on the first test image; A noise reduction operation is performed to remove any false regions in any corner of the first two-dimensional effective region, thereby generating a second two-dimensional effective region. A center region merging operation is performed on the second two-dimensional effective region to generate a third two-dimensional effective region, which is used to eliminate false boundaries by merging adjacent regions near the center of the second two-dimensional effective region. Acquire a second set of three-dimensional point data, which are used to model the block region for training; as well as A third set of three-dimensional point data is formed from the second set of three-dimensional point data and the third set of two-dimensional effective regions. This third set of three-dimensional point data is used to model the ground real effective region, thereby determining the ground real effective region of the block region used for training.

15. The method according to claim 14, wherein, The preparation of the training dataset further includes: Whether the square region used for training is suitable for training is determined by checking the following: Does the image block region have insufficient contrast? Does the first test image or the second test image have an abnormal distribution of brightness? Does the determined true effective area of ​​the ground have an anomalous geometric center of that true effective area of ​​the ground? as well as Does the defined effective area of ​​the ground contain one or more holes larger than the predetermined size? 16. A computer-implemented method for defect detection of one or more object samples, the method comprising: An inspection path is planned on the inspection surface of an object, the inspection path being used to guide a movable camera to move along the inspection path to perform optical inspection of the inspection surface for defect detection, wherein the inspection path is planned by the planning method according to claim 1. Processing the one or more object samples for defect detection, wherein the inspection path is used by the camera to optically inspect the one or more object samples; and To process a single object sample for defect detection, the camera is controlled to move along the inspection path, and a first image of the field of view associated with a single surviving viewpoint is captured when the camera is above a single surviving viewpoint on the single object sample. The captured first image is then processed to detect any possible defects on the single object sample.

Images