Automated 360 degree dense point object inspection

Abstract

Methods for calibrating an object inspection system, methods for maintaining performance of a scanning system, and methods for aligning and fusing point clouds are disclosed. A method of calibrating an object inspection system includes scanning a calibration target to obtain a point cloud representing the target; subdividing the point cloud into one or more tiles, wherein the tiles comprise a plurality of points of the point cloud; fitting, for each of the one or more tiles, a planar model to the points included in the tile; adding the normal vectors from each of the one or more tiles and calculating an average of a set of normal vectors; defining a reference plane outside the point cloud using an average normal vector orientation, and calculating each cloud point relative to the reference plane; and using the histogram to find a local maximum and create a set of points, each point having a distance from a defined reference plane that falls within a tolerance threshold for the local maximum.

Description

[0001] This application is a divisional application of the invention patent application with the application date of May 1, 2020, application number "202080048597.0" (international application number PCT / US2020 / 030940), and invention name "Automated 360-degree dense-point object inspection".

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to U.S. Patent Application No. 16 / 401,890, filed on May 2, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0004] The present invention relates to automatic inspection of physical objects, and in particular to an inspection system and method for performing non-contact three-dimensional scanning on physical objects. Background Art

[0005] In many manufacturing processes, it is important that all parts are formed within tolerances defined by industry standards, company standards, or required specifications. For example, when making parts through molding, additive manufacturing, or machining, it is important that each manufactured part meets certain standards and specifications. The same is true for parts made using 3D printing, sintered metal, or other component forming processes. In modern production, parts, tools, die casting dies, and molds are often designed based on computer-aided design (CAD) models. For example, in computer-aided manufacturing (CAM), computer numerical control (CNC) machine tools use CAD designs as input to control the operation of the tools when machining the product.

[0006] There are several known systems and methods in the art for verifying the surface shape of manufactured parts and products. For example, coordinate measuring machines (CMMs), laser trackers, and three-dimensional scanning measurement systems, among other devices, are used to perform precision inspections. However, these inspection systems require lengthy and complex setups, and also require that the part or product to be inspected be specifically positioned in the inspection system before the inspection can begin. For example, in a CMM, the part needs to be positioned in a fixture to properly hold the part. Therefore, a significant amount of time and cost is required just to set up the product or part and the inspection system, and the setup process needs to be completed separately for each different part to be inspected. CMMs also capture measurements at specific points, resulting in a sparse sampling of the measurements of the object. Therefore, to measure the entire object, it may be necessary to measure multiple specific points. Increasing the number of points increases the complexity of the measurement, increasing time and cost.

[0007] Automated manufacturing techniques used to make components and parts, such as molding, machining, and assembly, require that the manufactured parts meet specifications. Manual inspection of each part is laborious and time-consuming, and if not completed in a timely manner, will result in the manufacture of non-conforming parts, requiring rework and wasting time and materials. Automated inspection as part of the manufacturing process can identify non-conforming parts and track the following trends, which indicate and predict that non-conformance will occur at some point in the future due to molds, tools beginning to wear, or non-conforming parts provided by suppliers. A non-contact profilometer is used to acquire a 3D image of the manufactured part to compare the acquired image with the part specifications to verify part compliance. However, relative mechanical movement between the non-contact profilometer, the part, and the supporting surface is required to generate a point cloud that defines the 3D image of the inspected part. Overlaid contour traversals are used to ensure that the part is adequately illuminated from multiple angles when generating and recording the point cloud. In addition, overlaid contour traversals allow larger parts to be inspected.

[0008] What is needed is an easy to use inspection system that can inspect parts and products regardless of orientation and that does not require the inspected parts or products to be first placed into any fixture or other mechanism to hold them during inspection. Dense sampling of objects so that the entire object can be measured in a reasonable time and cost is necessary to overcome the above-mentioned shortcomings of existing object inspection systems. Summary of the invention

[0009] The present invention relates to an online system and method for quality inspection of manufactured parts and products, including first article inspection, key functional parameters (CTF) and defect detection. The inspection determines whether the material or article is in the appropriate quantity and condition, and whether it meets the applicable or specified requirements, whether it is a specific user's requirements or an industry standard. Inspections are generally divided into three categories: 1) receiving inspection, 2) in-process inspection, and 3) final inspection. The system and method perform non-contact quality inspection by scanning the three-dimensional (3-D) surface geometry of the inspected object and comparing the information in the scanning process with the 3-D model of the object. The object can be solid, opaque, translucent or transparent, and can have mirror or diffuse surface features. The object can include a combination of these features and can still be inspected using the system and method of the present invention. In addition, the object can be inspected at several different stages of its manufacture, so that multiple in-process inspections can be performed at each stage. Each stage may require its own inspection process, as described herein.

[0010] The system and method may include an optical scanning system and a laser module for accurate online inspection of objects, the laser module including a laser profiler. In a specific implementation, the wavelength of the beam of the laser profiler may be in the violet or ultraviolet range, although other wavelengths may also be used. When objects such as parts or products are inspected, they may be placed on a transport system such as a conveyor belt at a predetermined position and in a predetermined orientation, which moves them past the optical scanning system and the laser profiler. As the objects move along the conveyor belt, they pass under the optical scanning system, which scans the objects and identifies the location and orientation of the objects on the conveyor belt. The system and method may simultaneously inspect multiple objects placed on a conveyor, wherein the conveyor moves the objects past the inspection system components during the online process. The inspected objects may be the same or different parts or products, wherein the optical scanning system can identify each object as it passes by. In addition, the optical scanning system may identify contaminants, such as dust, oil or other foreign matter present on the object, and may also detect and measure the object for the accuracy of the expected color, texture or finish of the object. A computer processor in the quality inspection system receives the determined identity of the object being inspected from the optical scanning system and loads a data file corresponding to the determined object or product being inspected. Alternatively, the object to be inspected may be predetermined and a predetermined data file corresponding to the object may be loaded. In both cases, the data file may be a CAD model which is then converted into a unified point cloud representing the object.

[0011] After the optical scanning system, a laser module including a laser profiler is provided in a scanning system for scanning the inspected object, the laser profiler having, for example, an illumination beam with a wavelength in the violet or ultraviolet range. The laser system outputs a three-dimensional (3-D) coordinate point cloud representing one or more objects at regular intervals. The system and method then compares the obtained 3-D coordinate point cloud representing the object with a unified point cloud retrieved from a computer processor. The difference between the two is determined, and the difference is used to determine the degree of change between the inspected object and the expected product (based on the stored data file). The system and method can then identify that a part or portion of the inspected object may be defective and alert a system operator when a defective object is identified.

[0012] In particular, the system can be used to inspect parts manufactured by a mold. The mold will have the desired dimensions of the final part, and it is expected that the parts produced by the mold will have these desired dimensions within specific tolerance levels. Real-time determination of defect data from injection molded parts can be used as a predictive measurement method to determine when significant wear has occurred and the mold needs to be replaced before the defects become unsatisfactory. Therefore, the system can be used as part of a geometric dimensioning and tolerance quality inspection process. Dimensional specifications define the nominal, simulated or expected geometry. Tolerance specifications define the allowable variations in shape and possible sizes for individual features, and define the allowable variations in orientation and position between features. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1A is a high-level diagram showing the components of a quality inspection system.

[0014] Figure 1B is a diagram of the components of a laser system in a quality inspection system.

[0015] Figure 2 is a graphic representation of an overview functional diagram of the inspection system modules.

[0016] Figure 3 is a diagram of the parameter setting suggestion system.

[0017] Figure 4 is a diagram of lasers 4009-4010 and sensors 4011-4012 showing ray casting and focal volumes 4001-4008.

[0018] Figure 5 A block diagram illustrating a ray casting technique for finding capture setup parameters that enable an adequate scan of an object.

[0019] Figure 6 is a simulation of an object being scanned by a dual laser camera system.

[0020] Figure 7 Depicted is a calibration target containing fiducial marker holes on the rungs.

[0021] Figure 8 An example scan is illustrated that has been segmented into planar segments.

[0022] Fig. 9 A histogram of the distances between points and a defined reference plane is illustrated.

[0023] Fig.10 A histogram of the distances between points and a poorly oriented defined reference plane is illustrated.

[0024] Fig.11An example of a planar image is illustrated in which the presence of a point is indicated by one value, and the absence of a point is indicated by another color.

[0025] Fig. 12A and Fig. 12B is an illustration of the product inspection system in use, showing the movement of the conveyor and laser module during inspection.

[0026] Fig.13 It is a block diagram of the components of a quality inspection system.

[0027] Fig.14 is a graphic representation of the scanning path of the laser system when moving only perpendicular to the transport path.

[0028] Fig.15 is a graphical representation of a scan path of a laser system when the laser system moves both perpendicular to the transport path and in the same direction as the transport path.

[0029] Fig.16 Processing details for converting a stored CAD model of an object into a point cloud for comparison with a point cloud from a laser system are described.

[0030] Fig.17 Details are described of processing image data from an optical acquisition unit to identify an inspected object.

[0031] Fig.18A A process is described for merging data from a laser system with CAD model information to determine the difference between actual object dimensions and expected object dimensions.

[0032] Fig.18B A system flow chart for generating a library based on a reference mesh model of an object is described.

[0033] Fig. 18C A flow chart of the system for registering and matching a target point cloud to a reference mesh model is described.

[0034] Fig.19 A process for determining tolerances by systems and methods is described.

[0035] Fig. 20 An example of a transport system is described that may be used in an inspection system to move an inspected object past an optical acquisition unit and a laser scanning system.

[0036] Fig.21 Describes the process of determining whether the object being inspected is defective.

[0037] Fig. 22 Example user interfaces that may be displayed to an operator are described.

[0038] Fig.23 A process is described for monitoring the performance of a scanning system, correcting for spatial shift, and notifying a user when the degree of spatial shift exceeds a threshold. DETAILED DESCRIPTION

[0039] The present invention relates to a system and method for performing high-precision quality control inspection on an object. The system is configured to perform geometric inspection on manufactured parts for quality control and inspection (including first article inspection, key functional parameters (CTF), and defect detection). As described above, the inspection determines whether the material or article is in the appropriate quantity and condition, and whether it meets the applicable or specified requirements, whether it is user-specific requirements or industry standards. Inspections are generally divided into three categories: 1) receiving inspection, 2) in-process inspection, and 3) final inspection. Receiving inspection includes first inspecting the received object to determine whether the object initially meets the applicable requirements and standards. In-process inspection is performed when the object is manufactured in several stages, and the inspection can be performed after all or specific manufacturing or preparation stages required to form the final product. Final inspection can be performed on a completed object that is not expected to be further prepared or modified. The system performs scanning on the three-dimensional (3-D) surface geometry of an object to detect defects, and the object is such as a manufactured component, part, and product.

[0040] The system performs automated, on-line or off-line quality inspection by scanning an object on a movable platform or platen using both an optical acquisition unit and a scanning system and comparing data from the resulting object scan with a predetermined data file accessible by a system processor, the data file corresponding to the object being inspected. The scanning system may include a depth measurement device whose measurements may be used to capture the geometry of the 3-D object during the scanning process. The depth measurement device may be, for example, a laser profiler system, a confocal system, a structured light system, or an interferometer configured to scan an object on a movable platform.

[0041] These different sensors or selected subsets thereof can be included in the inspection system. When more than one sensor is included, the system can be configured so that an operator or automated program can select which sensors to use, and sensors that are not used will be retracted or repositioned so that they do not interfere with the inspection. Alternatively, a sensor mount can also be included to allow one or more sensors to be interchangeably mounted to the system for use during the inspection process. Such a sensor mount allows the operator to replace imagers that are not required for a particular inspection. The system can use different sensors or sensor combinations to image different parts, object types, specific features of the same object, or different stages of preparing an object. Therefore, the system can have new setup parameters implemented for each new sensor or sensor combination used.

[0042] The system can generate an inspection report by comparing the captured dimensions of the actual object with the geometric dimensioning and tolerance (GD&T) specifications provided for the object. The dimension specifications define the nominal, simulated, or expected geometry. The tolerance specifications define the allowable variations in shape and size for individual features of the inspected object, and also define the allowable variations in orientation and position between features and the expected allowable variations.

[0043] To perform quality inspection, the system includes: a mobile transport; an optical system for imaging the object at the beginning of the transport path; and one or more depth measurement systems, such as a laser profiler, a confocal system, a structured light system, or an interferometer, for recording the 3-D object geometry located after the optical system on the transport path. The laser profiler or other depth capture modality can capture the 3-D object geometry in the form of an unstructured point cloud representing the object. The system also includes a computer processor for calculating and storing data, which includes stored object data information, such as CAD data showing the expected size of the inspected object. An operator control panel and display can be provided to allow the operator to: view CAD images in a storage database, view 3-D images of the inspected object captured by the scanning system, and view thermal maps and histogram information that details the differences between the CAD images in the storage database and the actual imaged objects. The operator control panel can also allow the operator to: modify the parameters of the inspection system, control the operation of the inspection system or pause the inspection system, and review defect information about the inspected object.

[0044] like Figure 1AAs shown, the quality inspection system 100 includes an optical acquisition unit 110 located at one end of a transporter 150 that moves an object 160 to be inspected through the inspection system. The transporter 150 can be, for example, a conveyor belt or a roller system, and moves the inspected object in a direction A along a transport path. The optical acquisition unit 110 can be an area or linear array optical system including one or more image acquisition units. The optical acquisition unit 110 can be connected to a processing device that is configured to analyze captured images of objects on the transporter 150 to identify the object and the position and orientation of the object on the transporter. The laser module 200 is included after the optical acquisition unit 110 along the transport path and is connected to the operator control panel 120. As previously described, in addition to the laser module, or as an alternative to the laser module, other depth measurement systems such as confocal systems, structured light systems, or interferometers can be used to record the geometry of the 3-D object. The laser module includes a laser driver 210 and a laser sensor 250. The laser may include a mounting assembly 260 that connects the laser to a traversal mechanism 270 such as a fixed or movable lead screw. The traversal mechanism 270 may be configured to allow the laser module to move in three-dimensional space relative to the object being inspected. The laser module emits a laser beam 140 that is reflected from the object being inspected 160 and then sensed by a laser sensor 250. The laser module 200 may be, but is not limited to, a laser profiler in which the wavelength of the illumination beam is in the violet or ultraviolet range for scanning an object and outputting a three-dimensional (3-D) unstructured point cloud. The laser profiler may measure depth using, but is not limited to, laser triangulation or time-of-flight shift sensing. The system may also include an operator control panel 120 that allows the system operator to: perform inspections on objects, modify inspection parameters, review object inspection results, and respond to defects or warning notifications. The operator control panel may include a user input 125, such as a keyboard, a mouse, or a touch screen, and may also include a display 128. The warning notification 130 may be included in the quality inspection system and may be a visual or audible warning. The user loads the appropriate CAD model and tolerance data of one or more objects to be inspected into the inspection system 100. Objects that exceed the predetermined tolerance are identified as defective. The user can view the three-dimensional (3-D) coordinate point cloud changes of the defect according to the CAD model through a heat map displayed on the user display 128, or define the actions to be taken for the defective objects. The user display can be a local display of the inspection system, or it can be a display that communicates with the computing device of the inspection station through a network connection.

[0045] Figure 1B2 is a diagram of components of a laser module 200, which includes a laser driver 210 that outputs a laser beam 140 in the form of a laser cone 230 that projects a line 240 representing a field of view across the scanned object. The length of the formed line 240 is related to the projection angle θ 280 of the laser beam 140 and the distance between the laser driver 210 and the inspected object. The laser beam 140 is reflected from the inspected object back to a laser sensor 250 on the laser module 200. In an alternative embodiment, the laser driver 210 can output a laser beam where the laser beam 140 is a point that strikes and reflects from the object.

[0046] The laser module 200 may be a laser profiler. In particular, a laser profiler having a blue-violet laser using a 405 nm wavelength may be used because of its shorter wavelength. Laser modules having an illumination beam with a wavelength in the violet or ultraviolet range are superior to standard sensors using, for example, a red laser diode on metal and organic materials. Additionally, the shorter wavelength is less susceptible to objects that emit light due to thermal properties than laser modules using longer wavelengths in the red region. The wavelength of violet or ultraviolet light minimizes the propagation of the laser light on the surface of the inspected object 160 and does not penetrate the object, thereby producing more accurate measurements, allowing for the detection and measurement of a wide range of objects made of plastics, metals, glass, and other materials, regardless of the characteristic surface finish.

[0047] Figure 2 An overview functional diagram of the inspection system modules is provided, including a point cloud data capture subsystem 2000, a part view processing subsystem 2020, and a part inspection subsystem 2030. The point cloud data capture subsystem 2000 works by acquiring system setup parameters 2002 and CAD pre-processing data 2006 from a library 2010 of objects and associated data. After acquisition, the system captures scan data 2001 and then normalizes the scan data 2003. The normalized scan data 2003 includes calibration data 2004. After the scan data is normalized, the data is aligned and stitched 2005. If more scan data is needed, more scan data can be captured 2001. After the scan data is aligned and stitched, it is generated as a 3D point cloud file 2007. All information associated with the intended object to be inspected is stored in a library 2010 of object databases. This information includes the CAD model, GD&T information associated with the object, system setup parameters for scanning the object, and other data calculated during pre-processing.

[0048] As part of preprocessing, the following operations are performed on the CAD model to prepare all necessary data for point cloud comparison when the captured point cloud is available. Preprocessing can be performed by software containing instructions executable by a computer, or can be included as part of an embedded chip or as a cloud computing resource. To begin the preprocessing operation, the user specifies the CAD file and accepts (or modifies) certain default settings if necessary. These calculations are performed only once for each CAD model, and it is not necessary to repeat the procedure for each captured point cloud. The following operations performed on the CAD model can be performed as part of preprocessing:

[0049] ● Load a stereolithography (STL) file of a CAD model. STL files convey the geometry of an object in the form of a list of 3D triangles. Each triangle specifies 3 vertices and an outward-facing normal vector.

[0050] ● Each triangle is interpolated at a high sampling rate to produce a point cloud covering the entire triangular surface. All resulting points are retained to represent objects with a dense point cloud over the entire surface, not just at the vertices of the face triangles.

[0051] ● Calculate all possible resting positions of the object, which correspond to the positions where the object might be if it were resting on a horizontal flat surface. This is done by:

[0052] ○ Find the convex hull of a CAD geometry

[0053] ○ Find the center of mass of the object

[0054] ○ For each face of the convex hull, test if the center of mass is directly above that face. This is a test to see if the object's center of mass is above the object's footing, otherwise it will tip over.

[0055] ○Leaving in the rest position, the rotation matrix corresponding to each face that passes the above test.

[0056] o As described above, all information related to the object including the convex hull, the object's centroid, and resting position determined is associated with the object and stored in the library 2010 of the object database.

[0057] ● Z-axis projection images are calculated for each rest position. These images can be visualized as a shadow directly below the part at the rest position. The intensity within the "shadow" area represents the height of the part at that position.

[0058] • Additional calculation of the Z-axis projection image for the principal component analysis (PCA) of the 2D registration as described in more detail in the following description of the invention.

[0059] ● Read in GD&T data containing specific measurements to be made. The described inspection system also provides a GUI tool to the administrator in performing pre-processing setup. The administrator sets up the GD&T information during pre-processing and then saves the settings associated with the object in the library 2010 of the object database. As part of this process, the GUI tool allows the administrator to:

[0060] ○ Manually select reference planes and measurement planes on the CAD model

[0061] ○Set measurement tolerance

[0062] Location

[0063] ■Orientation angle

[0064] ■Roughness

[0065] ■ Coverage

[0066] ○ Save all results in the object's library 2010 for future use

[0067] The administrator also selects the number and type of imagers required based on the number of surfaces to be imaged, and based on the specific measurement requirements of the object determined based on the GD&T information. For example, required in-bore measurements may require a confocal imager, measurements of small displacements and surface irregularities may require the use of an interferometer, and correct color measurements may require a calibrated vision camera. The inspection system may provide multiple selectable imaging options, where unselected imagers may be swapped or retracted. The system settings including the selected imagers are stored in a library entry associated with the object in library 2010. If it is not possible to use multiple imaging options required to obtain the desired measurements simultaneously (due to system limitations), the inspection process is divided into multiple parts, where each part specifies the imager to be used and the required measurement. The library entry for the object will then have multiple steps corresponding to each part of the inspection process.

[0068] Setting up the system includes at least setting up the following: the configuration of the imagers, sensors, and imaging angles for the inspection system; the speed of the platen or conveyor; and the object placement instructions for the specific object to be inspected. The system is initially calibrated using a dedicated calibration target, such as an object with known dimensions and properties. The dedicated calibration target can be an ideal version of the type of object expected to be inspected. During operation, the system is set up for the specific object being inspected using system setup parameters 2002 determined to be appropriate for the object. The setup parameters 2002 include: the position and orientation of the imager, such as a laser module, a confocal system, a structured light system, or an interferometer, or a combination thereof; and the placement of the object on the platen or conveyor. The initial setup can be performed when the system is offline (i.e., not used to inspect the object), but can also be performed online and refined when the system detects a new part or object. The repositioning of the imaging device can be automatic, depending on the object being inspected, so when the system recognizes a new object, the system executes a new setup procedure and the position of the imaging device is automatically updated for the new object. Multiple placements of the object can be included to cover all views required for measurement. For example, when the object to be inspected has multiple different faces, specific placement of the object and the imager for each face may be included. In addition, if the entire 3-D geometry of the object can be captured by performing a scan of the object in three different resting positions, then the setup parameters 2002 may include each of these three different positions and orientations of the object, as well as the position and orientation of the imager for each object resting position.

[0069] After the system has been set up to inspect the desired object, the imager is traversed over the object using the motion of the imager, the motion of the platen or conveyor carrying the object, or both, to record depth measurement data. That is, the object can be left in a fixed position and the imager can be moved relative to the object, thereby capturing scans as the imager moves. The data from the scanning element or imaging device is processed to produce a point cloud for each traversal, and then the point cloud for each traversal is fused in a stitching process to form an overall point cloud corresponding to the scanning element. Afterwards, the data from all scanning elements will be merged and analyzed to combine the data for a particular object obtained by each scanning element during the process. Alternatively, the object can be advanced along a movable platen conveyor so that the object passes under the imager. The imager can then be fixed or moved relative to the platen or conveyor during scanning. The data from the scanning traversal of the object is aligned using calibration data obtained by scanning a specific object. Any drift in the calibration over time is tracked, and the system can generate a notification when recalibration is required. The calibration process is described in more detail herein. The data from the aligned scan passes are merged to produce a point cloud. Multiple point clouds may be generated from different placements of the object to cover all surfaces that need to be measured, as indicated by the system setup parameters. For example, if the system setup parameters indicate that three different scans need to be performed, each with a different object and / or imager position and orientation, then these three scans are performed and three point clouds are generated.

[0070] The function of the part view processing subsystem 2020 is to align and fuse the point clouds generated during point cloud capture to create a single point cloud covering a 360-degree view of the inspected object. To this end, the system performs ground subtraction 2021 on the obtained point cloud to remove points corresponding to the ground (conveyor, platen or other platform) on which the object rests when inspected. After ground subtraction, the 3D part view file of the object is retrieved 2022 and imported into the system 2023 for comparison with the point cloud data. The point cloud data is aligned and fused 2024 with reference to the 3D part view file to obtain a 3D part file 2025 of the scanned object. The part inspection subsystem 2030 analyzes the obtained 3D part file to determine whether the inspected object has defects. To this end, the 3D part files obtained from the part view processing subsystem 2020, each representing a single point cloud of each inspected object, are imported 2033 into the part inspection module 2030, where the 3D part file is compared with the CAD model 2031 of the specific object being inspected. The comparison process involves CAD part registration 2035, which can be adjusted based on user preferences 2038. The deviations 2037 between the acquired 3D part file and the expected data based on the CAD model are used to generate an inspection report 2039, indicating to the operator whether the inspected object falls within acceptable limits, or whether it is determined to be defective 2040. The generated report 2039 can be customized based on user preferences 2038.

[0071] The part inspection subsystem 2030 begins by receiving a part CAD model 2031, which is then imported 2032 into the part inspection subsystem 2030. If no 3D part file has been imported 2033, the subsystem will identify the CTF 2034 and then provide inspection parameters 2036. The deviation between the part CAD model and the CTF will then be calculated 2037, which will generate a report 2039 indicating to the operator whether the inspected object falls within acceptable limits, or is determined to be defective 2040. The generated report 2039 can be customized according to user preferences 2038.

[0072] Scanning setting parameter suggestion system

[0073] Prior to inspecting a particular object, the system may first be set up using the parameter recommendations to ensure that the object is placed in the proper location and orientation, and that the system is set up to properly capture the necessary data for the particular object. An inspection setup typically has many parameters that need to be set before a scan can be performed. Many of these parameters may depend on the specific object being scanned. For example, the size of the object determines the distance that needs to be covered with each pass of the scan and the number of passes required. The configuration and type of imager required is based on the number of surfaces to be imaged and the specific measurement requirements. In a repositionable imaging system, the area that needs to be scanned affects the angle and position of the imager based on the geometric dimensioning and tolerancing (GD&T) measurements required. As Figure 2 The scan setup parameter suggestion module of a portion of the point cloud capture 2000 shown guides the settings for a particular object being scanned. The scan setup parameter suggestion module includes software that models the imaging settings of the system to calculate the capture setup parameters required to adequately cover the inspected object. To this end, the CAD model and GD&T measurements required for the particular object being inspected are provided to the system processor, which executes a software simulation of the inspection system to calculate the appropriate parameters for the particular object. The calculated parameters are stored in an object inspection information library, which is stored in a computer memory.

[0074] As mentioned above, some inspection applications involve in-process inspection, where manufactured objects are inspected at intermediate points during their manufacturing process. These objects may have multiple CAD models representing the expected state of the object at the intermediate points. System setup requirements are determined separately for each of these intermediate CAD models, and a separate entry is maintained in the library for each.

[0075] Figure 3 The use of a scanning setting parameter suggestion system is shown, which is implemented as a software module including a modeling simulator. Figure 3As shown, during the offline procedure 300, the module retrieves 301 the CAD model and GD&T information for a particular object. This information is used to perform a scanning setup parameter suggestion system software simulation process 302, the result of which is the determination of the setup parameters of the system for the particular object. The simulation software includes a complete specification of the system, including: the number, type and positioning of sensors and capture devices; characteristics of the sensors and capture devices, such as the focus area, supported adjustments to angles and heights, and motion characteristics of the sensors and capture devices; and transport, platen or conveyor speed and direction parameters. The determined setup parameters are saved in the system library 303, in an entry associating the setup with the object. The simulation process is performed for each of the objects expected to be scanned by the system, and the calculated setup parameters are stored in the library. This process is completed offline before the inspection system is ready for scanning and before the system performs a scan. The system can be considered offline when pre-processing steps such as calculating parameters to be stored in the library are performed. Only after the offline stage is completed, the system is set up, and deployed for use, can the scan be performed.

[0076] For a given object or inspection process, the system may have a plurality of different stored setup parameters. In use, the system may select from the stored setup parameters based on the requirements or preferences of the user, and may be based on, for example, the type of object being inspected, the type of inspection being performed. The operator may select which setup parameters to use, or the system may automatically determine which setup parameters to use based on the identification of the object to be inspected by the optical acquisition system.

[0077] After the scan setup parameters have been stored, the inspection system is ready to inspect the object corresponding to the entry in the scan setup library. If a new type of object is to be inspected, an administrator with access to the proposed system software can complete a new simulation so that the setup parameters for the new object are stored in the library. When the system is in use, or online, it will follow Figure 3 The process 310 shown. First, the CAD model and GD&T information for a particular object is retrieved 311. The system then reads the scan setup instructions for that object from the library 312 before each scan and sets up the system accordingly. After setting up the system according to the retrieved parameters and instructions, the system performs the scan according to those setup parameters and instructions 313. This ensures that the resulting scan will have the necessary data to inspect the object.

[0078] This part can be set up using an automated system such as a robotic arm according to system setup parameters, or can be done manually by an operator following provided guidelines. Guidelines can be provided by displaying graphics and charts in the GUI, and by projecting visual guidance marks on a platen or conveyor. When projection is used, the projection can be formed by using a laser module or other light source of a depth measurement system, and then used to obtain a depth measurement capture during scanning of the inspection object. The projection can be in the form of a scaled outline of the part and text and / or graphic instructions and instructions to guide the operator to correctly place and position the part. When an optical acquisition unit is included in the inspection system, the optical acquisition unit can capture an image of the object to be inspected. The system can then analyze the captured object image to determine whether the object has been placed in the correct position and orientation for inspection before it moves through the scanner during the inspection process. For example, a camera can capture an image of an object and a light-emitting outline area where the object is expected to be located. These images can be analyzed, and if the object is placed within the light-emitting area, the system can determine that the object is ready for inspection.

[0079] If, before the scanning inspection begins, analysis of the images captured by the camera system indicates that the object is not in the proper position and orientation, the scanning process can be interrupted and an alert can be issued to reposition the object. This procedure can be repeated until the object is determined to be in the proper position and orientation. In addition, if the captured images from the camera indicate that an object with a different finish or texture than expected has been placed in the system for inspection, the system can again issue an alert and allow new inspection setup parameters to be loaded before the inspection begins.

[0080] The scanning setup parameter recommendation system includes a simulator that calculates the point cloud that will be obtained for a given configuration of: laser and optical receiver; object; motion of the laser imaging module; and the platen or conveyor that carries the object. The calculation uses ray casting to determine the points on the surface of the object that are visible to the imaging module at any given location. Ray casting is the process in which the expected path of a ray from its origin source to the receiving source is projected, implemented as a computer graphics algorithm that determines the intersection between the ray and the surface. For example, in a laser system, ray casting projects the path of the laser from the laser to the receiver. Visible points are those points where the ray from the source is incident on the surface of the object and the ray reflected from the point is received at the receiver without being blocked (usually by a portion of the object itself). If there is an obstructing surface on the path, the ray casting will show that the emitted light will not be received at the receiver. Visible points are calculated over the entire traversal route to produce the point cloud that will be detected. For example, a typical laser profiler setup such as Figure 44009-4010 indicate lasers and 4011-4012 indicate optical receivers that detect reflected laser light. The laser cone covers the area 4001-4002-4003-4004 on the platen, but only the volume included in 4001-4002-4003-4004-4005-4006-4007-4008 is in focus and structures in this volume are included in the visible points. Points closer than the plane 4005-4006-4007-4008, or points beyond the plane 4001-4002-4003-4004 are not imaged and therefore are not considered visible points. The lasers 4011-4012 and the optical receivers 4011-4012 are in fixed positions relative to each other, but the entire laser-camera assembly can be moved across the platen. The depth may also be adjusted so that the planes 4001-4002-4003-4004 are on the platen surface. By performing this ray casting and calculation of the visible points, the simulator can then determine the setup parameters.

[0081] Using this ray casting technique, the simulator is used to find the capture setup parameters that will produce an adequate scan of the object, e.g. Figure 5 As shown. Figure 5 As shown, input data 501 including the CAD model, lay-up location, GD&T information, and imagers required for the desired object to be simulated is loaded into the program from the object library 2010. If the object has a multi-part inspection process, each such part is simulated separately. The system simulates the selected imager as an emitter and receiver, which can be modeled as points or lines based on the type of imager.

[0082] The program estimates initial capture parameters 502 based on the size and extent of the CAD model. The range and step size for each capture parameter is also determined based on the system settings. For example, the right laser can be set to a range of -25 to +25 degrees around the Y axis with a step size of 5 degrees. The simulator program uses ray casting to calculate a simulated scan of the object 503. During the simulated traversal of the laser, at specified time intervals, the intersection of the laser line with the object is calculated by ray casting. If there is no intersection, then the object is not in the path of the laser at that location. If there is an intersection, then the first intersection with the object is on the surface where the laser is incident. A ray is constructed from the intersection of the laser with the object to the receiver, and the number of intersections of this new line with the object is calculated. If there are zero intersections, it means that the camera can see the intersection and this point is added to the detected point cloud. If there is one or more intersections, it means that although the laser is incident on the surface of the object, the camera cannot see the surface because it is blocked by other parts of the object (i.e., it is in a shadow cast by the object structure itself). These calculations are repeated after the interval of moving the laser camera setting by one step. Smaller intervals will produce denser estimated point clouds, but will also be slower due to the computational cost of calculating more intersections.

[0083] The simulator evaluates the estimated point cloud and calculates a score 504 using a weighting function 505 for the CAD model surface, which indicates the importance of that portion of the object. In one embodiment, the weighting function 505 is calculated based on the GD&T information, as described below. The CTF region of the object is obtained from the GD&T information of the object, and the CTF region is a region that has special relevance to whether the inspected object is acceptable. The weighting function at the cell level is defined as a set of weights between 0 and 1 for each CAD model cell. For example, a cell containing a feature that is critical to function may have a weight of 1.0, while a region of the object with a greater tolerance for error may have a weight of 0.1. A region of the object that does not need to be measured may have a weight of 0.0. If a cell does not have a specific CTF region, a default weight of 0.5 may be assigned. When it is desired to have finer control over the weighting function in the cell containing the CTF region, the cells in the original CAD model may also be resampled to a finer resolution (creating more cells). The sampling resolution of the surface mesh (i.e. the area covered by each element) can also be adjusted to be finer (using more elements) based on the tolerance specified in the GD&T information.

[0084] The estimated point cloud is scored based on the density of the points, a weighting function, and any other factors deemed important to downstream processing steps, such as the overlap of coverage between the left and right imagers. The density of points around each point can be calculated as the number of points within an area given a radius from the point. Although the most accurate, this method of determining point density can be computationally slow given that there are typically hundreds of thousands of points in a point cloud. Although less accurate, a faster determination of approximate density can be obtained by calculating the number of points within each cell of the CAD model and dividing by the area of ​​the cell. CAD models are often described in a format where the surface of a 3D object is modeled as a collection of cells, and these cells are typically triangular. If the identity of the cell where the laser beam and the object surface intersect occurs is maintained during the calculation of the intersection using ray casting, then this approximate density can be calculated with very little additional calculations.

[0085] The point density calculated for each cell of the CAD model is multiplied by the weight for that cell from the weighting function to obtain a cell-level score. The normalized (scaled between 0 and 1) weighted sum of the point densities of all cells is used as the point cloud score for comparison between point clouds. A determination is then made to determine whether the score is sufficient 506. If the score exceeds a predetermined threshold, it is considered sufficient. For finer discrimination, the score of each cell may need to meet a cell-level threshold; or for a coarser score, a single threshold may be specified for the entire point cloud. Additional factors may also be used to determine whether the score is sufficient. For example, if a downstream processing step such as alignment of views requires an overlap area between two views, an overlap score may be calculated based on the number of points in each triangular cell of the CAD model of the object in view A and view B. The overlap score for the cell is then calculated as (1-the difference in counts between the two views / the larger of the two counts), and the overall overlap score is calculated as the normalized weighted sum of the cell-level overlaps. A threshold may be set for the minimum overlap required, and point clouds whose overlap scores do not meet the threshold are considered to have insufficient scores.

[0086] If the simulation parameters produce a score that passes the adequacy score check 506, then the system parameters and corresponding rest positions that achieved that score in the simulation are output 515 to be stored as the recommended system setup parameters for that particular object in the library 2010. The scanning system can then access the parameters prior to inspecting a physical sample of the modeled object to ensure that the captured scans can be used to adequately inspect the object.

[0087] When the score is insufficient, determine whether there are any parameter combinations remaining to be tested 507. When there are additional parameter combinations remaining 507, update the capture parameters 508 and select the next available parameter combination. The parameter combination is generated by incrementing the capture parameters to step one step (equal to its step size) within its range. The capture parameters include the start and end points of the imager scanning path, the start and end positions of the platen or conveyor, the height and angle of the imaging system, the resting position of the object, and the number of traversals. Parameters can be incremented one at a time, or some parameters can be grouped to be incremented / decremented simultaneously. For example, if the angle of the imager is incremented along the direction of the object's motion, the starting position of the object can move in coordination. A new simulation 503 is started using the updated capture parameters 508. The process is repeated using the new parameter combination until a sufficient score is obtained.

[0088] If no scan with a sufficient score is obtained using the available parameter combinations, and the determination 507 indicates that no parameter combinations remain for testing, then the parameter settings for the best scoring scan are saved 509 along with the shelf positions used. The cell-level weighting function is recalculated 510 so that cells that have met the threshold in the previously stored best scan are reallocated to 0.0 and cells that have not met the threshold are reallocated to 1.0. A different shelf position is selected for simulation, and the process is repeated using the new cell-level weighting function. It is determined whether additional shelf positions remain for consideration 511. At each available shelf position, as determined in 511, the weights of additional cells that meet the threshold are reallocated to 0.0 before selecting the next shelf position 513 until a sufficient score is achieved. The overall scan parameter recommendation includes capture parameters for each shelf position used to achieve a final sufficient score.

[0089] Multiple captures 513 may be required at different shelf positions to achieve an adequate simulated scan. If the score is not adequate after all shelf positions have been exhausted, a determination 511 indicates that there are no more shelf positions to consider, the saved best scoring scan 512 is output, and a notification is sent to the administrator that the simulation failed to find a recommended parameter set. The administrator may then choose to change the number and type of imagers to address this problem.

[0090] For example, Figure 6 A dual laser profiler system modeled by a simulator is shown. In this case, the laser transmitters are modeled as lines shown in dark blue, and the receivers are also modeled as lines shown in light blue. As shown, the simulation is set up to include a right laser 6001, a right receiver 6002, a left laser 6003, and a left receiver 6004. The object 6000 used for the simulated inspection is rendered based on the CAD model. Figure 6As shown, the light emitted from lasers 6001 and 6003 will be reflected to receivers 6002, 6004. The focal area 6005 can be imaged by the system, while the area falling outside the focal area will not be imaged. In the simulated scan, a simulated traversal of the object with two laser profilers is modeled, where each traversal includes multiple steps. The intersection of the laser and the object is calculated at each step of the traversal, and it is determined whether the receivers 6002, 6004 can obtain a measurement for each point where the laser is incident on the surface of the object. If the number of intersections of the ray casting from the receiver to the laser intersection with the object is zero, then the point is visible. That is, the path of the light emitted from the laser to the receiver is not blocked by another part of the object, and the reflection angle of the light allows the light to be received at the receiver. These points are added to the estimated point cloud.

[0091] During the in-line inspection process, the operator retrieves the system configuration and setup parameters from the library. An optical acquisition unit such as a vision camera system can be used to identify the object on the transport or platen, verify the imager settings and the location and orientation of the object to verify that the settings match the recommended settings for the object. In the event that a mismatch is detected, a notification can be sent to the operator.

[0092] When scan settings suggestions are not adequate or further fine-tuning is required, administrators can use the system's visualization tools to achieve the desired results. The simulator can generate a density map visualization of the calculated point cloud, indicating the density of points on the object surface. A visualization of the point cloud showing the points visible to each imager and the overlap between any two imagers can also be presented to the user. Another visualization can show the points visible in one rest position, in a second rest position, and in the overlap between two point clouds. The point cloud can be displayed as an overlay of the CAD model to visualize the coverage of the object surface area. These visualizations can be run with different parameters, and side-by-side or overlay views that compare the point cloud of one set of parameter settings with the point cloud of another set of parameter settings can be used to fine-tune the capture setting parameters.

[0093] System calibration before, during or after object inspection

[0094] In the above-mentioned object inspection system, one or more laser profilers can be mounted on a mechanical stage such as a rack and pinion assembly so that the profiler can sweep the inspected object. The object is placed on a support surface, which can be a fixed platform, a mechanical stage that can control up to six degrees of movement, or a conveyor belt. As an alternative, one or more profilers can be mounted to a mechanical arm that can inspect an array or matrix of objects composed of multiple manufactured parts or components. However, mechanical components such as rack and pinion mechanical components used to move the object support surface or one or more laser profilers drift over time due to wear, repeated reaching of the end of travel and / or change of direction. These types of mechanical hysteresis can be compensated by recalibration of the system. The sensitivity of the rotational component of the transformation to the mechanical hysteresis of the system is lower than that of the translational component. The translational component can be averaged, but the rotational component cannot be averaged.

[0095] The calibration of the scanning system depends on the detection and accurate positioning of the reference elements on the calibration target. The calibration target can include multiple surfaces at different depth levels from the laser profiler, where the multiple surfaces include the reference elements at predetermined positions. For example, Figure 7 As shown, the calibration target may include a plurality of stepped surfaces 701, wherein the surfaces include reference elements in the form of marked holes 702 at predetermined locations. Other types of reference marks may also be used, such as raised surfaces, lines, grooves, or gaps. The reference element marked holes 702 may be a single hole, or may be a group of holes arranged in different patterns. After the inspection system scans the calibration target, a point cloud representing the calibration target is generated, such as Figure 8 As shown in FIG. 8 . In the obtained point cloud, the reference element hole 802 and the surface 801 of the calibration target are identifiable and their positions can be determined. Detecting and locating the reference element hole is implemented as a calibration process. First, as Figure 8 As shown, the obtained point cloud of the calibration target is segmented into separate planes, where each plane represents one of the surfaces at different depths and the bottom of the hole in the surface. Figure 7 In the calibration target shown, each step surface and the bottom of the hole in that step surface will form planes. The calibration process then investigates these planes to detect and locate the reference element holes. After the reference holes are located in the point cloud, the hole positions determined in the points can be compared to the known dimensions of the calibration target, and the inspection system can be calibrated so that the measurements in the obtained point cloud match the known dimensions of the calibration target. The calibration process determines an affine transformation matrix (of size 4x4) that is used to remove rotations and shear caused by scanning with the laser module at a non-orthogonal angle to the direction of motion of the platen or conveyor carrying the object.

[0096] The holes or other fiducial elements in the calibration target are distributed in a pattern that allows the holes to be identified based on the number of holes in the cluster and the alignment with other clusters, in which clusters of holes are located at different known locations on the calibration target. The number of holes in the pattern also makes it unlikely that all hole centers will be aligned with their expected locations unless the transformation matrix is ​​sufficiently accurate. For example, Figure 7 The calibration target shown includes a plurality of surfaces 701 and a cluster of a plurality of spaced-apart holes 701 arranged in a pattern. The calibration process uses a random sample consensus (RANSAC) algorithm to find a transformation matrix that produces the best fit between the hole locations in the acquired point cloud and the known hole center locations of the calibration target. After the calibration target is scanned and a point cloud is acquired, a calibration process is performed on the acquired point cloud, which generally includes:

[0097] 1. Selecting random groups of 4 hole centers in the point cloud, and;

[0098] a. Calculate the transformation matrix required to align the 4 positions with their expected positions;

[0099] b. Apply the transformation matrix to all other points and count the number of points that fall within a certain threshold of their expected position (inliers);

[0100] 2. Repeat step 1 for N iterations (eg, N=1000); and

[0101] 3. Determine the final transformation matrix as the matrix with the best fit, as indicated by the greatest number of inliers.

[0102] The calibration process requires two parts. In the first part, the calibration target must be scanned to obtain a point cloud, and the point cloud must be separated into planes representing each surface of the object. In the second part, the fiducial elements in each surface must be detected and their positions determined so that the transformation matrix can then be found. There are two alternative methods to implement the fiducial element detection and positioning process of the first part.

[0103] In the first method, the obtained point cloud of the calibration target is separated into planes for each different depth surface in a process using the following steps:

[0104] Method 1, part 1

[0105] 1. Scan the calibration target to obtain a point cloud representing the target.

[0106] 2. Subdivide the point cloud into multiple tiles, which are usually non-overlapping.

[0107] 3. For each tile, try to fit a plane model to these points. The parameters of the plane model include four values: the three-dimensional unit vector of the surface perpendicular to the plane, and the distance of the plane from the origin of the coordinate system.

[0108] 4. Add up the normal vectors from each tile where the plane can be successfully fit into the set.

[0109] 5. Calculate the average of the set of normal vectors.

[0110] 6. For each element of the set of normal vectors, compare it to the mean and remove any vectors that are outliers (or not similar enough to the mean) from the set. The comparison to the mean is done as the dot product of the mean normal vector and the normal vector under consideration.

[0111] 7. If the membership of the set has changed, return to step 5.

[0112] 8. Use the resulting average normal vector for orientation, defining a reference plane outside the point cloud.

[0113] 9. Calculate the distance between each cloud point and the defined reference plane.

[0114] 10. Create a histogram of the distances from the cloud points to the defined reference plane. Since the defined reference plane should be parallel to the surface, such as the steps of the calibration target, the histogram is expected to show a spike at the distance from the reference plane to each step. Fig. 9 An example of such a histogram is illustrated, where a peak 9002 is included at the distance of each step from the reference plane 9000 to the calibration target, as represented in the point cloud. Population 9001 represents the number of points at a given depth.

[0115] 11. Analyze the histogram to find local maxima. Each of these maxima corresponds to the distance of one of the steps to a defined reference plane.

[0116] 12. For each local maximum in the histogram, create groups of points, each of which has a distance from the defined reference plane that falls within the tolerance threshold of the local maximum. Each of these groups is a plane segment of the original cloud, corresponding to a detected plane (step surface or bottom of a hole).

[0117] This first method of determining plane segments may also include the following to provide refinement of the plane segments.

[0118] 13. For each plane segment, fit the plane model; specifically, calculate the normals of the segment points.

[0119] 14. Compute the average of the normals weighted by the population of points in each plane segment. (Since the target should have parallel planes, these should all be the same).

[0120] 15. Apply the average normal vector to each plane segment and recalculate the distance of each plane from the coordinate origin.

[0121] In the second method, the obtained point cloud of the calibration target is separated into planes for each different depth surface in a process using the following steps:

[0122] Method 2, part 1

[0123] 1. Scan the calibration target to obtain a point cloud.

[0124] 2. Using a priori knowledge of the rough orientation of the calibration target, define a reference plane outside (below) the point cloud.

[0125] 3. Measure the distance from each point to the defined reference plane.

[0126] 4. Create a histogram of the distances from the points to the defined reference plane. If the reference plane is indeed parallel to the plane representing the surface of the object in the point cloud, the histogram will show a clear peak 9002 at a certain depth, as shown in Fig. 9 However, if the reference plane is not parallel to the plane, Fig.10 The illustrated histogram has fewer peaks 10002 at various distances from the reference plane 10000. The population 10001 represents the number of points at a given depth.

[0127] 5. Measure the entropy of the histogram. The sharper the histogram, the lower its entropy, and the smoother the histogram, the higher its entropy.

[0128] 6. Follow the gradient descent method to refine the orientation of the defined reference plane.

[0129] a. Orientation of the reference plane for perturbation definition

[0130] b. Create a histogram of the distances of each point from the perturbed defined reference plane.

[0131] c. Calculate the entropy of the resulting histogram.

[0132] d. Choose the perturbation with the lowest entropy.

[0133] e. Reduce the perturbation step size and repeat until the termination condition is met. Different termination conditions can be used, including a maximum number of iterations, an entropy threshold, a statistical evaluation of the distance around each spike, or other analytical measures. The final histogram is then analyzed.

[0134] 7. For each local maximum in the histogram, create a group of points, each of which has a distance from the defined reference plane that falls within a tolerance threshold from the local maximum. Each of these groups represents a plane segment of the original cloud, corresponding to a detected plane (step surface or bottom of a hole).

[0135] After the plane is detected in part 1 using either method 1 or method 2, the calibration process then analyzes the detected plane to determine the position of the reference element in the second part of the process.

[0136] Method 1 and 2 - Part 2

[0137] Each detected plane is analyzed to detect reference holes. Figure 8 , the top plane is cyan, and the bottoms of holes on that plane are magenta. The calibration process can search for these holes on the surface plane rather than by finding the bottoms of the holes, which may be incomplete due to shadow-like occlusions caused by the oblique angle of detection. Points on the top surface plane are less subject to such occlusions. In order to detect every hole represented by a missing point, the area around the hole must have a sufficiently dense population of points to define the hole boundary.

[0138] To more conveniently achieve detection of holes in each plane, the planar segments of the point cloud representing the area around the hole are transformed into a two-dimensional image. This two-dimensional image is processed to detect and locate the holes in the image. The location of each detected hole is then transformed back into the original three-dimensional space of the original point cloud, obtained by recording depth information using a recording device (such as a laser profiler) that scans the image by moving relative to a calibration target. The steps of this process are outlined in more detail below, with the moving parts (including the scanner, camera, and platen or conveyor) represented as the x and y directions, and the depth represented as the z direction.

[0139] 1. Analyze the point cloud to determine the median (or average) space between points in the x and y directions.

[0140] 2. Obtain the set of points that constitute the piecewise plane and the associated plane model parameters (from part 1).

[0141] 3. Create a bionic transformation matrix, which will be used later to map the positions of points on the plane to the pixel positions of the two-dimensional image.

[0142] 4. Calculate the rotation parameters of the biomimetic transformation needed to rotate the point along the x-axis or y-axis so that the normal vector of the rotation plane is parallel to the z-axis. Record these values ​​in the biomimetic transformation matrix.

[0143] 5. Determine the values ​​needed to scale the points in the x and y directions so that the spacing between points is approximately one pixel.

[0144] 6. Use the bionic transformation matrix to rotate the points so that the resulting plane is parallel to the xy plane.

[0145] 7. Analyze the range of x, y, and z values ​​of the resulting point cloud. The minimum values ​​of x and y and the average (or median) value of z are used to create translation values ​​in the biomimetic transformation matrix. With increasing translation values, the biomimetic transformation matrix completely describes the mapping of points on the plane in three-dimensional space to pixels in the two-dimensional image. The range of x and y values ​​represents the dimensions of the desired image size.

[0146] 8. Create an image of a plane in which the presence of a point is represented by one value (such as white) and the absence of a point is represented by another color (such as black). Fig.11 An example of such an image is illustrated, with dots (corresponding to portions of a planar surface) shown as white 1101, and areas without dots (corresponding to fiducial element holes in the surface) shown as black 1102. When creating the image, the position range of the dots in the x and y dimensions corresponds to the pixel positions.

[0147] 9. Optionally, the image can be processed to reduce noise.

[0148] 10. Detect and locate holes using prior knowledge of the expected appearance of holes and image processing calculations. A variety of image processing techniques can be used to detect and locate holes, such as template matching by correlation convolution, Haar object detection, convolutional neural networks, and Hough transforms of edge pixels. Due to the binary nature of images and anisotropic scaling of images, the preferred embodiment uses convolutional template matching using the entire image of the plane.

[0149] 11. Invert the bionic transformation matrix to produce a new matrix (the inverse matrix) that maps pixel locations in the image to points in three-dimensional space.

[0150] 12. Use this matrix to transform the positions of the detected holes to their corresponding positions in three-dimensional space.

[0151] 13. The detected hole centers are used to calculate a transformation matrix that determines the orientation of the calibration target in 3D space. The transformation matrix is ​​4 by 4 and is used to transform homogenous coordinates.

[0152] Therefore, it has only 12 variables because the last row is fixed to (0, 0, 0, 1). Solving for these 12 values ​​will require four points in three dimensions. Among the different algorithms that can be used to calculate the transformation matrix, the preferred embodiment uses the RANSAC algorithm.

[0153] After the transformation matrix is ​​determined, it is applied to subsequent scan data. The application of the transformation matrix can eliminate skew or shear caused by the relative motion between the object and the scanner combined with the orientation of the scanner.

[0154] Object Inspection

[0155] Fig. 12A and Fig. 12B A system configuration for performing product quality inspection on one or more objects is illustrated. Fig. 12A As shown, according to one embodiment, the inspection system may include a transporter 150 that moves the inspected object 160 along a direction A. The laser module 200 may be included on a guide rail that allows it to move perpendicular to the transport direction, for example Fig. 12A The laser module 200 moves in directions B and C as shown. As the laser module moves perpendicular to the transport direction, the laser field of view 240 passes across the inspected object multiple times. Recall that the laser beam 140 forms a line on the conveyor 150 corresponding to the field of view 240, which is parallel to the direction of the conveyor 150, as shown in direction A. At each pass, the field of view of the laser module scans the object multiple times. Overlapping scans of the object (i.e., raster scans) can be used, which can reduce the noise inherent in the translation of both the object 160 and the laser module 200 on the conveyor 150 and improve the sampling resolution. Having multiple data per point on the point cloud allows the overall scanning error to be reduced by minimizing the data variance.

[0156] exist Fig. 12A In the configuration shown, the laser module 200 can be rotated 90 degrees so that the laser field of view 240 is parallel to directions B and C. In this case, the laser module 200 captures a scan of the object equal to the width of the laser field of view 240. The laser module 200 continues to move across the conveyor 150 in directions B or C. When the laser module 200 has traveled a distance equal to the laser field of view 240, the laser module 200 captures another scan. Fig. 12B In the alternative configuration shown, the laser module 200 can be positioned on rails that allow it to move parallel to the transport 150. The laser module can be moved back and forth over the inspected object 160, with the reflected laser cone repeatedly passing over the object. Fig. 12B In the example, the orientation of the laser module 200 is from Fig. 12AThe laser module 200 is rotated 90° so that the laser field of view 240 is perpendicular to the direction of the conveyor 150. The laser module 200 moves in direction D to scan the predetermined area. When the laser module 200 reaches the end of the predetermined area, the laser module 200 travels a distance equal to the width of the laser field of view 240 in direction C and then begins another scan in direction E. As the laser module 200 moves in direction C, the laser module continues to scan in directions D and E. After the laser module 200 scans the width of the conveyor, the laser module repeats the scanning process in directions D and E when it moves in direction B.

[0157] During the inspection process, a known object is placed on the transport 150 at a predetermined position and orientation. Alternatively, the object moves along the transport and is first imaged by the optical acquisition unit 110. As described above, the object image captured from the optical acquisition unit 110 is used to identify the object and determine its orientation. The predetermined or determined coordinate position and orientation of the object 160 on the transport 150 can be sent to a processing unit connected to the laser scanning profiler module. The processing unit can control the laser profiler to scan only the area of ​​the transport where the object is located. Alternatively, the profiler can be configured to scan the entire width of the transport, rather than focusing on a specific area to determine the object to be located. When the profiler scans the entire transport, the point cloud generated by the scan is cropped for the area in the transport where the object is present. When scanning the entire width of the transport, multiple objects can be present in the same profiler scan, which can then be separated during subsequent processing. In addition, the laser profiler module can locate the part and directly extract the part from the background.

[0158] After the inspected object is identified based on the image captured from the optical acquisition unit, the system processing unit loads the CAD data corresponding to the identified object from the CAD model database and converts it into a unified point cloud. As described above, a predetermined object can also be loaded, so it is not necessary to first capture an image from the optical acquisition unit. In this case, only the CAD model of the predetermined object needs to be loaded from the database. The point cloud from the laser profiler will be superimposed and registered with the CAD point cloud. The deviation between the CAD point cloud and the point cloud from the laser profiler is calculated to create a heat map of the difference. By analyzing the CAD model of the object, the point cloud from the laser profiler, and the 3-D registration heat map of the difference with the original CAD model, the operator can use the 3-D registration heat map to observe any changes in the object from the CAD model. These visualizations of object changes are recorded over time together with the number of objects, mold cavity locations, material composition, and other factors related to the monitored manufacturing process. The visualization is presented in a selectable time order, where the object illustration contains the 3-D registration heat map. Each individual image of the object is also registered so that the operator can observe the performance of the degradation condition represented by the heat map over a selected period of time. This form of dynamic visualization can be used to prevent the manufacture of defective objects, such as parts or components, by providing illustrations of wear patterns for diagnostic purposes and by projecting monitored degradation conditions to determine in advance when production of a given object will exceed tolerances. To further distinguish minor changes in the 3-D registered heat map, the visualization of these changes can be enhanced by using different colors, textures, and or patterns. Computed and synthesized heat map projections of monitored degradation conditions can also be provided for diagnostic purposes.

[0159] In addition, a histogram of the errors can be displayed to the operator. The system can be configured to identify parts that exceed a predetermined tolerance and mark them as defective. The user can configure the system to generate a notification, stop production, or use a part picker or similar device to sort the defective parts from other parts.

[0160] Fig.13 is a block diagram showing a schematic diagram of the components of a quality inspection system. Fig.13As shown, the inspection system 100 includes an optical acquisition unit 110. The optical acquisition unit includes an optical imager 400 to capture an image of the inspected object, and an image A / D converter 402 converts the analog image signal output from the optical imager 400 and generates digital image data, which is then output to a pixel corrector 404. As described above, the image device 400 has an image capture element using a CMOS (complementary metal oxide semiconductor), which is arranged in a row in a main scanning direction perpendicular to the conveyor 150, as shown by arrow A. As described above, the image capture device may be a CIS (contact image sensor). In addition, an image capture sensor using a CCD (charge coupled device) may be used instead of a CIS. The optical acquisition unit 110 may also include a stereo imager. The pixel corrector 404 corrects any pixel or magnification abnormalities. The pixel corrector 404 outputs the digital image data to the image controller 440 within the system processing unit 170. As Fig.13 As shown, the system also includes a first laser module 200a and an optional second laser module 200b. Although two laser modules are shown in the figure, the system may include any number of laser modules. With more laser modules, the system is able to capture the outline of a larger object, or capture more outlines of a complex object during the inspection process. The use of multiple laser modules also minimizes occlusions or shadows. Occlusion refers to the obstruction of the laser beam 140 from the laser driver 210a, 210b to the object and then back to the laser sensor 250a, 250b. For example, by knowing the object orientation determined by the optical acquisition unit 110 or by using a predetermined object placed in a known orientation, the laser modules 200a, 200b installed in the quality inspection system 100 can be manually or dynamically changed to optimize the viewing angle, field of view or working distance, thereby minimizing occlusions or shadows to control the position and movement of the laser modules 200a, 200b to obtain the best scan. Each laser module 200a, 200b includes: a laser driver 210a, 210b, a laser sensor 250a, 250b, and a laser processor 405a, 405b. The laser processor 405a, 405b can be a microprocessor or processing circuit, or can include a communication module to receive processing commands from an external processing device. The laser processor packages and sends an unstructured point cloud representing the measurement data to the image controller 440, which stores the data in the laser coordinate memory 435.

[0161] The optical acquisition unit 110 sends the digital image data to the image controller 440. The image processing unit 485 then processes the digital image to identify the scanned object. Alternatively, the object may be a predetermined object and may not necessarily be identified by the optical acquisition unit. Based on the features of the identified object 160 to be inspected, predetermined setting parameters or recipes for the laser modules 200a, 200b may be extracted from the system memory 455 to optimize the configuration of the laser modules 200a, 200b. The object features associated with the predetermined setting parameters in the system memory 455 may be, for example, reflectivity, color, geometry, or surface roughness. The predetermined setting parameters or recipes may be the intensity, profile per second, or exposure of the laser drivers 210a, 210b.

[0162] The operator configures the inspection system 100 to perform the required inspection through the operator control panel 120 or the network interface 445. The operator control panel 120 includes an operator input 125, which can be, for example, a mouse, a keyboard, a touch screen or a touch pad. An operator display 128 and an alert notification 130 can also be included in the operator control panel. The alert notification 130 can be, for example, a red, yellow, and green light stack, an audio alert, or other visual alert mechanism. The network interface 445 allows the system to communicate with an external processing unit and a network device. The network interface can be connected to a local area network, a wide area network, or to the Internet via a wired or wireless connection. The network interface can be any form known to those skilled in the art, including but not limited to Ethernet, USB, Wi-Fi, cellular or other data network interface circuits. The network interface 445 also provides a device for remotely controlling the inspection system 100 by supplying various types of information required for the inspection. The network controller 490 manages the network interface 445 and guides the network communication to the machine controller 430.

[0163] The system memory 455 can also be connected to the system processing unit 170, and other processing components in the inspection system. The system processing unit 170 and other processing units can be, for example, one or more microprocessors, one or more coprocessors, one or more multi-core processors, one or more controllers, processing circuits, ASICs (Application Specific Integrated Circuits) or FPGAs (Field Programmable Gate Arrays), or combinations thereof. The processing unit can be configured to execute instructions stored in the system memory 455 or accessible to the processing circuit, such as a communication network through a network controller 490. These instructions can enable the inspection system to perform one or more functions described herein when executed by the processing circuit. The memory 455 can include volatile and / or non-volatile memory, and can be a hard disk, random access memory, read-only memory, cache memory, flash memory, optical disk, or a circuit configured to store information. The system memory 455 can be any non-temporary computer-readable storage medium that communicates with the system processing unit. The memory can be configured to store information, data, applications, or program instructions that allow the system processing unit 170 and other processing units to control the inspection system and perform the inspection process.

[0164] The system processing unit 170 includes an image controller 440 configured to receive and analyze images from the optical acquisition unit 110. The images received from the optical acquisition unit 110 are stored in an image buffer memory 475 to save the processed images. An image processing unit 485 is connected to the system processing unit 170. The image processing unit 485 is configured to receive the captured images from the optical acquisition unit in digital form and after pixel correction, and further process the images. These captured images can be stored in an image buffer memory 475, which is connected to the image processing unit 485, the system processing unit 440, or both. The image buffer memory stores the most recently captured images sent by the optical acquisition unit during the inspection process. As part of the processing, the image processing unit 485 can interact with the image controller 440 of the system processing unit 170. The system also includes a CAD model database 425, which stores CAD data information of the objects inspected during the quality control inspection process. The CAD model database 425 may include predetermined CAD configuration files for expected inspection objects, and may also be an updateable database, allowing a user to manually or automatically store additional CAD configuration files for new inspection objects in the database. For example, the CAD model database 425 may be updated by performing the following process: uploading a CAD model stored on a local storage; or uploading a CAD model through a network communication interface.

[0165] The image controller 440 is also configured to receive and analyze unstructured point cloud measurement data from the laser module units 200a and 200b. The unstructured point cloud data received from the laser module units 200a, 200b is stored in the laser coordinate memory 435 to be retained for further processing by the image processing unit 485.

[0166] The machine controller 430 is configured to control the operation of the optical acquisition unit 110 and / or the laser module unit 200a, 200b. For example, the machine controller can control the time of image capture, and / or use the image capture data to control the movement of the conveyor 150, including the transport speed, and the movement and scanning of the laser module unit. The system processing unit 170 may include a timer unit 480, as well as a part position detector 450 and a defect detector 460. The part position detector can be configured to determine the position and orientation of the object being inspected, including the position on the conveyor, and the orientation relative to the transport direction or another fixed coordinate in the system. The defect detector 460 is configured to determine whether the product is within a specific tolerance level based on a comparison between the laser module scan of the inspected object and a stored data file about the expected size of the object. If the product is not within the tolerance level, the defect detector can issue a defect warning. The defect detector can also be configured to determine defects based on images captured from the optical acquisition unit 110.

[0167] The part rejection unit 470 can communicate with the defect detector 460 in the system processing unit 170. The part rejection unit 470 can receive information from the defect detector 460 and issue a command to stop the inspection process and mark the inspected object as defective based on the determination of whether the defect meets the predetermined quality threshold requirement. The part rejection unit 470 and the timer unit 480 will track the defective object and take action to remove the defective object from the transporter 150 or alert the operator of the defect condition through, for example, the alert notification 130 or the network interface 445.

[0168] The inspection system also includes a controller for controlling the transporter 150 and the laser modules 200a, 200b during the inspection process. The transport motor controller 465 can control the speed and time of the transport mechanism that moves the object along the inspection transport path. For example, the transport motor controller 465 can be connected to the motor that drives the conveyor, platen or roller that forms the transporter 150, and can drive the motor at different speeds to control the speed of the inspected object. When a defective object is detected, the transport motor controller can also stop the transporter 150, allowing the operator to have time to remove the object from the inspection line or take other appropriate actions. Two laser motor controllers can be included, wherein the laser XY motor controller 410 controls the movement of the laser modules 200a, 200b in the XY plane, and the laser Z motor controller 420 controls the movement of the laser module unit in the Z direction (or height). For example, the laser XY motor controller 410 can move the laser module unit 200a, 200b in the X direction, so that the laser traverses the width of the transporter 150, so that the laser field of view 240 scans multiple times across the inspected object. The laser Z motor controller 420 can move the laser module up and down so that the system can inspect objects at different heights. This allows the distance between the laser drivers 210a, 210b and the inspected object 160 to be maintained.

[0169] Fig.14 The movement of the laser module 200 during a scanning process is illustrated, wherein the object moves in direction A, and the laser module moves only in a direction perpendicular to direction A. The laser module 200 is rotated 90 degrees so that the laser field of view 240 is parallel to direction A. As can be seen from the figure, the laser field of view 240 scans a path across the conveyor 150 as the laser module moves in directions B and C perpendicular to direction A. As part of the raster scanning process, the laser module moves alternately in directions B and C, repeatedly traversing the object as it scans. The laser module thus produces an overlay scan pattern with a slope relative to the inspected object 160, where the slope is imparted because the inspected object continues to move in direction A as it scans. This scanning method produces an overlay region between successive traversals of the object. The gray area 1400 depicts the overlay region between traversals C1 and B1. Similar overlay regions are created between each pair of successive traversals. Overlay scans can be performed by varying the speed of the conveyor 150 or the laser module 200. In order to fully cover the scanned portion, the following relationship must hold:

[0170]

[0171] Where Y is the length of the laser field of view 240, and W is the distance that the laser module 200 scans across the conveyor 150, as Fig.14As shown, V_B and V_L are the velocities of the transport 150 and the laser module 200, respectively. When this relationship holds, the laser field of view 240 will overlap with the object being inspected, allowing multiple point cloud data points to be captured at the same spatial location. Additionally, this allows the system to be calibrated for position discontinuities, such as those caused when the transport is not parallel to the laser module.

[0172] You can also consider the angular relationship between the laser field of view and the transport. If the angular relationship is not zero, that is, they are not parallel, one end of the laser field of view will be closer to the transport than the other end. Fig. 12A If the laser module moves in the direction A shown, any point on the object surface will appear to increase or decrease in Z direction or height based on the angular relationship of the laser field of view relative to the transport. Fig.14 As the object travels through the laser field of view angular relationship in direction A, each laser pass across the object will create a step in the scan data along the Z direction. This angular relationship can be compensated for by performing a calibration. In this calibration, the transport is scanned without any object present. It is known that any height difference between the two end points of the laser field of view is due to the angular relationship. The angular relationship can be stored in the system memory, allowing the system processing unit to subtract any steps in the scan data.

[0173] Likewise, the angular relationship direction A of the object must be zero degrees relative to the laser field of view. In this case, any point on the surface of the object will appear to move in direction B or C based on the angular relationship of the laser field of view relative to the transmission direction A. In this calibration, a scan of the calibration target is used to determine the angular relationship. It is known that any step-like differences along the edge between the two laser module traversals are due to the angular relationship. The angular relationship can be stored in the system memory, allowing the system processing unit to subtract any steps in the scan data.

[0174] When scanning the entire conveyor, the distance W that the laser module scans on the conveyor can be the width of the conveyor 150, or if the position of the object on the belt is known, W can be the conveyor area. By setting W to correspond only to the conveyor area when the position of the object is known, a higher scanning throughput can be achieved at the same resolution, or the same throughput can be achieved at a higher resolution.

[0175] By performing this stacked scanning process, objects larger than the laser field of view 240 can be scanned and noise is also reduced. This configuration allows for higher resolution images to be captured based on the distance between the motor steps and the acquisition rate when traversing the laser over a larger area, and is not limited to the native resolution of the profiler itself.

[0176] As described above, performing overlapping scans helps reduce sensitivity to noise. Noise is caused by inherent measurement errors and mechanical noise caused by movement of the object and laser module 200 during the inspection process. By performing scans so that scan segments of the inspected object overlap, multiple point cloud data points are created for the same spatial location on the object. These multiple point cloud data points for a single spatial location allow the mean and variance (or confidence interval) of the spatial coordinate to be calculated. By scanning a portion of the inspected part several times and averaging the data, not only is the overall noise in the system reduced, but the remaining noise can also be accurately calculated and reported. This article describes more details about the procedures used to maintain scanning system performance.

[0177] By moving along only one axis at a time, an overlap region can be created between consecutive passes. For example, the object can be fixed in direction A while the scanner moves in direction B or C. In this case, the overlap region will be rectangular.

[0178] Fig.15 1 illustrates a scanning process in which the laser module 200 moves in the same direction as the object 160 along the transport path 150 and moves perpendicular to the transport path. By moving in the same direction as the object and at the same speed in that direction as the object, no tilt is introduced into the scan. For example, Fig.15 As shown, the laser module 200 can move in a direction C perpendicular to the direction A of movement of the object 160 along the conveyor 150, wherein the laser field of view 240 is scanned across the path of the conveyor 150 in a raster scanning process. Fig.14 The laser module 200 only moves in directions B and C perpendicular to direction A. Fig.15 In the process shown, the laser module 200 also moves in direction A at the same speed as the object while performing the scan. When the laser module reaches the end of the scan, it then moves a distance A' equal to the length of the laser's field of view 240. The laser module 200 then reverses direction and moves in direction B while moving in direction A at the same speed as the object 160. Thus, with respect to the object 160, the laser module 200 performs one scan of the entire object, as shown in FIG. Fig.15 shown.

[0179] Fig.16425 is a flow chart illustrating the processing performed to convert a CAD model into a format that can be used to interface with the corrected 3-D point cloud data obtained from the laser module 200 scan of the inspection object. The CAD model can be in a dgn, dwg, dxf or stl file format, or other formats that allow for the generation of CAD files. The process can be performed in real time as the object is inspected, or can be performed before initiating a quality inspection on the object. When performed before initiating an inspection, a CAD model of the object to be inspected can be retrieved and processed, with the results stored in the CAD model database 425. Fig.16 As shown, CAD model 1600 is loaded into the system, and mesh conversion step 1601 is performed to the CAD model to generate CAD mesh 1602. Then, plane extraction step 1603 is performed to CAD mesh 1602 to generate CAD plane mesh 1604. CAD plane mesh 1604 represents the plane or view of the object from a predetermined viewpoint. These planes or views can be manually generated in advance by editing the CAD model using CAD editing software such as MeshLab, and can be used for extracting plane 1603 steps. Doing so can make the CAD viewpoint match the position and orientation of the predetermined object, which is set by the operator at a specific position and in a specific orientation in the inspection system. In an alternative embodiment, the plane or view can be automatically identified based on the natural resting position of the object with reference to the orientation of the transporter. The orientation of the transporter can be obtained by overscanning the object so as to capture the transporter in the scan. After determination, these planes can be automatically generated.

[0180] The key function (CTF) parameters for the CAD plane mesh 1604 can then be extracted by extracting CTF parameters step 1605. These CTF parameters are user-defined dimensions that need to be measured and verified. These parameters can be manually identified by an operator, or can be automatically determined by system processing. For example, many times a soft copy or hard copy illustration of an object is provided including tolerances and CTF data, which can be extracted using document processing or OCR software. Based on the step of extracting CTF parameters 1605, CAD CTF parameters 1609 are generated and stored in the CAD model database 425. The CAD plane mesh 1604 is also subjected to a format conversion step 1606 to generate a CAD point cloud 1607 and a CAD range image 1608, which can also be stored in the CAD model database 425. The CAD range image 1608 represents the six faces of the object being inspected.

[0181] Fig.171 is a flow chart illustrating the processing performed on the image data from the optical acquisition unit 110. Such processing includes identifying objects on the transport 150 to be inspected, as well as the position and orientation of the object 160 on the transport 150. As shown, the optical acquisition unit 110 performs image capture 1701 on the object 160, generating at least one digital image 1702 of the object. The digital image 1702 then undergoes a segmentation process step 1703, and generates bounding boxes 1704 that can be applied to the digital image 1702 to locate regions of interest that may be objects or features of the inspected object. These bounding boxes 1704 can be used as input to the system processing unit 170 to determine the scanning area of ​​the laser module 200. In a part extraction step 1705, separate parts 1706 defined by the bounding boxes within the digital image 1702 are extracted. A part identification and lookup process 1707 is then performed on each separate part 1706 from the digital image 1702. As part of this process of identifying objects, a pattern matching process is performed in which the system performs a CAD model lookup 1708 to compare objects located in the CAD model database 425 to the separated parts 1706. The bounding box 1704 may include different inspected objects that require reference to different CAD models. The pattern matching process may be any generally known pattern matching algorithm. When a match is found in the CAD model database 425, the associated CAD point cloud 1607 is identified for subsequent processing. Additionally, a CAD range image 1608 is identified and then used in a part registration step 1709 to identify a translation and orientation matrix 1710 that is used to rotate the CAD data to match the orientation of the point cloud obtained from the laser module 200 scan of the object 160, as described below. In an alternative embodiment, the surface dimensions of the object 160 in the digital image 1702 may be determined and used as an alternative to the pattern matching used in the part identification and lookup process 1707.

[0182] Fig.18A The process of merging data from laser module scanning with information obtained from the CAD model information of the scanned object is illustrated. The process determines the difference between the actual object size obtained from the laser scan and the expected object size based on the CAD model tolerance. Fig.18A As shown, the original, unstructured point cloud data 1810 is obtained from the laser module scanning of the object 160, as described above with respect to Fig.14 and Fig.15As described. Due to the continuous traversal of the part, as the laser module scans orthogonally, the line profile data sets (raw point cloud data acquired by a single laser field of view 240 line of the laser driver 210) will be offset from each other by a distance equivalent to v*Δt, where v is the traversal speed of the object and Δt is the time interval between a line scan of the object and the next line scan. Alternatively, the timer unit 480 generates a trigger signal to the laser module 200 to capture the line profile. The timer unit 480 also generates a trigger to the machine controller 430 to capture the position data of the transport and the laser module. The use of a trigger signal allows the system to read position data from, for example, an encoder, rather than calculating the position based on speed. In both cases, a re-phasing of the raw point cloud data 1811 is implemented to align and average these multiple point clouds together into a single re-phased object point cloud 1812. In order to obtain an accurate point cloud representation of the object, the raw point cloud data is re-phased by translating all points in the point cloud by an amount equivalent to -v*Δt. Alternatively, Δt may be considered as the total time elapsed from a set time, such as the time the machine is in use, to the current row scan timestamp.

[0183] The inspected object is then registered to the CAD model 1813 for that particular object. As part of this registration process, the CAD point cloud 1607 and the translation and rotation matrices 1819 are retrieved to match the CAD point cloud with the re-phased object point cloud 1812. The re-phased object point cloud is also registered to the CAD model at 1814 so that the point cloud is aligned with the CAD model. That is, the obtained re-phased object point cloud is superimposed on the CAD model information for the particular object. The difference between the CAD model point cloud 1607 including the structured data and the registered point cloud 1814 is then calculated at step 1815. A subtractive match may have been performed to register the CAD model with the point cloud data as closely as possible. The match finds the minimum total square difference between the CAD model and the point cloud to determine the optimal overlay between the two. When calculating the difference 1815, for each point on the CAD model that has an associated re-phased point cloud point, a D value representing the absolute difference between the spatial coordinates of the two points is calculated. The D value will be calculated as the magnitude of the vector of the two points.

[0184] D=√(〖|x_1-x_2|〗^2+〖|y_1-y_2|〗^2+〖|z_1+z_2|〗^2)

[0185] Where x, y, and z represent the spatial coordinates of CAD and point cloud data.

[0186] This same D value is used in the CAD model registration process described above, where the CAD model is registered to the point cloud data based on the orientation where the D value variance is at its minimum.

[0187] After calculating the differences, a set of point cloud differences 1816 is obtained in the form of D values ​​for each point. A heat map creation process 1817 is then implemented to generate a heat map 1818 that shows areas of high deviation between the CAD model and the point cloud obtained from the object scan. The heat map can be used by an operator to determine whether the deviation between the dimensions of the actual object being inspected and the expected dimensions of the object from the CAD model database is outside of a predetermined tolerance level set in the extract CTF parameters 1605. If the heat map 1818 shows areas outside of the tolerance level, the part can be identified as defective. The heat map can be user configurable, where the user sets predetermined limits for when deviations are considered relevant enough to be indicated on the heat map.

[0188] Fig.18B and Fig. 18C Illustrated Fig.18A Details of the registration process 1813. The entire registration process is divided into two stages: 1) library generation, and 2) point cloud registration. Fig.18B The library generation step is described to create a set of feature vectors (or weights) for use in the next stage. As shown, the system receives a reference 3D mesh model 1820 representing the entire geometry of the rigid object. Then, based on the convex hull and center of gravity of the reference mesh model, the various resting positions 1821 of the rigid object are calculated. The resting position of the rigid object is the position where the object is stable on the horizontal plane. For each of these resting positions of the rigid object, a view library 1822 is generated from several rotation versions (e.g., 360, in 1 degree increments) of the reference mesh model. Specifically, the view corresponds to a 2D projection of the reference 3D mesh model at a given resting position and a specific angle. Note that downsampling can be applied during the 2D projection process to reduce computational costs. The view library is then processed in the feature space using principal component analysis to generate a set of feature vectors (or weights) 1823 for each element in the library set. The elements in the library set can be determined by performing eigendecomposition on the library view set 1822 generated by the image rotation version corresponding to the resting position, after subtracting the mean image (which is the average of the grouped views). The data after the mean is subtracted contains information about the differences between the views in the set. The top eigenvectors corresponding to the highest eigenvalues ​​are selected to generate a projection matrix. Each image in the library view set has its vector representation w in the feature space, calculated by multiplying with the projection matrix to generate an eigenvector. Finally, the system saves both the feature projection library (or projection matrix) and the set of eigenvectors 1824 in the library for use in identifying new point clouds generated in future scans. If more than one rigid object is considered in the system, steps 1820 to 1824 can be repeated to generate the final library (or projection matrix) and set of eigenvectors. These saved libraries and sets of eigenvectors will be used in the next stage for the registration of 2D and 3D point clouds, such as Fig. 18C As described in.

[0189] Fig. 18C The process of 2D and 3D registration and matching of point clouds is depicted. As shown, the system receives a 3D point cloud 1830, which may be a re-phased part point cloud. The target point cloud represents the geometry of a rigid object at one of its resting positions. The corresponding rigid object may be one of the rigid objects considered in the previous library generation stage. Next, the saved feature projection library (or projection matrix) and feature vector set 1831 are loaded. Then, in step 1832, the 2D projection of the target point cloud is calculated to generate its feature vectors (weights) based on the feature projection library generated in the previous stage. Note again that downsampling can be applied during the 2D projection process to reduce computational costs. It is assumed that the feature projection library is generated using the same rigid object. In step 1833, a rough registration is performed by identifying the resting position and rotation angle of the rigid object that best matches the target point cloud. This is achieved by finding the best match or closest distance between the feature vector from the target point cloud and the saved feature vector set of the rigid object and the corresponding reference mesh model from the previous stage. Metrics such as minimum mean square error or other error metrics can be used to calculate the distance (or matching error) between feature vectors. Matches can also be calculated as the Euclidean distance between two vectors. By using a KD tree representation to store vectors, the search for the closest match can be optimized. The object and orientation of the closest match are output as hypotheses for the object and orientation of the target point cloud representation.

[0190] The initial corresponding geometric transformation 1834 between the target point cloud and the reference grid model resulting from the shelving position and angle identification step is then generated. The translation is calculated as the position difference of the center of mass between the image from the 2D projection of the target point cloud and the closest match determined above. The rigid 2D geometric transformation includes information such as translation and rotation matrices. Next, the resulting geometric transformation is applied to the target point cloud 1835. The output of this step is a translated and rotated target point cloud that will be roughly aligned with the identified reference grid model (and associated rigid objects). This can be achieved using the calculated translation and rotation as a 3D transformation matrix T. The transformation matrix T is applied to align the target point cloud with the CAD model, creating a registration between the point cloud and the CAD model.

[0191] Next comes the fine 3D registration step 1836 which completes the overall registration process. In some cases, the scanning process can produce artifacts in the scan data due to secondary reflections of the laser on the scanned surface. This typically occurs in recessed portions of the surface, or in areas of the part that are close to being parallel to the laser path. The resulting artifacts appear as smaller clusters of points that are clearly separated from the main point cloud. These can be effectively filtered out using the artifact filtering methods below.

[0192] At each point in the cloud, all nearby points are identified within a predetermined connection radius threshold. These points are then all considered connected. Directly connected points are logically clustered together. Second-level connections also constitute connectivity. Therefore, if point A is connected to B and B is connected to C, then A is connected to C. This relationship is determined over the entire point cloud, with the ideal result being one large continuous connected cluster representing the entire scanned surface. If artifacts are present, they are typically not connected to the main point cloud and will be in a separate cluster. After connectivity for all points has been established, filtering can be achieved in several ways.

[0193] First, and simplest, one could accept the first level largest cluster as the only useful points and discard all others - leaving only the largest connected cluster as the point cloud. Second, is to sort the clusters by size (number of points) and then progressively aggregate them in order of decreasing size until a predetermined percentage of all the original points are included. This could include the largest connected cluster alone, or the largest and one or more smaller clusters. This approach allows for scanning of geometries where there are discontinuities in the actual surface data due to the geometry of the part. For example, a large vertical step parallel to the laser, so no points would be collected on the vertical surface of the step. Finally, a third approach is to aggregate all point clusters (either in number of points or geometric size) that are larger than a predetermined size threshold into the output cloud.

[0194] Fine 3D registration 1836 may use an algorithm such as Iterative Closest Point (ICP), which is a popular algorithm due to its simplicity. The inputs to the algorithm are: target and reference point clouds, an initial estimate of the geometric transformation (from step 1834) to align the target with the reference, and criteria for stopping iterations. The output of the algorithm is a refined geometric transformation matrix. The algorithm includes:

[0195] 1. For each point in the target point cloud (from the entire set of vertices, often referred to as dense pairs of vertices from each model or pairs of vertices selected from each model), match the nearest point in the reference point cloud (or selected set).

[0196] 2. Use the RMS point-to-point distance metric minimization technique to estimate the combination of rotation and translation that best aligns each target point with its matching point found in the previous step. Additionally, points can be weighted and outliers can be removed before alignment.

[0197] 3. Transform the target point using the obtained transformation.

[0198] 4. Iterate (reconnect points, etc.).

[0199] Then, based on the refined geometric transformation matrix obtained in the above refined 3D registration step, the final geometric transformation between the target point cloud and the reference mesh model is determined 1837. Please note that the geometric transformation is usually an affine transformation consisting of: a combination of one or more of translation, scaling, shearing or rotation transformations.

[0200] Iterative Closest Face Registration The Iterative Closest Face (ICF) algorithm can also be used instead of the ICP algorithm to achieve a more refined 3D registration and iterative process to determine the refined geometric transformation matrix. ICF is an algorithm that iteratively optimizes the transformation matrix to rotate and reposition the point cloud so that it is optimally aligned with the CAD model mesh. In ICP, the movable point cloud is aligned with a fixed reference point cloud over the course of multiple iterations. In ICF, the movable point cloud is aligned with a fixed grid. In both algorithms, the resulting transformation matrix is ​​usually restricted to only perform translations and rotations. There is no spatial distortion and usually no scaling.

[0201] In both algorithms, iterative passes are performed, where the alignment gradually approaches the ideal state after each pass. In each pass of ICP, the nearest point in the reference point cloud is found for each point in the movable point cloud. For each point in the movable point cloud, an error vector is calculated. After all error vectors are found, they are summed to form the corrective translation and rotation of the point cloud. The corrective translation and rotation are compounded into the optimized transformation matrix and the process is repeated. In contrast, ICF locates the nearest point on the CAD model mesh face during each pass, rather than utilizing the nearest point in the reference point cloud. This point can appear anywhere: on a continuous planar surface of any face of the mesh, within the boundary of the face, at the edge of the face, or at a vertex of the face. It may not appear on an extended face plane outside the boundary of the CAD model face.

[0202] The vertices of the CAD model mesh may be very "sparse". For example, a simple cube model may contain only 8 points. Using only these 8 points, ICP can only associate each point in the movable point cloud with one corner of the cube. This will not produce a good model fit, especially when the point cloud contains points on all surfaces of the cube. To circumvent this problem, interpolation can be performed on all faces of the CAD mesh to produce a point cloud with uniform mesh coverage on all surfaces. ICP is then performed relative to the interpolated CAD point cloud rather than the original mesh. In this interpolated CAD mesh method, ICP has granularity / quantization errors caused by the discrete points at which the interpolation is performed. ICF does not need to interpolate the CAD mesh, thereby saving processing time. Alternatively, ICF uses triangular plane segments, where each face is described by the equation ax+by+cz+d=0, and the spatial extent of the plane segment is limited by its three vertices. Provided with three-dimensional vertices A, B, and C, the values ​​of a, b, and c can be determined from the cross product of vectors AB and AC. The value of d can be determined by substituting these values ​​into the equation and solving for d using one of the vertices. Since planar segments can describe large spatial regions using only three points, CAD models are often much more compact than point cloud representations of the same object. During the ICF process, the point on the CAD model surface towards which each movable point is optimized is precisely located without the granularity / quantization errors that can be present in ICP methods. ICF also allows for reduced memory usage when executing the algorithm, since CAD models that are more compact than point clouds can be used without interpolation, and computations may also be faster since fewer reference surfaces need to be searched to find the nearest.

[0203] Whether ICP or ICF is used to iteratively optimize the transformation matrix to rotate and reposition the point cloud, the final geometric transformation produced by the process is then applied 1838 to the target point cloud to obtain the final alignment or registration with the reference mesh model. Finally, the difference between the transformed target point cloud and the reference 3D mesh is measured 1839. The matching results are recorded online or passed to the next step 1840 of the inspection system.

[0204] Fig.19 An illustration of how a quality inspection system can inspect for defects in an inspected object is provided. For example, the system can automatically check tolerances based on defined CTF specifications and if the system tolerances are exceeded during the inspection of the object, the system will notify the operator. For example, when inspecting manufactured parts from a mold, the mold will wear over time. As the mold wears, the dimensions of the part from the mold will change from the expected dimensions in proportion to the amount of wear on the mold. When the mold is excessively worn, the part produced will no longer be within the tolerance range for its proper and intended use. As Fig.19 As shown, also retrieved as Fig.18AThe point cloud difference 1816 calculated for each point in the registered point cloud and the CAD CTF parameters 1609 are used to determine whether each point in the registered point cloud is a CTF in step 1901. If the point is not a CTF point, then the point is ignored in 1902. However, if the point is a CTF, then an update to the operating tolerance is performed in 1903. A check is then performed at 1904 to determine whether the operating tolerance exceeds a predetermined threshold. If the threshold is not exceeded as defined by the extracted CTF parameters 1605, then it is determined at 1905 that the operating tolerance is within acceptable limits. However, if the operating tolerance exceeds the predetermined threshold, the system will issue a notification 1906 to the operator.

[0205] Defective objects may also be automatically removed from the transporter 150 using any known removal method in the industry. Defects that are unique to manufacturing parts by injection molding may include flow lines, burn marks, sink marks, spray, delamination, short shots, flash, or other defects. The CAD CTF parameters 1609 may also include surface defects or optical characteristics, such as but not limited to color, texture, or finish, to identify contaminants such as dust, oil, or other foreign matter present in the separated parts within the digital image captured by the image acquisition unit 110. In this embodiment, step 1901 will determine whether the optical characteristics exceed the specified optical CAD CTF parameters.

[0206] Fig. 20 An example of an inspection system 1100 is provided that uses a conveyor belt as a transport 150. The conveyor belt can be placed around rollers 1120 that move the conveyor belt when rotated in the direction shown in A. The rollers can be driven by a transport motor controller 485. As shown, an optical collection unit 110 can be located at a first position along the transport 150, and a laser module 200 including a laser driver 210, a laser sensor 250, and a laser beam 140 is located further downstream on the inspection path. The object 160 to be inspected is moved by the conveyor belt so that it passes under the optical collection unit 110 - if present - and the laser module 200. Optionally, a belt tensioner 1140 can be provided underneath the moving conveyor belt to provide a solid surface for inspection and eliminate any vibratory motion on the conveyor belt. The reduction in vibratory motion reduces noise in the obtained optical image and laser scan of the object to be inspected. As shown in FIG. Fig. 20As shown, the conveyor belt can be marked to allow the optical collection unit 110 or the laser module 200 to be automatically calibrated. Bevel marks such as scale marks 1110 along one or both sides of the conveyor are used for calibration and position identification on the conveyor. In addition to the scale marks, other marks can be included to assist in identifying the inspected object or the position of the object on the conveyor. These references can include numbers, scales of different shapes, or other identification marks. As an example, a single reference scale can be used on both sides of the conveyor. The scale mark will be elongated relative to other uniform scales for easy identification. The system can perform a modulation transfer function (MTF) on the scale marks, and the optical collection unit 110 can be additionally calibrated for sharpness, halo, focus, and depth of field using the scale marks as a reference. Due to the uniformity of the scale marks, the position, depth of field, and resolution of both the optical collection unit 110 and the laser module 200 can be calibrated, as well as image and point cloud distortion caused by the non-verticality of the tool can be identified. Based on this data, the distortion can be calculated and corrected. The tick marks can also control velocity chatter by identifying any velocity changes not registered by the transport motor controller 465. This information can be used to correct the re-phased point cloud data 1811 by adding a phase offset.

[0207] Fig.21 A flow chart is provided, detailing an overview of the process 2100 of determining whether an inspected object is defective. As shown, a CAD model of the object is uploaded and converted at 2101, generating a CAD model mesh point cloud. As the object 160 moves down the conveyor 150 in the inspection system 100. If present, the object is imaged by the optical acquisition unit 110. The object passes under the laser acquisition module 200, which performs a laser scan. The result of the laser scan is a 3-D point cloud of the object captured at 2102. At step 2103, the 3-D point cloud of the object is aligned with the CAD model point cloud, and then a comparison between the two is performed at 2104. The difference in the points is determined, and it is determined at 2105 whether the inspected object 160 is defective. If the object 160 is defective, the object can be rejected from the conveyor at 2106, and the statistics about the defective object can be updated. If the object is not defective, the statistics are updated at 2107 to calculate another acceptable product that passed the inspection process. The statistics may include environmental characteristics such as temperature and humidity, as well as operating tolerances for key functional parameters. The statistics may be related to a mold or a specific cavity based on the quantity or other unique features formed on the object. The statistics may be used to schedule preventive maintenance for both the machine and the mold.

[0208] Fig. 22An example user interface is provided that can be displayed to the operator at the end of the inspection process, showing the inspection results. The user interface provided to the operator also includes the following functions:

[0209] Import / Export

[0210] ○ Load the point cloud data file into the memory

[0211] ○ Load the CAD model STL file into the memory

[0212] ○ Export point cloud as PCL file

[0213] ○ Export pre-processed CAD models as proprietary “Part Inspection Profile” files

[0214] ●Display / Visualization

[0215] ○ Display point clouds in 3D view with the ability to rotate, translate and zoom

[0216] ○Display CAD models in 3D view with the ability to rotate, pan and zoom

[0217] ○ Display point cloud overlaid on CAD model with fitting error annotations

[0218] ●CAD model preprocessing

[0219] ○ Calculate the list of pending position transformations

[0220] ○Calculate the Z-axis 2D projection image of the CAD model in the shelf position

[0221] ○Interpolate surface points of CAD models

[0222] Point cloud operation

[0223] ○Center the point cloud

[0224] ○Rotate the point cloud to a new orientation

[0225] ○Remove captured ground data points from the point cloud

[0226] ○Perform capture artifact filtering on the point cloud

[0227] ○Calculate the Z-axis projection image of the point cloud

[0228] ○ Find the placement of the object based on comparison with the projected image from CAD

[0229] ○ Perform 3D alignment of the point cloud with the CAD model by

[0230] ■Iterative Closest Surface Algorithm

[0231] ■Iterative Closest Point Algorithm

[0232] Comparison between cloud and CAD

[0233] ○ Calculate the error / deviation from the CAD model to the point cloud

[0234] ○Calculate the RMS and peak error of the entire point cloud

[0235] ○Calculate RMS and peak errors on specific surfaces of CAD models

[0236] ○Compare the surface error to the inspection threshold

[0237] ○Calculate the orientation angle of the plane surface and compare it with the inspection tolerance

[0238] ○ Use "hot" colors and "blurred" lines to illustrate errors in CAD rendering

[0239] like Fig. 22 As shown, the user interface 2200 can be displayed to the operator on the operator display 128 or remotely through the network interface 445. The user interface 2200 can be updated in real time as the object 160 is inspected, or the operator can view statistical trends or historical data for the inspected objects. The user interface 2200 can organize the inspected objects to show only those that are out of tolerance or close to being out of tolerance so that the operator can focus on problem areas.

[0240] The user interface 2200 may include a display of a CAD model 2210, a point cloud 2220 from the laser model 200, and a histogram 2250 of errors and a 3D heat map 2230 of errors. The operator may select the CAD model 2210 to open the CAD model database 425 to display, add, or edit the CTF parameters used to extract the CTF parameters 1605. The point cloud 2220 may be the original part point cloud 1810 or the registered point cloud 1814, as defined by the operator's preferences. The operator may also define whether data from one or more plane scans is displayed in the 3D heat map 2230 of errors. An inspection report 2240 may also be displayed. The inspection report 2240 may include key functional parameters with ranges and actual measurements. The results may be color coded to enable the operator to quickly identify measurements that are out of tolerance. For example, red may be used to indicate that a measurement exceeds its tolerance, while yellow may indicate close tolerance. The user interface 2200 may also allow the operator to edit key functional parameter limits to change what is acceptable or unacceptable. The inspection report 2240 may also include a graphical representation of the object with the object's measured dimensions shown directly on it. The measured dimensions are superimposed on the graphical representation to display the measurement results in a format similar to the original graphical specifications. The user interface 2200 may also include historical trends of key functional parameters and statistical process control information across multiple batches or shifts. Using trends and statistical process control, mold wear over time can be identified and tracked. This data can be used to correlate changes in feed or process changes, such as temperature or pressure. The operator can take appropriate action to repair or replace molds that produce objects that are out of tolerance. In the case of a mold with multiple cavities, objects from a specific cavity can be identified and tracked across multiple batches or shifts.

[0241] In injection molding, operators need the ability to qualify new molds or assess the wear of existing molds. A baseline CAD model of the mold can be loaded and a color overlay heat map of the errors for an object or batch of objects can be overlaid on the mold to indicate where on the mold the tolerances are exceeded. This will characterize and track the mold from its initial installation to when the mold needs to be repaired, refurbished, or replaced. A heat map display over time can be provided to the operator, showing the operator the wear of the mold over time.

[0242] Each mold may contain one or more cavities that produce an object. When the mold is built, each cavity will be produced based on the CAD model. When the mold is designed, each cavity will have a unique deviation from the CAD model. When the mold is made, each cavity can be characterized to produce a cavity CAD mesh unique to each cavity. Using the cavity number or by using the cavity's unique deviation as an identifier for the cavity, the operator can identify which cavity produced a defective object. This will characterize and track each cavity from its initial installation to the time when the cavity needs repair. This can be displayed to show the operator the wear of each cavity over time.

[0243] Maintaining scanning system performance

[0244] As described above, the inspection process of an object requires several motion passes between the object and the scanning device to ensure that the entire region of interest of the object is scanned. Components of the motion system of both the mobile scanning device and the object itself are subject to wear and performance degradation, which may cause the scan data from each pass to be misaligned with other scan data. This degradation can invalidate the carefully obtained calibration information, which is intended to enable accurate and precise registration of scan data from multiple passes of the object. There are registration algorithm methods that can compensate for some of the degradation, but without intervention, the degradation may persist or worsen. In addition, when adequate compensation is no longer possible, analysis of the degradation can be used to improve compensation and predict system failures.

[0245] As the inspection system is used, the mechanical components used to move the profilometer or other scanning device and the platen or conveyor that advances the inspected object may cause misalignment and noise in the measurement. In addition, as the mechanical components become worn, shift positions, and degrade with use, and as the connections between components change due to the operation of the system, the noise may change over time. The introduction of noise and misalignment requires continuous calibration of the system to compensate for these conditions. In addition, due to the production environment, it is desirable to reduce the frequency and time required to calibrate the system so as not to affect the productivity of the manufacturing operation. Therefore, a calibration process is used to monitor the overlapping area between subsequent traversals projected onto a known surface, such as the top of the surface supporting the inspected part. This top surface of the supporting object can be referred to as the "ground" of the inspection system. If a misalignment is determined in the overlapping area, the system either notifies the operator that calibration is required or automatically calibrates the system and records the event.

[0246] During the scan, as described above Fig.14 As described above, the relative movement of the scanner and the object is accomplished by translation of either or both of the object and the scanner. Figure 1AAs shown, a transport mechanism 150 such as a conveyor belt or a platen can move the object along the inspection path, and a scanning device 200 can be moved relative to the object by a mounting assembly 260 and a traversal mechanism 270 such as a lead screw. Figure 1A The position of the scanner can be tracked by monitoring an encoder mounted on the traversal mechanism. The mounting assembly constrains the scanner's posture to ensure that it points in a consistent direction. Complete scanning coverage of the area of ​​interest of the object can be achieved by moving the scanner back and forth across the object several times, forming multiple traversal scans containing overlapping areas, such as Fig.14 This article Fig.14 as shown in the description.

[0247] A key aspect of this configuration is that some translation elements can change speed by changing direction or speed. Changes in direction, especially reversals of direction, may exhibit mechanical or electrical hysteresis in the position or attitude of the object or scanner. For example, when moving the scanner by rotating a lead screw, there may be mechanical hysteresis due to tolerance limitations of the mounting components. When the lead screw changes rotation direction or speed, this may cause several effects. Changes in rotation direction or speed may impart different amounts or directions of torque to the mounting components. Due to mechanical tolerance limitations, this may introduce changes in the scanner attitude. In addition, mechanical hysteresis may introduce offsets in the scanner position indicated by the encoder. Mechanical hysteresis and unnecessary changes in the scanner attitude may cause apparent spatial shifts of corresponding points in the overlapping area of ​​the traversal. In other words, points recorded during different traversals may correspond to the same physical location on the object, but have different coordinate values. The difference between these points is called spatial shift. This spatial shift causes the points in the traversal to be misaligned.

[0248] The performance of the scanning system can be maintained by monitoring changes in the scanner's posture or position. Fig.23 The following process is described: monitoring scanning system performance, correcting for spatial shift, and notifying the user when the degree of spatial shift exceeds a threshold, thereby allowing the user to request a service procedure or recalibration so that the scanning continues to perform accurate object scans. The process begins by scanning the object 2301, using the scanner to traverse across the object two or more times so that there is an overlap in the scan area. The overlapping area does not necessarily need to contain the scanned object, and can only include the surface on which the object rests, such as the top of a conveyor or platen.

[0249] Overlapping regions are detected 2302 and segmented based on the traversal. Detection of overlapping regions can be accomplished by assuming that the positional accuracy of one scan is sufficiently similar to the previous scan. Detection and positioning of reference elements on the support surface can also help establish overlapping regions.

[0250] The precise degree of displacement between overlapping regions is estimated and recorded 2303. Well-known algorithms such as iterative closest point (ICP) can be used to estimate displacement. The degree of displacement is classified 2304 and recorded in memory 2305. The classification can be based on changes in speed. For example, one classification can be defined as when one of the mechanical components used to move the profilometer or other scanning device changes from one direction of travel to another. In this case, one category occurs when the direction of travel changes from left to right, and the other category occurs when the direction of travel changes from right to left. If the classified degree of displacement exceeds a predetermined threshold 2306, a transformation matrix is ​​created to correct the displacement and align the entire traversal scan with each other to form a complete scan of the object 2308. If the classified degree of displacement does not exceed the predetermined threshold 2306, the classified degree of displacement in the overlapping region is compared with the previous record 2307. If there is a change in spatial displacement, an estimate of when the spatial displacement in the overlapping region will exceed the predetermined threshold will be calculated 2312, and the user will be notified 2309 or 2313. The user may optionally be notified of the extent of the shift 2309. If the extent of the shift exceeds a predetermined threshold or cannot be corrected 2310, the user may be notified to request a service procedure to correct the cause of the shift or to recalibrate the system 2311.

[0251] The history of class shifts may be tracked and analyzed 2313 to predict when the degree of shift may exceed a predetermined threshold 2310. The user may optionally be notified of a predicted failure 2313, allowing the user to take action to recalibrate or perform a service procedure before the performance of the inspection system degrades to unacceptable levels.

[0252] General systems and processes Summarize

[0253] The following is a general summary of the system and process. An operator using the inspection system first loads a CAD model of the object to be inspected into the inspection system. As described above, the CAD model may already be included in a CAD model database, or loaded into the CAD model database by the operator before the operator begins the inspection process. The object to be inspected is then placed on a transporter in a known position and predetermined orientation, either manually or automatically by the operator, and the transport motor controller drives the transporter so that the object moves into the system at a known speed.

[0254] Alternatively, the position and orientation can be determined by imaging the object using an optical acquisition unit. As the object moves into the optical acquisition system, the optical acquisition system captures at least one image of the object. The at least one image is sent to a system processing unit, such as described above with respect to Fig.13The system processing unit analyzes the image and determines the contour of the object surface. The system processing unit performs a pattern matching technique between the determined object surface contour and a CAD range image from a CAD model database. Based on the pattern matching results, a corresponding CAD model is selected from the database for comparison with the inspected object. The system processing unit also uses the captured image to determine the position and orientation of the object on the transporter.

[0255] A corresponding CAD model of the inspected object is selected from the database for comparison with the object. As the object moves along the conveyor within the laser field of view, the system processing unit instructs the laser module to collect data along the conveyor. The laser module traverses perpendicular to the direction of transport, moving back and forth within a predetermined area as the object moves along the conveyor, so that the field of view of the laser module passes across the object one or more times. The reflected laser light is received at the laser sensor, and the received laser light is used to generate a 3-D point cloud of the scanned object. This raw point cloud of the scanned object is sent to the system processing unit, which corrects the point cloud based on the speed of the object when moving along the conveyor and the speed and direction of the laser module when performing the scan. As previously described, correction of the point cloud can also be performed by using positioning data. The corrected 3-D point cloud of the object is then prepared for analysis.

[0256] The system processing unit retrieves the CAD point cloud of the object. The CAD point cloud is rotated to match the determined object coordinate geometry. The corrected 3-D point cloud from the laser scan of the object is then interpolated to a predetermined geometric grid for comparison with the CAD model. Through subtractive reasoning, the interpolated corrected 3-D point cloud and the CAD model are paired, and a series of D values ​​are calculated and associated with each point in the point cloud. The D value is the subtracted difference between the relative position of the CAD model corresponding to the inspected object and the corrected 3-D point cloud data of the object 160. The D value can correspond to a set color based on the user's preference. For example, the user can select red for D values ​​that exceed a predetermined tolerance limit and green for values ​​within the tolerance. The predetermined tolerance is based on how much deviation is allowed between the CAD model of the object and the point cloud of the scanned object. A CAD model with a color overlay will be generated and saved in a report, and a smoothing process can be applied to make the colors in the final overlay of the heat map look uniform. Smoothing the D value parameters can be performed by averaging or other means to obtain a smooth gradient between color-coded segments of the visual representation of the point cloud data. The visual representation of the point cloud data can be presented as: a set of color-coded point cloud data points; a set of color-coded point cloud data points layered on an image of a registered CAD model; or a color-coded polygonal mesh created by forming polygonal surfaces between the point cloud data points. When presented as a color-coded polygonal mesh, the polygonal surfaces are color-coded according to their assigned D values, which are the average of the D values ​​of the points connected by the polygonal surface.

[0257] If the D value exceeds a certain threshold value defined by the CAD CTF parameters, the conveyor stops or the inspected object is removed from the inspection line, and a color overlay heat map for that particular object can be shown on the operator control panel or saved to a fault report. Based on the operator or customer's specifications, if the D value enters a certain threshold area, the system will issue an alert to notify the operator. The alert can be an audible or visual alert presented at the operator control panel. The alert can be color-coded to give urgency based on the user's preferences. When the alert is an audible alert, different tones, sounds, or volumes can be used to symbolize urgency. The alert can also be a predetermined or customized email or text message sent to the predetermined recipient through a network interface.

[0258] If the D value does not exceed a certain threshold defined by the CAD CTF parameters, the transporter and object are allowed to continue. A color overlay heat map for that particular object can be shown on the operator control panel or saved in an operator report.

[0259] The optical collection unit and laser module 200 inspect the plane or face of the object presented to the optical collection unit and / or laser module. However, the object may have CAD CTF parameters on the plane or face not presented to the optical collection unit and / or laser module. That is, one surface of the object will be in contact with the conveyor and therefore will not be visible to the optical collection unit and / or laser module located above the conveyor. In order to image the bottom surface of the object in contact with the conveyor, thereby obtaining a complete scan of all surfaces of the object, the system may include an additional laser module located at least below the top surface of the conveyor. The belt can be made of a transparent material, allowing the bottom laser module to scan the surface of the object through the transparent conveyor. In such a configuration, both the bottom and top laser modules will scan the inspected object, capturing laser scans of all surfaces. The scan of the bottom surface is combined with the scan from the top laser module to form a combined point cloud representing all surfaces of the object. This combined point cloud is then compared with the CAD module in the database as described above. An additional optical collection unit can also be placed below the conveyor to capture images of the surface in contact with the conveyor. Alternatively, the conveyor can include two separate belts with a small gap between the two separate belts. The laser module can be placed below the gap with its field of view parallel to the length of the gap. As the inspected object crosses the gap from one conveyor belt to another, the laser module located below the gap will capture scans of the bottom surface. Again, these scans of the bottom surface can be combined with the scans from the top laser module to form a combined point cloud representing all surfaces of the object. As an alternative to using a transparent conveyor and placing another laser module below the conveyor, the inspected object can be flipped or rotated to expose hidden planes of the object as it moves along the conveyor. The object can be simply flipped, or can be automatically captured and rotated so that all planes of the object are presented to the optical acquisition unit and / or laser module for inspection.

[0260] In another embodiment, the objects to be inspected can be brought to an optical acquisition unit which can identify the objects to determine if the correct plane with the CAD CTF parameters is presented to the optical acquisition unit. If the correct plane of the object is presented, it is allowed to continue along the conveyor. If the wrong side is presented, the object is flipped before being inspected by the laser module. This has the advantage of putting all objects in the same orientation and allows for a simpler flipping mechanism and reduces the area on the conveyor that needs to be scanned by the laser module.

[0261] In some cases, when a CAD model for an object is not available, a golden object known to meet the specifications may be available. In this case, the golden object will be scanned with a laser module to automatically generate and upload a CAD profile for the expected object to be inspected, and further updates will be performed in the CAD model database. In this case, the operator will place the golden object on the transporter and configure the system processing unit to store the CAD plane mesh and associated CAD point cloud, CAD range image into the CAD model database for future identification and part lookup. The operator can add the required CAD CTF parameters to the CAD model database through the operator control panel or network interface. In another embodiment, when the inspection system detects a scanned object that it fails to identify during the part identification and lookup process, the inspection system can send an alert to the operator so that the operator can take appropriate action.

[0262] In another embodiment, a gold object of known dimensions may be inspected to verify the calibration of the inspection system. In this example, an operator configures the inspection system for calibration and manually or automatically places the gold object on a transport for the laser module to scan. Additionally, by using the gold object for calibration, the working distance, focus, magnification, and similar image parameters of the laser module may be calibrated.

[0263] Polygon scanners are commonly used in laser print engines, barcode scanners. Polygon scanners can be used to perform line-oriented scans of objects with fine resolution. A rapidly rotating polygon mirror can be used to create a laser beam that performs high-speed linear scans across a transport. The polygon mirror can be rotated by a motor and can be supported by a ball bearing or air bearing rotating spindle to produce smooth rotation, thereby reducing distortion in the laser beam. In addition, the laser sensor can be a linear array, CMOS, or similar technology known in the art.

Claims

1. A method for calibrating an object inspection system, comprising: scanning a calibration target to obtain a point cloud representing the target; subdividing the point cloud into one or more tiles, the tiles containing a plurality of points of the point cloud; For each of the one or more tiles, fitting a plane model to the points contained in the tile, wherein the plane model includes values ​​of: a three-dimensional vector perpendicular to a surface of the plane; and a distance of the plane from a reference point of a coordinate system; adding normal vectors from each tile of the one or more tiles and calculating an average of the set of normal vectors; defining a reference plane external to the point cloud using the average normal vector orientation and computing each cloud point relative to the reference plane; and Use the histogram to find local maxima and create groups of points where each point's distance from a defined reference plane falls within a tolerance threshold of the local maximum.

2. The method according to claim 1, wherein: The calibration target includes reference holes at known locations and positions, and the method further includes analyzing one or more planar models to detect the locations and positions of representations of the reference holes in the one or more planar models.

3. The method according to claim 2, further comprising: Based on the detected locations and positions of the representations of the fiducial holes, a transformation matrix is ​​determined that determines the orientation of the calibration target in three-dimensional space.

4. A method for calibrating an object inspection system, comprising: Scan the calibration target to obtain a point cloud; Creating a reference plane and measuring the distance of each point to the reference plane; A gradient descent method is followed to refine the orientation of the defined reference plane; as well as A group of points is created, each point having a distance from the defined reference plane that falls within a tolerance threshold of a local maximum of a histogram of distances of the points from the defined reference plane.

5. The method according to claim 4, wherein: The calibration target includes reference holes at known locations and positions, and the method further includes analyzing one or more planar models to detect the locations and positions of representations of the reference holes in the one or more planar models.

6. The method according to claim 5, further comprising: Based on the detected locations and positions of the representations of the fiducial holes, a transformation matrix is ​​determined that determines the orientation of the calibration target in three-dimensional space.

7. A method for maintaining performance of a scanning system, comprising: obtaining two or more traversals of the object by one or more non-contact profilometers; detecting overlapping regions between the obtained traversals and classifying and recording the amount of spatial displacement; determining whether the classification degree of the recorded spatial displacement in the overlapping area exceeds a predetermined threshold; as well as Compare the recorded spatial shift data with the previous set of records.

8. The method according to claim 7, wherein: The degree of spatial displacement is estimated based on the overlap area between the two or more traversals and classified by the change in velocity.

9. The method according to claim 7, wherein: When the recorded spatial shift exceeds a predetermined threshold, a transformation matrix is ​​created to correct the shift and register the entire traversal scan to each other, thereby creating a complete scan of the object.

10. The method according to claim 7, wherein: When a change in spatial displacement is recorded compared to a previous record, an estimate is calculated as to when the spatial displacement in the overlapping region will exceed a predetermined threshold.

11. A method for aligning and fusing point clouds, comprising: receiving a target three-dimensional point cloud representing a rigid object; Calculate the 2D projection of the target point cloud to generate its feature vector; generating a corresponding geometric transformation between the target point cloud and a reference model mesh; performing fine 3D registration of the transformed target point cloud with the reference model mesh; Determine the final geometric transformation between the target point cloud and the reference model mesh to achieve 3D registration; as well as Applying the final geometric transformation to the target point cloud; as well as The difference between the transformed target point cloud and the reference model mesh is measured.

12. The method according to claim 11, wherein: The corresponding geometric transformation includes a translated and rotated target point cloud that is roughly aligned with the reference mesh model.

13. The method according to claim 11, wherein: The 3D registration is performed using an iterative closest point algorithm or an iterative closest surface algorithm to generate a refined geometric transformation matrix.

14. The method according to claim 13, wherein: The target point cloud is a movable point cloud, and wherein using the iterative closest point algorithm comprises: For each iterative pass find the closest point in the reference point cloud; For each point in the movable cloud, calculating an error vector; and A corrective translation and rotation is formed that is compounded into the transformation matrix of the movable point cloud.

15. The method according to claim 13, wherein: The iterative closest face algorithm locates the closest point on a face of the CAD model mesh during each iterative pass.

Images