Inspection of a reflective surface based on image correlation

Abstract

The present invention relates to inspection of a reflective surface based on image correlation. A system for inspecting a reflective surface includes a first imaging assembly configured to capture a first image of the reflective surface, the first image including depth information, a second imaging assembly configured to capture a second image of the reflective surface, the second image including contrast information, and a processor configured to acquire the first image and the second image. The processor is configured to perform: estimating a depth profile of the reflective surface based on the depth information, correlating the depth profile with the second image, and identifying a feature of the reflective surface based on the correlation.

Description

Technical Field

[0001] This disclosure relates to the inspection of surfaces, and more specifically to the inspection of reflective surfaces to identify defects, discontinuities and / or other features. Background Technology

[0002] In the automotive and other industries, various types of joining technologies are used to assemble various components. One example of such a joining technology is laser brazing. Brazed surfaces are typically polished immediately after brazing to achieve improved surface aesthetics and become highly reflective to optical illumination. Laser-brazed joints are usually inspected to ensure that the laser-brazed surface of such joints is free of defects that would reduce effectiveness and affect surface aesthetics. Defects can be of various sizes and shapes, and in some cases, they can be difficult to detect. A system and method for inspecting laser-brazed surfaces and other reflective surfaces that can effectively identify surface defects is desired. Summary of the Invention

[0003] In one exemplary embodiment, a system for inspecting a reflective surface includes: a first imaging component configured to capture a first image of the reflective surface, the first image including depth information; a second imaging component configured to capture a second image of the reflective surface, the second image including contrast information; and a processor configured to acquire the first image and the second image. The processor is configured to perform: estimating a depth profile of the reflective surface based on the depth information; associating the depth profile with the second image; and identifying features of the reflective surface based on the association.

[0004] In addition to one or more of the features described herein, the reflective surface is the surface of the brazed joint.

[0005] In addition to one or more of the features described herein, associating the depth profile with the second image includes constructing an image having image properties based on the depth profile.

[0006] In addition to one or more of the features described herein, the first imaging component includes a first camera and a laser oriented toward the reflective surface, the laser being configured to project a laser line onto the reflective surface, and the first image includes a series of images, each of which depicts the laser line along the reflective surface at multiple locations.

[0007] In addition to one or more of the features described herein, the depth profile is estimated based on laser triangulation techniques.

[0008] In addition to one or more of the features described herein, the second imaging component includes a second camera and an illumination device configured to project a diffuse beam of light onto the reflective surface.

[0009] In addition to one or more of the features described herein, the first imaging component and the second imaging component are in fixed positions relative to each other, and the first imaging component and the second imaging component are configured to scan along the reflective surface and simultaneously capture the first image and the second image.

[0010] In addition to one or more of the features described herein, the illumination device is positioned at a selected distance from the reflective surface and oriented relative to the reflective surface at a selected angle.

[0011] In addition to one or more of the features described herein, the selected distance and the selected angle are determined by: capturing a plurality of reference images of the reference surface, each of the plurality of reference images being captured with a corresponding combination of distance and angle values; calculating the intensity distribution of each reference image; estimating the homogeneity of each reference image based on the intensity distribution; determining the reference image with the maximum homogeneity; selecting the distance value associated with the determined reference image as the selected distance, and selecting the angle value associated with the determined reference image as the selected angle.

[0012] In addition to one or more of the features described herein, identifying the features includes selecting an image window corresponding to a segment of the constructed image, the image window having a size selected based on a desired size of the feature.

[0013] In one exemplary embodiment, a method for inspecting a reflective surface includes: capturing a first image of the reflective surface using a first imaging component, the first image including depth information; and capturing a second image of the reflective surface using a second imaging component, the second image including contrast information. The method also includes: estimating a depth profile of the reflective surface based on the depth information; associating the depth profile with the second image; and identifying features of the reflective surface based on the association.

[0014] In addition to one or more of the features described herein, associating the depth profile with the second image includes constructing an image having image properties based on the depth profile.

[0015] In addition to one or more of the features described herein, the first imaging component includes a first camera and a laser pointing toward the reflective surface, the laser being configured to project a laser line onto the reflective surface, wherein capturing the first image includes capturing a series of images, each of the series of images depicting the laser line along the reflective surface at multiple locations.

[0016] In addition to one or more of the features described herein, the second imaging component includes a second camera and an illumination device configured to project a diffuse beam of light onto the reflective surface, wherein the second image is captured when the illumination device is positioned at a selected distance from the reflective surface and oriented relative to the reflective surface at a selected angle.

[0017] In addition to one or more of the features described herein, the selected distance and the selected angle are determined by performing the following: capturing multiple reference images of the reference surface, each of the multiple reference images being captured with a corresponding combination of distance and angle values; calculating the intensity distribution of each reference image; estimating the homogeneity of each reference image based on the intensity distribution; determining the reference image with the maximum homogeneity; selecting the distance value associated with the determined reference image as the selected distance, and selecting the angle value associated with the determined reference image as the selected angle.

[0018] In addition to one or more of the features described herein, identifying the features includes selecting an image window corresponding to a segment of the constructed image, the image window having a size selected based on a desired size of the feature.

[0019] In addition to one or more of the features described herein, the size of the window is selected based on a machine learning model.

[0020] In one exemplary embodiment, the computer program product includes a computer-readable storage medium having instructions executable by a computer processor to cause the computer processor to perform a method. The method includes: capturing a first image of a reflective surface using a first imaging component, the first image including depth information; and capturing a second image of the reflective surface using a second imaging component, the second image including contrast information. The method also includes: estimating a depth profile of the reflective surface based on the depth information; associating the depth profile with the second image; and identifying features of the reflective surface based on the association.

[0021] In addition to one or more of the features described herein, the first imaging assembly includes a first camera and a laser oriented toward the reflective surface, the laser being configured to project a laser line onto the reflective surface, and capturing the first image includes capturing a series of images, each of which depicts the laser line along the reflective surface at multiple locations.

[0022] In addition to one or more of the features described herein, the second imaging component includes a second camera and an illumination device configured to project a diffuse beam of light onto the reflective surface, wherein the second image is captured when the illumination device is positioned at a selected distance from the reflective surface and oriented relative to the reflective surface at a selected angle.

[0023] The present invention also provides the following solutions:

[0024] Option 1. A system for inspecting reflective surfaces, comprising:

[0025] A first imaging component configured to capture a first image of the reflective surface, the first image including depth information;

[0026] A second imaging component configured to capture a second image of the reflective surface, the second image including contrast information; and

[0027] A processor configured to acquire the first image and the second image, the processor being configured to execute:

[0028] The depth profile of the reflective surface is estimated based on the depth information;

[0029] Associate the depth contour with the second image; and

[0030] The characteristics of the reflective surface are identified based on correlation.

[0031] Option 2. The system according to Option 1, wherein the reflective surface is the surface of the brazed joint.

[0032] Option 3. The system according to Option 1, wherein associating the depth profile with the second image includes constructing an image having image attributes based on the depth profile.

[0033] Option 4. The system according to Option 1, wherein the first imaging component includes a first camera and a laser oriented toward the reflective surface, the laser being configured to project a laser line onto the reflective surface, and the first image includes a series of images, each of the series of images depicting the laser line along the reflective surface at multiple locations.

[0034] Option 5. The system according to Option 4, wherein the depth profile is estimated based on laser triangulation technology.

[0035] Option 6. The system according to Option 4, wherein the second imaging component includes a second camera and an illumination device configured to project a diffuse beam of light onto the reflective surface.

[0036] Option 7. The system according to Option 6, wherein the first imaging component and the second imaging component are in fixed positions relative to each other, and the first imaging component and the second imaging component are configured to scan along the reflective surface and simultaneously capture the first image and the second image.

[0037] Option 8. The system according to Option 6, wherein the illumination device is placed at a selected distance from the reflective surface and oriented relative to the reflective surface at a selected angle.

[0038] Option 9. The system according to Option 8, wherein the selected distance and the selected angle are determined by performing the following:

[0039] Multiple reference images of a reference surface are captured, with each of the multiple reference images captured using a corresponding combination of distance and angle values;

[0040] Calculate the intensity distribution for each reference image;

[0041] The homogeneity of each reference image is estimated based on the intensity distribution;

[0042] Determine the reference image with the greatest homogeneity; and

[0043] The distance value associated with the determined reference image is selected as the selected distance, and the angle value associated with the determined reference image is selected as the selected angle.

[0044] Option 10. The system according to Option 3, wherein identifying the feature includes selecting an image window corresponding to a segment of the constructed image, the image window having a size selected based on a desired size of the feature.

[0045] Option 11. A method for inspecting a reflective surface, comprising:

[0046] A first image of the reflective surface is captured by a first imaging component, the first image including depth information;

[0047] A second image of the reflective surface is captured by a second imaging component, the second image including contrast information;

[0048] The depth profile of the reflective surface is estimated based on the depth information;

[0049] Associate the depth contour with the second image; and

[0050] The characteristics of the reflective surface are identified based on correlation.

[0051] Option 12. The method according to Option 11, wherein associating the depth contour with the second image includes constructing an image having image attributes based on the depth contour.

[0052] Option 13. The method according to Option 11, wherein the first imaging component includes a first camera and a laser oriented toward the reflective surface, the laser being configured to project a laser line onto the reflective surface, wherein capturing the first image includes capturing a series of images, each of the series of images depicting the laser line along the reflective surface at multiple locations.

[0053] Option 14. The method according to Option 13, wherein the second imaging component includes a second camera and an illumination device configured to project a diffuse beam onto the reflective surface, wherein the second image is captured when the illumination device is positioned at a selected distance from the reflective surface and oriented relative to the reflective surface at a selected angle.

[0054] Option 15. According to the method of Option 14, wherein the selected distance and the selected angle are determined by performing the following:

[0055] Multiple reference images of a reference surface are captured, with each of the multiple reference images captured using a corresponding combination of distance and angle values;

[0056] Calculate the intensity distribution for each reference image;

[0057] The homogeneity of each reference image is estimated based on the intensity distribution;

[0058] Determine the reference image with the greatest homogeneity; and

[0059] The distance value associated with the determined reference image is selected as the selected distance, and the angle value associated with the determined reference image is selected as the selected angle.

[0060] Option 16. The method according to Option 12, wherein identifying the feature includes selecting an image window corresponding to a segment of the constructed image, the image window having a size selected based on a desired size of the feature.

[0061] Option 17. The method according to Option 16, wherein the size of the window is selected based on a machine learning model.

[0062] Option 18. A computer program product including a computer-readable storage medium having instructions executable by a computer processor to cause the computer processor to perform a method comprising:

[0063] A first image of the reflective surface is captured by a first imaging component, the first image including depth information;

[0064] A second image of the reflective surface is captured by a second imaging component, the second image including contrast information;

[0065] The depth profile of the reflective surface is estimated based on the depth information;

[0066] Associate the depth contour with the second image; and

[0067] The characteristics of the reflective surface are identified based on correlation.

[0068] Option 19. The computer program product according to Option 18, wherein the first imaging component includes a first camera and a laser oriented toward the reflective surface, the laser being configured to project a laser line onto the reflective surface, wherein capturing the first image includes capturing a series of images, each of the series of images depicting the laser line along the reflective surface at multiple locations.

[0069] Option 20. The computer program product according to Option 19, wherein the second imaging component includes a second camera and an illumination device configured to project a diffuse beam of light onto the reflective surface, wherein the second image is captured when the illumination device is positioned at a selected distance from the reflective surface and oriented relative to the reflective surface at a selected angle.

[0070] The above-described features and advantages, as well as other features and advantages of this disclosure, will become readily apparent when taken in conjunction with the accompanying drawings and the following detailed description. Attached Figure Description

[0071] Other features, advantages, and details are shown in the following detailed description by way of example only, with reference to the accompanying drawings, in which:

[0072] Figure 1 An embodiment of a system for inspecting reflective surfaces is described, the system including a first imaging component and a second imaging component;

[0073] Figure 2This is a flowchart depicting various aspects of a method for inspecting a reflective surface according to an embodiment;

[0074] Figure 3 Examples depicting depth contours estimated from images captured by a first imaging component, images captured by a second imaging component, and images constructed based on the depth contours;

[0075] Figure 4 This is a flowchart depicting various aspects of a method for determining depth contours based on an image captured by a first imaging component, according to an embodiment.

[0076] Figure 5 An example of a reference image used in an embodiment of a method for selecting operating parameters of a second imaging component, the operating parameters including the size of the illuminated area and one or more angles;

[0077] Figure 6 This is a flowchart depicting a method for selecting operating parameters of a second imaging component according to an embodiment;

[0078] Figure 7 It is a flowchart depicting various aspects of a method for examining a reflective surface to identify features of interest in an image based on the reflective surface and / or selecting one or more windows used to identify features of interest in an image;

[0079] Figure 8 An example depicting the features of interest and the window used to identify features of interest in an image;

[0080] Figure 9 An example depicting the features of interest and the window used to identify features of interest in an image;

[0081] Figure 10 An example is depicted of the computational time mapping and accuracy mapping used to select a window for recognizing features of interest in an image; and

[0082] Figure 11 A computer system according to an exemplary embodiment is depicted. Detailed Implementation

[0083] The following description is merely exemplary and is not intended to limit this disclosure, its application, or its uses. It should be understood that throughout the drawings, corresponding reference numerals indicate similar or corresponding parts and features.

[0084] According to one or more exemplary embodiments, methods, apparatus, and systems for inspecting reflective surfaces are provided. In embodiments, the surfaces that can be inspected include highly reflective surfaces, such as the surfaces of ground and polished brazed joints (brazed surfaces or polished brazed surfaces) with low surface roughness.

[0085] The inspection system includes a first imaging component configured to capture one or more first images of a reflective surface, such as one or more first images of a laser line on the reflective surface. Each of the one or more first images includes depth information. "Depth" can be defined as the distance from a point on the surface to a reference plane (e.g., a plane orthogonal to the laser axis in the surface direction). In an embodiment, the first imaging component includes an optical camera and a laser configured to project a laser line onto a portion of the surface being imaged. A series of first images ("laser line images") (each displaying a laser line at different locations on the surface) can be analyzed using laser triangulation or other suitable techniques to extract depth information (e.g., surface profile or depth profile).

[0086] The inspection system also includes a second imaging component configured to capture one or more second images of the surface. The second imaging component includes an illumination device that projects diffuse light onto a portion of the surface (the illuminated portion). A camera captures one or more second images (“contrast images”) of the illuminated portion, which include contrast information (e.g., color and / or brightness contrast). The illuminated portion has an illuminated area based on the width of the illuminated portion.

[0087] The processing apparatus or system is configured to analyze an image by reconstructing or calculating a three-dimensional or two-dimensional depth profile based on a first image and associating the depth profile with a second image. For example, an image of a portion of a surface (“constructed image”) is constructed based on the depth profile, and the constructed image is then associated with a second image of that portion of the surface. The constructed image and / or the second image are analyzed to identify features of interest, such as surface defects or discontinuities.

[0088] Embodiments may include a method for selecting operating parameters of a first imaging device. Embodiments of the method include selecting operating parameters (e.g., illuminated area size, working angle, and / or travel angle) of the first imaging device that provides homogeneous illumination. Embodiments also include a method for selecting a window configuration used to analyze the image for feature detection.

[0089] The embodiments described herein present numerous advantages and technical effects. These embodiments provide improved inspection methods capable of detecting defects that are difficult to detect using other techniques.

[0090] Figure 1An embodiment of the inspection system 10 is shown. The inspection system 10 includes an imaging system 12 and an analysis unit 14 configured to receive images of reflective surfaces, such as the surface of a laser-brazed joint or other joint. The imaging system 12 includes a first imaging component 16 configured to capture an image of the reflective surface including depth information. The imaging system 12 also includes a second imaging component 18 configured to illuminate the surface and capture an image of the surface including brightness and / or contrast information.

[0091] Discussion with reference to the example of reflective surface 20 Figure 1 This is an example of an embodiment. In this example, the reflective surface 20 is the surface of the brazed joint 21 used to attach the component to the vehicle roof 22. The embodiment is not limited to this, as it can be used to inspect any suitable reflective surface, such as the surface of a polished laser-welded joint, the brazed surface of a trunk lid, and the brazed surface of a liftgate.

[0092] In one embodiment, the first imaging assembly 16 includes an optical camera 30 and a laser 32, both oriented such that they are both pointed at a selected area of ​​the surface 20. For example, the laser 32 is a line laser configured to emit a laser beam 34 that projects a laser line 36 onto the surface 20. In use, the camera 30 captures images of the surface and the laser line 36 at multiple locations on the surface 20 to produce a series of first images or laser line images. For example, the laser 32 emits the laser beam 34 and scans the laser line 36 across the surface 20. The laser line 36 is scanned along the scanning direction indicated by arrow 38, and the camera 30 captures a series of first images of the surface 20 and the laser line 36 at multiple locations along the scanning direction.

[0093] In one embodiment, the second imaging assembly 18 includes an optical camera 40 and an illumination device 42. The illumination device 42 is oriented toward the surface 20 and aligned with the camera 40 such that when the camera 40 captures an image, a portion 23 of the surface (the illuminated portion having an illuminated area) is illuminated. An example of the illumination device 42 is an on-axis diffuse light (DOAL) device, although any light source suitable for providing the desired homogeneous illumination can be used. The second imaging assembly 18 is operated according to one or more operating parameters selected to increase or optimize the homogeneity of the illumination. Examples of operating parameters include the size of the illuminated area, the distance between the illumination device 42 and the surface 20, and angular parameters such as the working angle and / or travel angle.

[0094] Cameras can be configured to capture grayscale and / or color images. For example, camera 30 can capture a color image such that laser line 36 is visible in the laser line image, and camera 40 can capture either a color or grayscale image. Note that cameras are not limited to optical photography and can be configured to capture any suitable type of image. Examples of other types of images include infrared images.

[0095] Figure 2 An embodiment of method 50 for projecting and inspecting reflective surfaces. Aspects of method 50 can be executed by one or more processors, such as analysis unit 14 and / or processing devices incorporated into one or more imaging components 16 and 18. Note that method 50 can be executed by any suitable processing device or system or combination of processing devices.

[0096] Method 50 includes multiple steps or stages represented by boxes 51-55. Method 50 is not limited to the number or order of steps described herein, as some steps represented by boxes 51-55 may be performed in a different order than described below, or fewer steps may be performed than all of them.

[0097] At frame 51, as the imaging assembly 16 scans along the surface, the first imaging assembly 16 captures a series of first images or laser line images of the reflective surface. Each laser line image includes an image of the laser line located at a given location. The camera 30 can capture images at a high frame rate (e.g., approximately 70 frames per second or higher).

[0098] At frame 52, the surface is illuminated by illumination device 42 and camera 40 captures at least one second image or contrast image. Each contrast image includes contrast information, such as brightness and contrast information. For example, the contrast image is captured at a low frame rate (e.g., approximately 5 fps) and high resolution (e.g., approximately 0.025 mm / pixel). DOAL illumination is used to illuminate the surface; for example, the DOAL illumination may have a color different from the laser color (e.g., red, green, blue).

[0099] A contrast image is captured by camera 40 using selected values ​​of operating parameters, which are chosen to achieve uniform illumination on the illuminated portion of the surface being imaged. Figure 1 As shown, such parameters include the distance between the illuminating device 42 and the surface, and the width of the illuminated portion ( W And / or the area size and the angles of the illumination device 42 and the camera 40. In an embodiment, the selected angle includes the working angle. A W and angle of travel A T From a working perspective A W Defined as the axis of the light emitted by the illuminating device 42 L and the plane tangent to the surface p1 The angle between them, and the angle of travel. A T Defined as on the axis Land a plane parallel to the scanning direction p2 The angle between them.

[0100] In one embodiment, imaging components 16 and 18 are fixed relative to each other such that the imaging group will move together when the imaging components are scanned along the surface.

[0101] At box 53, analysis unit 14 or other processing device estimates surface depth information based on the laser line images. In an embodiment, each of the series of laser line images is analyzed, for example, using laser line triangulation. Depth information from each laser line is assembled to generate a depth profile and / or any other data structure indicating surface depth.

[0102] At box 54, the depth profile of the surface is associated with a contrast image captured by camera 40. In this embodiment, the depth profile is used to generate a constructed image of the surface. For example, the image is constructed to have image properties (e.g., pixel color, shadow, or grayscale) that are functions of depth. The constructed image is then associated with a contrast image of the surface (or a portion thereof). For example, one or more features of the contrast image can be initially identified (e.g., based on manual marking or image analysis), and one or more corresponding features can be identified in the constructed image. The constructed image and the contrast image can then be associated by aligning the corresponding features.

[0103] To perform feature detection (which is discussed further herein), constructed and / or contrast images can be analyzed to detect features of interest. Alternatively, a synthetic image can be generated by combining contrast and constructed images. Contrast images enhance feature detection by providing contrast information about potential features of interest in addition to contour information. The combination of contrast and contour information provides a clearer and more specific representation of the surface compared to presentations obtained using only contour or only contrast information. For example, the combined contrast and contour information can be used to detect the size and extent of relatively large features (such as holes) more accurately than conventional methods, and can be used to detect smaller features that are undetectable by conventional methods.

[0104] Figure 3An example of image construction and correlation is illustrated. In this example, imaging components 16 and 18 scan the surface of a brazed joint or other reflective surface. The laser line image is analyzed to generate a surface profile, represented by a three-dimensional image 60. A grayscale image 62 is constructed, where the shading (grayscale) is a function of the profile depth. In this example, each pixel in the constructed image 62 has a shading that is a function of depth, where lower depths correspond to darker shadings and higher depths correspond to lighter shadings. The constructed image 62 is then correlated with a contrast image 64. The correlation can generate a composite image with one or more image attributes (e.g., color, shading) indicating depth and contrast.

[0105] At box 55, the images generated at boxes 51-54 (e.g., constructed or synthesized images) are analyzed to detect surface features such as defects, discontinuities, and / or other features of interest. For example, regions of interest are selected in the images (e.g., constructed images, contrast images, and / or synthesized images), and defects or other features are identified via appropriate image analysis techniques, such as statistical analysis (e.g., regression) and / or machine learning (e.g., classifiers, neural networks, etc.). As further discussed herein, windowing techniques can be used, where one or more windows of a selected construct (e.g., shape and size) are determined to optimize feature detection.

[0106] Figure 4 The illustration shows an embodiment of a surface reconstruction method 70 for determining depth profiles. Aspects of method 70 can be executed by one or more processors, such as analysis unit 14 and / or processing devices incorporated into an imaging apparatus. Note that method 70 can be executed by any suitable processing device or system, or a combination of processing devices.

[0107] Method 70 includes multiple steps or stages represented by boxes 71-75. Method 70 is not limited to the number or order of steps described herein, as some steps represented by boxes 71-75 may be performed in a different order than described below, or fewer steps may be performed than all of them.

[0108] At box 71, corresponding to a given surface, the contrast image of the surface obtained by the second imaging component 18 is associated with the laser line image of the surface obtained by the first imaging component 16. This is achieved by scanning along the surface at a certain speed. v The scanning imaging component simultaneously captures a series of grayscale contrast images and a series of laser line images.

[0109] Select a portion of the surface and compare its contrast over time. t 1 The location was photographed. At that time... t 2 andt 3 Laser line images taken between intervals are selected such that the laser line images represent the same portion of the surface in the selected contrast image. Time t 2 and t 3 It can be determined based on the following equation:

[0110] t 2 = t 1 – d / v ,and

[0111] t 3 = t 1 – ( d + W ) / v .

[0112] like Figure 1 As shown in the middle figure, d It is the distance between the front end of the illuminated part (e.g., illuminated part 23) and the laser beam 34. W It is the width of the illuminated portion.

[0113] At box 72, the laser line image is examined to determine if the line image is in time. t 2 and t 3 They were photographed in between.

[0114] If so, the laser line image corresponding to the contrast image of the illuminated portion of the surface is selected.

[0115] At box 73, for each selected laser line image, the corresponding laser line is extracted. Laser line extraction can be performed using various algorithms and techniques. In this embodiment, an edge detection algorithm such as the Canny edge detection method is used to perform laser line extraction from the image.

[0116] Examples of edge detection methods for vertically oriented laser lines (e.g., perpendicular to the surface of interest and / or a reference plane) include locating left and right edge points along the laser line in an image using line-by-line computation. In each line, the center point of the laser line is calculated based on the initial estimated positions of the left and right edges. Xc The method then includes searching for the center point. XcThe number of neighboring pixels includes pixels covered by half the width of the laser line in the image. A laser line edge point in a row can be defined as the position of the brightest pixel among its neighbors or the average position of a selected number of brightest pixels. Extrapolation can be used in rows lacking laser line edge points. For example, if only one edge point is detected in a given row, the center point can be defined at the location of the detected edge point, and then neighboring pixels are searched to calculate the missing edge point. If two edge points are missing in a given row, the edge points can be directly extrapolated from the adjacent rows above and below the given row.

[0117] At frame 74, laser line triangulation is performed on the extracted laser line to extract depth information. Laser line triangulation involves calculating the distance from each point on the laser line to the imaging component 16. The position and distance of each point on the laser line are determined relative to a 2D coordinate system fixed relative to the imaging component 16.

[0118] At box 75, a surface profile corresponding to the surface portion of the contrast image is constructed. The surface profile can be a 3D graph, a table, or other data structure indicating profile information.

[0119] To achieve uniform illumination of the surface by the illumination device 18, diffuse illumination can be used. In this embodiment, the illuminated area and imaging angle of the illuminated surface are selected to further enhance the uniformity of illumination.

[0120] Figure 5 and Figure 6 The various aspects of calculating operating parameters (e.g., region and angle) are described. Figure 5 An example of an image depicting a reference surface 80. The reference surface 80 is a surface similar to the surface to be inspected, which is free of defects or discontinuities or features of interest (e.g., a defect-free brazed joint surface).

[0121] Figure 6 The illustration depicts an embodiment of a method 90 used to calculate operating parameters of an imaging apparatus 18. Aspects of method 90 can be executed by one or more processors, such as an analysis unit 14 and / or processing means incorporated into the imaging assembly. Note that method 90 can be executed by any suitable processing means or system, or a combination of processing means.

[0122] Method 90 includes multiple steps or stages represented by boxes 91-94. Method 90 is not limited to the number or order of the steps, as some steps represented by boxes 91-94 may be performed in a different order than described below, or fewer steps may be performed than all of them.

[0123] At frame 91, imaging device 18 captures multiple reference images. Each captured image (or set of images) is for one of multiple combinations of illuminated area size and angle parameters. In this embodiment, the combination of parameters includes the working angle. A W Parameter values, travel angle A T Parameter values ​​and the size or width of the illuminated area. A W and A T Examples of parameter values ​​include 0 degrees, 3 degrees, 6 degrees, and others. Examples of illuminated area widths include 50 mm and 75 mm. Note that these examples are provided for illustrative purposes and are not intended to be limiting.

[0124] At frame 92, the region of interest (ROI) in each reference image is divided into a grid with multiple grid cells. For example, reference image 80 includes a ROI 82 in the form of a brazed surface. The ROI is divided into a 5 x 20 (5 rows, 20 columns) grid comprising grid cells 86.

[0125] Estimating average pixel intensity for each grid cell (86) S It has the highest average pixel intensity. S max Unit 86 and having the minimum average pixel intensity S min Unit 86 was identified.

[0126] At box 93, the homogeneity of illumination for each reference image (e.g., reference image 80) is estimated based on pixel intensity calculations. In this embodiment, the homogeneity value is estimated based on the following equation. U :

[0127] .

[0128] At point 94, the homogeneity is maximized. U An image or set of images is identified, and parameters associated with the identified image or set of images are selected. Inspection of other surfaces similar to the reference surface can then be performed, including imaging the surface through the imaging device 18 at a working angle, travel angle, and illumination size selected at frame 94.

[0129] Defects and discontinuities can vary widely in size, shape, and location. For example, the size of defects that typically occur on laser-brazed surfaces ranges from 0.1 to 2 millimeters.

[0130] Embodiments of the inspection methods described herein may include windowing techniques, wherein one or more sub-regions of the reconstructed image are selected for feature extraction. These sub-regions, also referred to as “windows,” can be customized in size and shape to extract features of varying sizes.

[0131] The window is selected to facilitate accurate feature extraction. For example, selecting a window that is too large relative to the surface features can lead to inaccuracies due to contamination from background noise, while a window that is too small can result in the loss of important feature information.

[0132] Figure 7 This is a flowchart depicting aspects of a method 100 for window construction selection and / or feature extraction from images (e.g., constructed images, contrast images, and / or composite images). The aspects of method 100 can be executed by one or more processors, such as analysis unit 14 and / or processing means incorporated into an imaging assembly. Note that method 100 can be executed by any suitable processing means or system, or a combination of processing means.

[0133] Method 100 includes multiple steps or stages represented by boxes 101-114. Method 100 is not limited to the number or order of the steps, because some steps represented by boxes 101-114 may be performed in a different order than described below, or fewer steps may be performed than all of them.

[0134] In the following description, method 100 is used in conjunction with the analysis of a constructed image including a region of interest. The method is not limited to this, as aspects of the method can be repeated for multiple images if multiple images are constructed to cover a surface extending beyond the size of a single constructed image. Furthermore, although method 100 is described in conjunction with an image constructed from contour information, the method can be applied to contrast images, composite images, and any other images.

[0135] At box 101, imaging is performed using a scanning imaging assembly as described herein. At box 102, contour information from the laser line image and contrast information from the contrast image are correlated as discussed herein to generate a structural image of the surface.

[0136] At box 102a, a decision is made as to whether to use window optimization techniques to select a window construction. If the decision is no (i.e., no optimization techniques are used), a window construction is selected from one or more stored window constructions. If the decision is yes, a window construction is selected based on the optimization techniques described in conjunction with boxes 107-112.

[0137] At box 103, a window configuration is selected, for example, by retrieving a stored window configuration or by selecting parameters based on window optimization techniques. In an embodiment, the window configuration includes one or more windows selected for optimized defect detection. For example, the window configuration includes a large window with a size and shape selected to detect relatively large features. The large window may have a region corresponding to the entire region of interest (e.g., a brazed surface) within the constructed image. The window configuration may also include, for example, small windows with a size and shape selected to detect smaller features.

[0138] At box 104, a segment of the constructed image corresponding to the window size is selected for feature extraction. If the window is smaller than the region of interest (ROI), a separate segment corresponding to the window is selected such that the entire ROI is covered. Segments can be constructed to overlap based on the window.

[0139] At box 105, for each window, use any appropriate technique to extract defects or other features. In this embodiment, a machine learning model, such as a support vector regression model or a support vector machine (SVM), is utilized. Other types of machine learning or artificial intelligence models or other types of classifiers may be used.

[0140] At box 106, based on the results of feature extraction, feature information (e.g., defect identification information) is provided to the user, stored in memory, or otherwise provided. Note that the feature or defect identification results can be used as training data to further train a machine learning model.

[0141] Method 100 may include a window optimization method represented by boxes 107-112. At boxes 107-110, window construction parameters capable of being selected for the optimized construction are initially provided. Parameters include the number of windows at box 107 (e.g., two windows, one for a larger defect and one for a smaller defect), and the window type at box 108 (e.g., window shape, such as square, rectangle, circle). Other window construction parameters include the window size at box 109 (e.g., large and small), and the window overlap (the amount of overlap between adjacent windows) at box 110.

[0142] At box 111, the values ​​of each window construction parameter are determined, for example, through a machine learning model such as SVM. The model can be trained using training data in the form of other images of known defects similar to the expected defects. The model is used to output optimized window construction parameters based on defect or feature type and size. At box 112, the optimized construction parameters are used as the window construction discussed at box 103. The machine learning model can be further trained by manually labeling the features of interest in the constructed image (box 113) and using the labels to further train the machine learning model (box 114).

[0143] Figure 8 and Figure 9 Examples of window sizes that can be selected based on expected features or feature types are shown. Window sizes in this example include large and small window sizes. Large windows are selected to help detect large features, such as large holes. Small windows are selected to help detect smaller features, such as small holes and discontinuous edges.

[0144] Figure 8 An example of image 120 depicting the large features of the orifice 122 and showing the effect in mW / cm 2 Histogram 124 of multiple pixels measuring various intensities. Image 120 corresponds to a large window that covers the entire region of interest. For comparison, a reference image 130 of the defect-free region of interest and its corresponding histogram 132 are also shown.

[0145] Figure 9 An example of image 140 depicting a brazed surface including small defects is shown. In this example, small defects in the form of jagged edges 142 and pinholes 144 are visible. To extract defects, small windows are used to sequentially analyze segments of image 140. In this example, a 5x5 pixel square bounding box is used. Defect recognition image 150 is also shown, illustrating the results of defect detection, where individual defects are annotated using annotation 152.

[0146] Large and small are used as relative terms and do not attempt to limit windows and defects to any specific shape or size.

[0147] refer to Figure 10 In this embodiment, the window optimization method includes selecting a window size to optimize feature detection. The window size is selected to maximize accuracy while minimizing computation time. For example, two window sizes are provided (i.e., a large window and a small window). The large window is rectangular with a 2:1 aspect ratio, and the small window is square. For example, the large window has a size of 0.1-2 mm. 2 The area is selected within a certain size range, and the small window has a size of 0.001-0.1 mm. 2 The range of area sizes.

[0148] Accuracy values ​​for different combinations of small and large window sizes were calculated, as shown in accuracy map 160. Computation times in milliseconds (ms) for different combinations of small and large window sizes were calculated, as shown in computation time map 162. Maps 160 and 162 are color-coded. An optimized window size was selected that has a sufficiently small computation time while maintaining acceptable accuracy. An example of an optimized window size is 0.5 mm. 2 Large window area and 0.01 mm 2The small window area was represented as points 164 and 166. These sizes were 99.95% accurate, and the computation time was 110.23 seconds.

[0149] Figure 11 The illustrations depict various aspects of an embodiment of a computer system 140 capable of performing various aspects of the embodiments disclosed herein. The computer system 140 includes at least one processing means 142, which typically includes one or more processors for performing aspects of the image acquisition and analysis methods described herein.

[0150] The components of computer system 140 include processing device 142 (such as one or more processors or processing units), memory 144, and bus 146, which connects various system components, including system memory 144, to processing device 142. System memory 144 may include various computer system readable media. Such media can be any available media accessible by processing device 142, and includes both volatile and non-volatile media, as well as removable and non-removable media.

[0151] For example, system memory 144 includes non-volatile memory 148, such as a hard drive, and may also include volatile memory 150, such as random access memory (RAM) and / or cache. Computer system 140 may further include other removable / non-removable, volatile / non-volatile computer system storage media.

[0152] System memory 144 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments described herein. For example, system memory 144 may store various program modules that generally perform the functions and / or methodologies of the embodiments described herein. One or more modules 152 may be included to perform functions related to image acquisition and / or control of scan speed and operating parameters. An image analysis module 154 may be included for analysis of images as described herein. System 140 is not limited thereto, as other modules may be included. As used herein, the term "module" refers to processing circuitry, which may include application-specific integrated circuits (ASICs), electronic circuitry, processors (shared, dedicated, or grouped), and memory executing one or more software or firmware programs, combinational logic circuitry, and / or other suitable components providing the said functions.

[0153] Processing device 142 is also capable of communicating with one or more external devices 156, such as a keyboard, a pointing device, and / or any device that enables processing device 52 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Communication with various devices can occur via input / output (I / O) interfaces 164 and 165.

[0154] Processing device 142 can also communicate via network adapter 168 with one or more networks 166, such as a local area network (LAN), a general wide area network (WAN), a bus network, and / or a public network (e.g., the Internet). It should be understood that, although not shown, other hardware and / or software components may be used in conjunction with computer system 40. Examples include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, and data archiving storage systems.

[0155] While the foregoing disclosure has been described with reference to exemplary embodiments, those skilled in the art will understand that various changes can be made without departing from its scope and that its elements can be substituted with equivalents. Furthermore, many modifications can be made without departing from its substantial scope to suit particular situations or materials to the teachings of this disclosure. Therefore, this disclosure is not intended to be limited to the specific embodiments disclosed, but rather to include all embodiments falling within its scope.

Claims

1. A system for inspecting reflective surfaces, comprising: A first imaging component configured to capture a first image of the reflective surface, the first image including depth information; A second imaging component configured to capture a second image of the reflective surface, the second image including contrast information; as well as A processor configured to acquire the first image and the second image, the processor being configured to execute: The depth profile of the reflective surface is estimated based on the depth information; Associate the depth contour with the second image; and The characteristics of the reflective surface are identified based on correlation.

2. The system according to claim 1, wherein the reflective surface is the surface of a brazed joint.

3. The system of claim 1, wherein associating the depth profile with the second image includes constructing an image having image attributes based on the depth profile.

4. The system of claim 1, wherein the first imaging component includes a first camera and a laser oriented toward the reflective surface, the laser being configured to project a laser line onto the reflective surface, and the first image includes a series of images, each of the series of images depicting the laser line along the reflective surface at multiple locations.

5. The system of claim 4, wherein the depth profile is estimated based on laser triangulation technology.

6. The system of claim 4, wherein the second imaging component includes a second camera and an illumination device configured to project a diffuse light beam onto the reflective surface.

7. The system of claim 6, wherein the first imaging component and the second imaging component are in fixed positions relative to each other, and the first imaging component and the second imaging component are configured to scan along the reflective surface and simultaneously capture the first image and the second image.

8. The system of claim 6, wherein the illuminating device is positioned at a selected distance from the reflective surface and oriented relative to the reflective surface at a selected angle.

9. The system of claim 8, wherein the selected distance and the selected angle are determined by performing the following: Multiple reference images of a reference surface are captured, with each of the multiple reference images captured using a corresponding combination of distance and angle values; Calculate the intensity distribution for each reference image; The homogeneity of each reference image is estimated based on the intensity distribution; Identify a reference image with maximum homogeneity; and The distance value associated with the determined reference image is selected as the selected distance, and the angle value associated with the determined reference image is selected as the selected angle.

10. The system of claim 3, wherein identifying the feature includes selecting an image window corresponding to a segment of the constructed image, the image window having a size selected based on a desired size of the feature.

11. A method for inspecting a reflective surface, comprising: A first image of the reflective surface is captured by a first imaging component, the first image including depth information; A second image of the reflective surface is captured by a second imaging component, the second image including contrast information; The depth profile of the reflective surface is estimated based on the depth information; Associate the depth profile with the second image; as well as The characteristics of the reflective surface are identified based on correlation.

12. The method of claim 11, wherein associating the depth contour with the second image includes constructing an image having image attributes based on the depth contour.

13. The method of claim 11, wherein the first imaging component includes a first camera and a laser oriented toward the reflective surface, the laser being configured to project a laser line onto the reflective surface, wherein capturing the first image includes capturing a series of images, each of the series of images depicting the laser line along the reflective surface at multiple locations.

14. The method of claim 13, wherein the second imaging component includes a second camera and an illumination device configured to project a diffuse light beam onto the reflective surface, wherein the second image is captured when the illumination device is positioned at a selected distance from the reflective surface and oriented relative to the reflective surface at a selected angle.

15. The method of claim 14, wherein the selected distance and the selected angle are determined by performing the following: Multiple reference images of a reference surface are captured, with each of the multiple reference images captured using a corresponding combination of distance and angle values; Calculate the intensity distribution for each reference image; The homogeneity of each reference image is estimated based on the intensity distribution; Identify a reference image with maximum homogeneity; and The distance value associated with the determined reference image is selected as the selected distance, and the angle value associated with the determined reference image is selected as the selected angle.

16. The method of claim 12, wherein identifying the feature includes selecting an image window corresponding to a segment of the constructed image, the image window having a size selected based on a desired size of the feature.

17. The method of claim 16, wherein the size of the window is selected based on a machine learning model.

18. A computer program product including a computer-readable storage medium having instructions executable by a computer processor to cause the computer processor to perform a method comprising: A first image of the reflective surface is captured by a first imaging component, the first image including depth information; A second image of the reflective surface is captured by a second imaging component, the second image including contrast information; The depth profile of the reflective surface is estimated based on the depth information; Associate the depth profile with the second image; as well as The characteristics of the reflective surface are identified based on correlation.

19. The computer program product of claim 18, wherein the first imaging component includes a first camera and a laser oriented toward the reflective surface, the laser being configured to project a laser line onto the reflective surface, wherein capturing the first image includes capturing a series of images, each of the series of images depicting the laser line along the reflective surface at multiple locations.

20. The computer program product of claim 19, wherein the second imaging component includes a second camera and an illumination device configured to project a diffuse light beam onto the reflective surface, wherein the second image is captured when the illumination device is positioned at a selected distance from the reflective surface and oriented relative to the reflective surface at a selected angle.

Images