System and method for detecting lines in an image using a vision system

Abstract

System for detecting line features in a captured image, including: ▪ a vision system processor that receives image data of a scene containing line features and has an edge point extractor that a) Intensity gradient images calculated from the image data, b) performs gradient field projections of areas of the intensity gradient images that contain line features and c) identifies a large number of edge points in the respective areas of the intensity gradient images based on the gradient field projections; and ▪ a line detector that generates a multitude of lines that correspond to the edge points extracted from the image.

Description

RELATED APPLICATION

[0001] This application claims the benefit of copending U.S. Provisional Patent Application Serial No. 62 / 249,918, entitled SYSTEM AND METHOD FOR FINDING LINES IN AN IMAGE WITH A VISION SYSTEM, filed November 2, 2015, the teachings of which are incorporated herein by reference. SUBJECT OF THE INVENTION

[0002] This invention relates to machine vision systems and, more particularly, to vision system tools that detect line features in captured images BACKGROUND OF THE INVENTION

[0003] Machine vision systems (also referred to herein simply as "vision systems") are used for a wide variety of tasks in manufacturing, logistics, and industry. Such tasks can include surface and part inspection, placement of objects during assembly, reading patterns and ID codes, and any other operation in which visual data is captured and interpreted for use in further processes. Vision systems typically employ one or more cameras that capture images of a scene containing an object or subject of interest. The object / subject may be stationary or in relative motion. Motion may further be controlled by information derived from the vision system, as in the case of the handling of parts by a robot.

[0004] A common task for a vision system is to detect and label line features in an image. A variety of tools are used to identify and analyze such line features. Typically, these tools rely on a significant contrast difference occurring in a portion of the image. This contrast difference is analyzed using, for example, a caliper tool to determine whether the individual points in the image with contrast difference can be combined to form a line-like feature. If so, a line is identified in the image. In particular, the tools that detect edge points and those that attempt to fit a line to the points operate independently of each other. This increases processing overhead and reduces reliability. When an image includes multiple lines, such tools may be limited in their ability to accurately detect them.In addition, conventional line detection tools designed to detect a single line in an image can be problematic to use when the image contains multiple closely spaced lines with similar orientation and polarity. SUMMARY OF THE INVENTION

[0005] This invention overcomes disadvantages of the prior art by providing a system and method for detecting line features in an image that enables multiple lines to be efficiently and accurately identified and labeled. First, the process calculates x- and y-components of the gradient field at each location in the image, projects the gradient field over a plurality of image sub-regions, and detects a plurality of gradient extrema within each sub-region, yielding a plurality of edge points with associated positions and gradients. Next, the process iteratively selects two edge points, fits a model line to these edge points, and, if the gradients of these edge points match the model line, calculates the full series of non-outlier points whose positions and gradients match the model line.The candidate line with the largest number of non-outliers is retained as a line score, and the set of remaining outlier points is derived. The process then repeatedly applies the line-fitting procedure to this and subsequent sets of outliers to detect a variety of line scores. The line-fitting procedure can be exhaustive or based on a random sample consensus (RANSAC) technique.

[0006] In one illustrative embodiment, a system for detecting line features in a captured image is provided. A vision system processor receives image data of a scene containing line features. An edge point extractor generates intensity gradient images from the image data and detects edge points based on the intensity gradient images. A line recognizer then matches the edge points to one or more lines based on the intensity gradient in the edge points. Illustratively, the line recognizer performs a RANSAC-based process for matching non-outlier edge points to new lines, including iteratively defining lines from outlier edge points with respect to predefined lines. The edge point extractor performs gradient field projection of regions of the intensity gradient images containing line features.Illustratively, the gradient field projection is oriented along a direction determined in response to an expected orientation of one or more of the line features, and the gradient field projection may define a granularity based on a Gaussian kernel. Illustratively, the image data may comprise data from a plurality of images acquired by a plurality of cameras and mapped to a common coordinate space. The image data may be further smoothed using a smoothing (weighting) kernel, which may comprise a 1D Gaussian kernel or other weighting function. The edge points may be selected based on a threshold defined by an absolute contrast and a contrast normalized based on average intensity of the image data.Illustratively, the line detector is configured and arranged to swap edge points representing portions of parallel lines or intersecting lines to correct misalignments and / or identify lines with polarity variation, including mixed polarities in line features, based on gradient values ​​in the edge points. Furthermore, the edge point extractor is illustratively arranged to detect a plurality of gradient magnitude maxima in each of the gradient projection subregions. These gradient magnitude maxima may each be identified as some of the plurality of edge points and may be described by a location vector and a gradient vector.Additionally, the line detector may be configured to determine correspondence between at least one edge point of the extracted plurality of edge points and at least one candidate line of the detected plurality of lines by calculating a metric. This metric may be based on a distance of the at least one edge point from the candidate line and an angular difference between a gradient direction of the edge point and a normal direction of the candidate line. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The following description of the invention refers to the accompanying drawings, of which: Fig. 1 is a diagram of an exemplary vision system arrangement capturing images of an object including a plurality of edge features and a vision system processor including an edge detection tool / module according to an illustrative embodiment; Fig. 2 is a diagram illustrating an overview of the system and method for edge point extraction and line detection from a captured image according to an illustrative embodiment; Fig. 3 is a flowchart of an edge point extraction process according to the system and method of Fig. 2 is; Fig. 4 is a diagram of a field projection on a region of an image that includes edge features, which is part of the edge point extraction process from Fig. 3 is; Fig. Figure 5 is a diagram illustrating the application of a Gaussian kernel to the image to smooth the image for use in the edge point extraction process from Fig. 3 represents; Fig. 6 a diagram of a field projection including the application of a Gaussian kernel to smooth the projection for use in the edge point extraction process of Fig. 3 is; Fig. 7 is a diagram showing a graphical overview of the edge point extraction process from Fig. 3 including the application of Gaussian kernels and calculation of absolute and normalized contrast thresholds for edge points; Fig. Figure 8 is a graph illustrating a range of suitable contrasts for edge points having a sufficient absolute contrast threshold and normalized contrast threshold; Fig. 9 a flowchart of a line detection process based on detected edge points in Fig. 3 is a diagram illustrating an exemplary RANSAC process used in accordance with an illustrative embodiment; Fig. 10 and Fig. 11 are diagrams that depict incorrect or correct arrangement of edge points relative to closely spaced line features; Fig. 12 and Fig. 13 are diagrams illustrating incorrect and correct placements of edge points relative to intersecting line features that can be resolved according to the line recognizer of the illustrative system and method; Fig. 14-17 are diagrams illustrating examples of lines having light-dark polarity, dark-light polarity, either light-dark or dark-light polarity, or mixed polarity, respectively, that can be resolved according to the line detector of the illustrative system and method; and Fig. Figure 18 is a diagram illustrating a modification of the coverage value for a detected line with respect to a user-defined mask. DETAILED DESCRIPTION

[0008] An exemplary vision system assembly 100 that may be used according to an illustrative embodiment is shown in Fig. 1. The system 100 includes at least one vision system camera 110 and may include one or more additional optional cameras 112 (shown in phantom). The illustrative camera(s) 110, 112 include an image sensor (or image picker) S and associated electronics for capturing and transmitting images to a vision system processor 130, which may be instantiated in a single processor and / or computing device 140. The camera 110 (and 112) includes a suitable lens / optics 116 focused on a scene including an inspected object 150. The camera 110 (and 112) may include internal and / or external illumination devices (not shown) that function according to the image acquisition process. Computing device 140 may be any acceptable processor-based system capable of storing and handling image data in accordance with the illustrative embodiment.For example, the computing device 140 may include a PC (as shown), server, laptop, tablet computer, a smartphone, or a similar device. The computing device 140 may include suitable peripherals, such as a bus-based image acquisition card connected to the camera. In alternative embodiments, the vision processor may be partially or entirely contained within the camera body itself and networked with other PCs, servers, and / or camera-based processors that share and process image data. The computing device 140 optionally includes a suitable display 142 that may support a suitable graphical user interface (GUI) that may function according to vision system tools and processors 132 provided in the vision system process 130. It should be noted that a display may be omitted in various embodiments and / or provided solely for setup and maintenance functions.The vision system tools may be part of any acceptable software and / or hardware package suitable for use in object inspection, such as those commercially available from Cognex Corporation of Natick, MA. The computing device may further include associated user interface (UI) components, including, for example, a keyboard 144 and mouse 146, as well as a touchscreen within the display 142.

[0009] The camera(s) 110 (and 112) image some or all of an object 150 disposed within the scene. Each camera defines an optical axis OA around which a field of view is defined based on the optics 116, focal length, etc. The object 150 includes a plurality of edges 152, 154, and 156, each arranged in different directions. For example, the object edges may include those of a cover glass mounted within a smartphone body. Illustratively, the camera(s) may image the entire object or specific locations (e.g., corners where the glass meets the body). A (common) coordinate space may be defined with respect to the object, one of the cameras, or another reference point (e.g., a moving tray on which the object 150 is supported). As shown, the coordinate space is represented by axes 158.These axes illustratively define orthogonal x, y and z axes and rotation θ. z around the z-axis in the xy-plane.

[0010] According to an illustrative embodiment, vision system process 130 interacts with one or more applications / processes (executing on computing device 140) that collectively comprise a set of vision system tools / processes 132. These tools may include a variety of conventional and specialized applications used to resolve image data—for example, a variety of calibration tools and affine mapping tools may be used to map acquired image data to a predetermined (e.g., common) coordinate system. Tools that convert grayscale intensity data from images into a binary image based on a predetermined threshold may also be included. Similarly, tools that analyze the intensity gradient (contrast) between adjacent image pixels (and subpixels) may be provided.

[0011] The vision system process 130 includes a line detection process, tool, or module 134 that locates multiple lines in a captured image according to an illustrative embodiment. Therefore, Fig. 2, which graphically depicts an overview of a line detection process 200 according to the illustrative embodiment. The process 200 consists of two main parts. An input image 210 is provided to the processor. As shown, the image includes a pair of intersecting edges 212 and 214. These may represent a corner region of the object 150 described above. An edge point extractor 220 processes the input image 210 to obtain a set 230 of candidate edge points, including edge points 232 and 234 located along edges 212 and 214, respectively. The edge points 232, 234 and their associated data (e.g., intensity gradient information described below) are provided to a recursive line detector 240, which performs a series of iterative processes on selected edge points.The goal of the iterative processes is to attempt to match other detected edge points to candidate line features. The line detection process 240 results in detected lines 252 and 254, as shown. These results can be provided to other downstream processes 260 that utilize the information—e.g., assembly processes, robotic handling, inspection, ID reading, part / surface inspection, etc.

[0012] It will be Fig. 3, which describes a process for extracting edge points according to one embodiment. One or more images of the scene containing an object or surface with edge features to be detected are acquired (step 310). The image(s) may be extracted by a single camera or by a plurality of cameras. In either case, in step 320, the image pixels may (optionally) be mapped to a new and / or common coordinate space using appropriate calibration parameters. This step may further include smoothing the images as described below. In certain embodiments where a plurality of cameras image discontinuous regions of the scene—for example, focusing on corner regions of a larger object—the common coordinate space may represent the empty space between the cameras' fields of view.As described below, lines extending between such fields of view (e.g., the object edge connecting two detected corner regions) can be extrapolated by the system and method of the illustrative embodiment. The edge points required to detect lines are extracted from the image(s) in the appropriate coordinate space by the edge point extractor using gradient field projection in step 330. Gradient values ​​are first calculated for each pixel, generating two images for x- and y-gradient components. The image(s) is / are further processed by projecting the gradient field over many caliper-like regions.Unlike a conventional caliper tool that projects the intensity value, by projecting the gradient field according to the embodiment, the gradient orientation can be maintained, which facilitates the subsequent line detection process as described below.

[0013] In step 340 and further referring to the diagram of Fig. 4, a portion (a caliper-like region) 400 of the image containing the candidate edge features is subjected to a gradient field projection represented by a plurality of projections 410, 420, 430, searched over the (approximately) expected orientation of the edges in a search direction (arrow SR), with the projections being repeated over the region 400 in an orthogonal projection direction (arrow PR). For each projection (e.g., projection 420), edges appear as local maxima in a gradient field 440 associated with the projection. Generally, a series of edge points within the projection associated with an edge will have an intensity gradient (vectors 552, 554) that is orthogonal to the edge's extent direction. As described below, the user can define the projection direction based on expected line orientation.Alternatively, this may be provided by default or through another mechanism - e.g., analysis of the features of the image.

[0014] Two granularity parameters are involved in the gradient projection step described above. Before calculating the gradient field, the user can choose to smooth the image using an isotropic Gaussian kernel. A first granularity determines the size of this Gaussian smoothing kernel. As shown in diagram 500 of Fig. As shown in Figure 5, a suitably sized Gaussian kernel (e.g., large 512, medium 514, small 516) is applied to smooth the image 210. The first granularity parameter thus determines the size of the isotropic Gaussian smoothing kernel before the field calculation.

[0015] After the gradient field calculation, the process performs a Gaussian-weighted projection instead of the uniform weighting used in conventional caliper tools. Therefore, a second granularity parameter determines the size of the one-dimensional (1D) Gaussian kernel, which is Fig. 6 is used during field projection, in which the region 600 is subjected to a Gaussian smoothed kernel 610, 620, 630. During typical operation, the user (using the GUI) confirms all extracted edges superimposed on the image and then adjusts the granularities and contrast thresholds until the number of extracted edges along the lines to be detected appears satisfactory, while avoiding an excessive number of edges due to background noise in the image. In other words, this step allows the signal-to-noise ratio to be optimized for the image property. This adjustment may further be performed automatically by the system using a default value in various embodiments. It should be noted that the use of a Gaussian weight function is one of a variety of approaches to weighting the projection, including (e.g.) uniform weighting.

[0016] The overall process of gradient field extraction and projection is shown graphically in diagram 700 in Fig. 7. The two granularity parameters, the isotropic Gaussian kernel 710 and the 1D Gaussian kernel 720, are each shown in each half of the overall diagram 700. As shown, each acquired image 210 is subjected to smoothing and decimation 730. The resulting image 740 is then subjected to gradient field calculation 750 as described above to generate the two gradient images 752 and 754. These gradient images are further referred to as g x and g yand represent two orthogonal axes in the common coordinate space. It should be noted that in addition to two gradient images, the intensity image 756 is also typically subjected to the smoothing, decimation, and projection process 760 (using a Gaussian-weighted projection 770 based on the ID Gaussian kernel 720), since the processed intensity information is further used to calculate normalized contrasts according to one embodiment - described below. The result is the projection profiles of gradient images 772(g x ), 774 (g y ) and intensity image 776.

[0017] With further reference to step 350 of process 300 ( Fig. 3) Suitable edge points are then extracted by combining the 1D projection profiles of both the x- and y-gradient images. This is achieved using a coarse contrast calculation 780 and a normalized contrast calculation 790 based on the intensity image. In particular, all local peaks where both coarse projected gradient magnitudes and normalized projected gradient magnitudes exceed respective thresholds are considered a candidate edge point for subsequent line detection according to the following illustrative equations: (gx2+gy2)1 / 2>TABS (gx2+gy2)1 / 2 / l>TNORM where g x and g y are the values ​​of the x-gradient and y-gradient projections at a pixel location, I is the intensity, T ABS is an absolute contrast threshold for coarse projected gradient sizes and T NORMis a normalized contrast threshold for intensity-normalized projected gradient sizes.

[0018] In particular, a point is only considered a candidate edge point if its absolute and normalized contrast both exceed their respective thresholds. This is indicated by the upper right quadrant 810 in the example graph 800 of normalized contrast threshold T NORM compared to absolute contrast limit T ABSshown. The use of dual (absolute and normalized) thresholds generally differs from existing approaches, which typically employ an absolute contrast threshold. The advantages of dual contrast thresholds are clear when, for example, an image contains both dark and bright intensity regions that encompass both edges of interest. To detect edges in dark areas of the image, it is desirable to set a low contrast threshold. However, such a low contrast setting can lead to the detection of false edges in the bright portions of the image. Conversely, to avoid the detection of false edges in the bright areas of the image, it is desirable to set a high contrast threshold. However, with a high contrast setting, the system may fail to adequately detect edges in dark areas of the image.By using a second normalized contrast threshold in addition to the conventional absolute contrast threshold, the system can adequately detect edges in both dark and bright areas and avoid the detection of false edges in bright areas of the image. Therefore, the use of dual contrast thresholds maximizes the speed and robustness of the subsequent line detection stage of the overall process by enabling the detection of relevant edges while avoiding spurious edges.

[0019] With further reference to step 350 ( Fig. 3) Once all the edge points have been extracted, they are represented and stored in a data structure that is useful as a basis for subsequent line detectors. Consider, for example, the following tuple: p=(x,y,gx,gy,gm,go,l,gm / l,m,n) where (x,y) is the location of the edge point, (g x ,Gy ) are the values ​​of its respective x-gradient and y-gradient projections, (g m ,G o ) which consists of (g x ,G y ) is the calculated gradient magnitude and orientation, I is the intensity at the edge point location, g m / I is the intensity-normalized contrast obtained by dividing the gradient size g m is obtained by the intensity I, m is the image index, and n is the projection area index. The location of the edge point can be interpolated as in the standard caliper tool to improve accuracy.

[0020] It should be noted that the edge point extraction process generally operates to perform field projections in a single direction that essentially matches the expected line angle. The tool is therefore most sensitive to edges at this angle, and its sensitivity gradually decreases for edges at other angles, with the rate of decrease depending on the granularity settings, which indirectly determine the field projection length. As a result, the process is limited to detecting lines whose angle is "close" to the expected line angle, which depends on the angle range specified by the user.While the process is adapted to detect lines that are not orthogonal, it is contemplated that in various embodiments it could be generalized to detect lines of any angle over 360 degrees by performing projections in multiple directions (omnidirectional line detection), including orthogonal directions.

[0021] Referring now to step 360 of process 300 ( Fig. 3) Edge point candidates above the threshold are provided to the line detector according to an illustrative embodiment. For example, the line detector operates recursively and employs (e.g.) techniques based on random sample consensus (RANSAC). Further, reference is made to the line detection process 900 in Fig. 9. In step 910, the user specifies the maximum number of expected lines in an image, along with an expected angle, angle tolerance, distance tolerance, and (illustratively) a minimum coverage value (generally defined below), via (e.g.,) the GUI. These parameters are used by the line detector to perform the following processes. The lines are detected in each sub-region of the image by recursively running a RANSAC line detector, with the edge point outliers of one stage becoming the input points of the next stage. Therefore, in step 920, process 900 selects a pair of edge points that are part of the set of edge points identified as extrema in the edge detection process.Process 900 attempts to fit a model line to the selected edge points based on matching gradient values ​​(within the selected tolerance range) that match a model line. In step 924, one or more line candidates from step 922 are output. Each line detection stage outputs a candidate line, its non-outliers, and outliers. The output line(s) are subjected to a calculation of non-outlier edge points that have a position and gradient that match the line candidate (step 926). In step 928, the candidate line with the largest number of non-outliers is identified. The line detection stage described above (steps 920-928) terminates when it reaches the maximum number of allowed RANSAC iterations (decision step 930).The maximum number of iterations within each line detection stage is automatically calculated using an internally calculated worst-case outlier fraction and a user-specified confidence level. Each line detection stage outputs the line with the maximum number of detected edge points from all its iterations, subject to a user-specified fitting tolerance, geometric constraints, and polarity. Each edge point can only be assigned to the non-outlier list of a single line, and each line may only contain at most one edge point from each projection region. The gradient orientation of an edge point, along with its position, is used to determine whether it should be included in the non-outlier list of a candidate line. In particular, edge points should have a gradient orientation that matches the angle of a candidate line.

[0022] If decision step 930 determines that more iterations are allowed, the outliers from the best non-outlier candidate are output (step 940) to the RANSAC process (step 920) to be used to detect a line candidate.

[0023] During each RANSAC iteration, two edge points belonging to different projection domains are randomly selected, and a line is fitted to these two points. The resulting candidate line is only considered further if its angle matches the gradient angles of both edges in the point pair and if the angle of the line matches the user-specified uncertainty range. In general, the gradient direction of an edge point is nominally orthogonal, but it is allowed to deviate by a user-configured angle tolerance. If a candidate line passes these initial checks, the number of non-outlier edge points is evaluated; otherwise, a new RANSAC iteration is initiated.An edge point is considered a non-outlier of a candidate line only if its gradient direction and position match the line, based on user-specified gradient angle and distance tolerances.

[0024] When the RANSAC iterations reach the maximum (decision step 930), the non-outliers of the best detected line candidate are subjected to an improved line fitting using (for example) least squares regression or other acceptable approximation technique, and the set of non-outlier edge points is re-evaluated, repeating these steps a maximum of N (e.g., three or more) times until the number of non-outliers no longer increases or decreases (step 960). This is the line that is marked as detected in step 970.

[0025] Decision step 980 determines whether more lines are to be detected (based on (e.g.,) scanning additional subregions or some other criterion), and if so, the process loops back to step 920 to act on a new set of edge points (step 982). If the points are exhausted or a maximum iteration number is reached, process 900 outputs a set (i.e., multiple) of detected lines in the image in step 990.

[0026] The multi-line detector is designed to perform a final adjustment of existing results in cases where two line results intersect within the inspection area. As is generally the case in Fig. 10 and Fig. 11, closely spaced parallel lines 1010 and 1020 may sometimes produce false line results (ie Fig. 10). However, if such errors occur, an exchange of non-outlier point groups (arrow 1120 in groups 1110 in Fig. 11) can sometimes locate the correct lines with increased coverage values ​​and residuals with reduced matching. Point exchanges can be most effective when an image contains closely spaced parallel lines as shown. Conversely, if the image contains lines 1210 and 1220 that actually overlap each other as shown in Fig. 12 and Fig. 13, then the coverage values ​​are reduced after a point exchange (arrow 1230 in group 1240 in Fig. 12), so that the original results obtained before the exchange are retained by the process to successfully detect crossing lines.

[0027] It should be noted that the RANSAC process is one of the various techniques by which the line recognizer can match points to the line. In alternative embodiments, the candidate points may be selected according to a specified displacement between them, or the image may be processed using (e.g.,) an exhaustive search technique. Therefore, as used herein, reference to the RANSAC technique should be construed broadly to encompass a variety of similar point matching techniques.

[0028] Additional features of this system and method can be provided. These include support for mixed polarity, automatic calculation of the projection area width, support for multi-view line detection, and enabling the input image to be free of prewarp to eliminate optical distortion. These features are described below.

[0029] With further reference to the examples from Fig. 14-16, the system and method for line detection of the illustrative embodiment generally supports standard polarity settings of light-dark, dark-light, and (respectively) both for the contrast between edges being detected. In addition, the system and method may further support a mixed polarity setting ( Fig. 17), where both a light-dark and a dark-light property occur in the same line. Line detection results from all four settings are shown in the following figure. In an illustrative embodiment, the system and method may include a mixed polarity setting that enables detection of a single line containing edge points of opposite polarities. This differs from a conventional "both" polarity setting, where all edge points of a single line exhibit one of the polarities—but only one polarity. The mixed polarity setting may be advantageous, among other applications, when used to analyze the light and dark checkerboards of (e.g.,) a calibration plate.

[0030] The user can select improved shift invariance for line detection. In this case, the edge point extractor essentially uses overlapped projection regions to improve result stability. If the regions do not overlap, the pixels under consideration may move out of the projection regions when the image is shifted, resulting in poor shift invariance in line detection results. Overlapped projection regions ensure that the pixels under consideration are continuously covered by projection regions. When overlapped projection regions are used, incremental computation can be performed, along with possible low-level optimization to maintain speed.

[0031] The user can provide masks that exclude certain sections of the captured image and / or the mapped surface from analysis for line features. This may be desirable when the surface includes known line features that are not of interest, e.g., barcodes analyzed by other mechanisms, text, and other structures not related to the task for which lines are to be detected. Therefore, the edge point extractor can support image masking, where "irrelevant" areas in an image can be masked out and "relevant" areas masked in. When such masking occurs, the coverage values ​​of the detected lines are illustratively re-weighted according to the number of edge points that fall within the mask.

[0032] The exemplary image area 1800 from Fig.18, which illustrates coverage values ​​when image masks are present and the effect of image masking on such coverage values. The edge point extractor supports image masking when "irrelevant regions" in an image can be masked out. As shown, the detected line 1810 is characterized (based on the "relevant" mask regions 1820) by relevant edge points. Such relevant edge points consist of relevant edge point non-outliers 1830 to the line 1810 and relevant edge point outliers 1840 to the line 1810. Irrelevant edge points 1850 on the line 1810 lie between relevant regions 1820 of the mask, as shown in this example, and are not included in the calculation of the coverage value, even though they lie on the line as non-outliers. Potential locations 1860 for edge points along the line 1810 are also determined, as shown.These potential locations are positioned between known points with predictable spacing based on the spacing of detected points. Illustratively, the coverage values ​​of the detected lines are reweighted according to the number of edge points that fall within the mask. The coverage value is therefore modified as follows: Coverage value = number of relevant edge points − non-outliers to line / (number of relevant edge points − non-outliers to line + relevant edge points − outliers to line + number of relevant potential edge points).

[0033] After performing the line detection process according to the system and method herein, the detected lines can be sorted in various ways based on sorting criteria specified by the user (via (e.g.,) the GUI). The user can select from intrinsic sorting metrics such as non-outlier coverage value, intensity, or contrast. The user can further select from extrinsic sorting metrics such as signed distance or relative angle. When using extrinsic sorting metrics, the user can specify a reference line segment based on which the extrinsic metrics of the detected lines are to be calculated.

[0034] As generally described above, this system and method may incorporate multiple field of view (MFOV) overloading, where a vector of images from different fields of view may be fed into the process. The images should all be located in a common client coordinate space based on calibration. As noted above, this feature can be extremely useful in application scenarios where multiple cameras are used to capture partial areas of a single part. Because the edge points retain gradient information, line features projected between gaps in the field of view can still be resolved if the gradients in both fields of view match for a given line orientation and arrangement in each field of view.

[0035] In particular, the system and method does not require (allows the image to lack) warp removal (i.e., does not require the image to be decurved) to remove nonlinear distortion, assuming the distortion is not severe. If the image is not decurved, the system and method can still detect candidate edge points and map the point positions and gradient vectors through a nonlinear transformation.

[0036] It is understood that the line detector provided according to the system and method and various alternative embodiments / improvements is a powerful and robust tool for determining multiple line features under diverse conditions. In general, when used to detect line features, the system and method impose no particular limitation on the maximum number of lines that can be detected in an image. Only memory and computation time impose practical limits on the number of lines that can be detected.

[0037] The foregoing is a detailed description of illustrative embodiments of the present invention. Various modifications and additions may be made without departing from the spirit and scope of this invention. Features of each of the various embodiments described above may be combined with features of other described embodiments, as appropriate, to provide a variety of feature combinations in associated novel embodiments. Furthermore, while the foregoing describes a number of separate embodiments of the apparatus and method of the present invention, what is described herein is merely illustrative of the application of the principles of the present invention.For example, as used herein, the terms "process" and / or "processor" should be broadly construed to encompass a variety of electronic hardware and / or software-based functions and components (and may alternatively be referred to as functional "modules" or "elements"). Furthermore, a depicted process or processor may be combined with other processes and / or processors or divided into various sub-processes or sub-processes. Such sub-processes and / or sub-processors may be variously combined according to embodiments herein. Likewise, it is expressly contemplated that each function, process, and / or processor herein may be implemented using electronic hardware, software consisting of a non-transitory computer-readable medium of program instructions, or a combination of hardware and software.Additionally, as used herein, various directional and arrangement-indicating terms such as "vertical," "horizontal," "upward," "downward," "downward," "bottom," "top," "side," "front," "backward," "left," "right," and the like are used merely as relative conventions and not as absolute directions / arrangements with respect to a fixed coordinate space such as the direction of gravity. Additionally, the term "substantially" or "approximately," when used with respect to a stated measure, value, or characteristic, refers to an amount that is within a normal operating range for achieving desired results, but which includes variation due to inherent inaccuracy and errors within the allowable tolerances of the system (e.g., 1-5 percent). Accordingly, it is intended that this description be considered exemplary only and not otherwise limit the scope of this invention.

[0038] The following is claimed:

Claims

[1] A system for detecting line features in a captured image, comprising: ▪ a vision system processor that receives image data of a scene containing line features and that has an edge point extractor that a) Intensity gradient images are calculated from the image data, b) performs gradient field projections of areas of the intensity gradient images that contain line features and c) detects a plurality of edge points in the respective regions of the intensity gradient images based on the gradient field projections; and ▪ a line detector that generates a plurality of lines that match the edge points extracted from the image. [2] The system of claim 1, wherein the line recognizer performs a RANSAC-based process for fitting non-outlier edge points to new lines, including iteratively defining lines from outlier edge points with respect to predefined lines. [3] The system of claim 1, wherein the gradient field projection is oriented along a direction determined in response to an expected orientation of one or more of the line features. [4] The system of claim 1, wherein the gradient field projection defines a granularity based on a Gaussian kernel. [5] The system of claim 1, wherein the edge point extractor is arranged to detect a plurality of gradient magnitude maxima in each of the gradient projection sub-regions, the gradient magnitude maxima each being identified as some of the plurality of edge points described by a location vector and a gradient vector. [6] The system of claim 1, wherein the line recognizer is arranged to determine correspondence between at least one edge point of the extracted plurality of edge points and at least one candidate line of the recognized plurality of lines by calculating a metric based on a distance of the at least one edge point from the at least one candidate line and an angular difference between a gradient direction of the at least one edge point and a normal direction of the at least one candidate line. [7] The system of claim 1, wherein the image data comprises data from a plurality of images captured by a plurality of cameras, the images being mapped to a common coordinate space. [8] The system of claim 1, further comprising a smoothing kernel that smooths the image data. [9] The system of claim 8, wherein the smoothing kernel comprises a Gaussian kernel. [10] The system of claim 1, wherein the edge points are selected based on a threshold defined by an absolute contrast and a contrast normalized based on average intensity of the image data. [11] The system of claim 1, wherein the line recognizer is arranged to replace at least some edge points from the extracted plurality of edge points that represent portions of parallel lines or intersecting lines from the recognized plurality of lines to correct misalignments. [12] The system of claim 1, wherein the line detector is arranged to identify lines from the detected plurality of lines that have polarity variation. [13] The system of claim 12, wherein the identified lines are lines defined by blending polarities in the line features of the detected plurality of lines based on gradient values ​​in the extracted plurality of edge points. [14] A method for detecting line features in a captured image, comprising the following steps: ▪ Receiving image data of a scene with a vision system processor, the image data including line features; ▪ Calculating intensity gradient images from the image data; ▪ Performing gradient field projections of regions of the intensity gradient images containing line features; ▪ Detecting a plurality of edge points in the respective areas of the intensity gradient images based on the gradient field projections; and ▪ Generate a multitude of lines that match the edge points extracted from the image. [15] The method of claim 14, further comprising applying a RANSAC-based process for fitting non-outlier edge points to new lines, including iteratively defining lines from outlier edge points with respect to predefined lines. [16] The method of claim 14, wherein the gradient field projection is oriented along a direction determined in response to an expected orientation of one or more of the line features. [17] The method of claim 14, wherein the gradient field projection defines a granularity based on a 1D Gaussian kernel. [18] The method of claim 14, further comprising detecting a plurality of gradient magnitude maxima in each of the gradient projection sub-regions, the gradient magnitude maxima each being identified as some of the plurality of edge points described by a location vector and a gradient vector. [19] The method of claim 14, further comprising determining correspondence between at least one edge point of the extracted plurality of edge points and at least one candidate line of the detected plurality of lines by calculating a metric based on a distance of the at least one edge point from the at least one candidate line and an angular difference between a gradient direction of the at least one edge point and a normal direction of the at least one candidate line [20] The method of claim 14, wherein the image data comprises data from a plurality of images captured by a plurality of cameras and mapped to a common coordinate space. [21] The method of claim 14, further comprising smoothing the image data using a smoothing kernel. [22] The method of claim 14, further comprising selecting the edge points based on a threshold defined by an absolute contrast and a contrast normalized based on average intensity of the image data. [23] The method of claim 14, further comprising at least one of (a) exchanging edge points representing portions of parallel lines or intersecting lines to correct misalignments, and (b) identifying lines having polarity variation in the line features, including blended polarities in lines, based on gradient values ​​in the edge points.

Images